A Multimodal Information Verification Method and System for Foreign Affairs Documents Based on a Large Model

Abstract

This application relates to the field of machine learning technology, specifically to a method and system for verifying multimodal information of foreign affairs certificates based on a large model. The method includes: acquiring a certificate portrait image, a verification image, a chip image, a certificate text vector, and a chip text vector; extracting the skin region and watermark region from the certificate portrait image based on skin features and watermark features in the foreground region; dividing the certificate portrait image into multiple sub-regions, and adaptively enhancing each sub-region based on the degree of watermark occlusion and the average grayscale level of the skin in each sub-region, thereby obtaining a watermark-free certificate portrait image; determining whether the current certificate verification process has passed based on the similarity between the verification image and the chip image, the watermark-free certificate portrait image, and the similarity between the certificate text vector and the chip text vector. This application improves the accuracy of certificate verification by adaptively enhancing the certificate portrait image.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, specifically to a method and system for verifying multimodal information of foreign affairs certificates based on a large model. Background Technology

[0002] Foreign affairs documents, such as passports, are legal identification documents used for cross-border activities. In the travel service industry, effectively verifying these documents is a crucial step in confirming the authenticity of travelers' identities. With globalization, the demand for automated identity authentication based on passports and other foreign affairs documents in travel scenarios has increased significantly. However, traditional verification methods often rely on a single data dimension for judgment, which can easily lead to misidentification. Therefore, constructing a multimodal document verification method that combines multi-dimensional data is of great significance for improving the security of travelers' identities.

[0003] When verifying certificate information, existing technologies typically involve directly comparing a photograph of the certificate holder's current face with the image on the certificate into a large-scale visual inspection model. However, these technologies do not consider the differences in certificate styles across different regions. Some certificates have thin films or anti-counterfeiting watermarks on their surfaces. The presence of watermarks may obscure details of the image on the certificate. Consequently, directly inputting the certificate image and the certificate holder's current face image into the large-scale model for facial verification may reduce the accuracy of the verification, resulting in poor verification results. Summary of the Invention

[0004] To address the aforementioned technical problems, the purpose of this application is to provide a method and system for verifying multimodal information of foreign affairs certificates based on a large model. The specific technical solution adopted is as follows: In a first aspect, embodiments of this application provide a method for verifying multimodal information of foreign affairs documents based on a large model, the method comprising the following steps: Acquire the certificate image, verification image, chip image, certificate text vector, and chip text vector during the current certificate verification process, and extract the certificate portrait image from the certificate image; The foreground region in the ID photo image is obtained based on the grayscale values ​​in the ID photo image, and the foreground region is divided into several connected components. Based on the area, tonal dispersion, and distance between the center of each connected component and the center of the ID photo image, the skin confidence of each connected component is obtained, thereby obtaining the skin connected components and detection connected components in the foreground region. Based on the degree of edge direction change and tonal dispersion of each detected connected component, the watermark deviation of each detected connected component is obtained; the watermark deviation and label data of all detected connected components in historical ID photo images are obtained, and a classification model is trained to obtain the watermark connected components in the current ID photo image. The certificate image is divided into multiple sub-regions. Based on the intersection degree of the skin connected component and the watermark connected component in each sub-region, as well as the average gray level of each sub-region, the optimized restriction threshold of each sub-region is obtained, thereby obtaining the enhanced certificate image, and then obtaining the watermark-free certificate image. Based on the similarity between the verification image and the chip image and the watermark-free certificate image, as well as the similarity between the certificate text vector and the chip text vector, it is determined whether the current certificate verification process has passed.

[0005] Preferably, the method for obtaining the skin confidence score of each connected component is as follows: Calculate the ratio between the area of ​​each connected component and the maximum area among all connected components; Convert the ID photo image into an HSV image, calculate the variance of the hue values ​​of all pixels in the H channel within each connected component, and denot it as the color dispersion of each connected component. Calculate the Euclidean distance between the centroid coordinates of each connected component and the centroid coordinates of the ID photo image; The skin confidence of each connected component is positively correlated with the ratio, and negatively correlated with the color dispersion and the Euclidean distance.

[0006] Preferably, the skin connected component and the detection connected component in the foreground region refer to: the connected component corresponding to the maximum skin confidence value in the foreground region, and all other connected components in the foreground region other than the skin connected component.

[0007] Preferably, the method for obtaining the watermark deviation of each detected connected component is as follows: Obtain all edge pixels within each detected connected region; calculate the variance of the gradient direction of all pixels within a local window centered on each edge pixel; The watermark deviation of each detected connected region is positively correlated with the variance of all edge pixels in each detected connected region and the color dispersion of each detected connected region.

[0008] Preferably, the method for obtaining the watermark connected components in the current ID photo image is as follows: the watermark deviation of each detected connected component in the current ID photo image is used as the input of the trained classification model, and the output is the binary classification label data corresponding to each detected connected component; the detected connected component with label data of 0 is recorded as the watermark connected component.

[0009] Preferably, the optimized limitation threshold of each sub-region is calculated using the following formula: In the formula, The optimized constraint threshold for the v-th sub-region; This is the minimum value of the preset limit threshold; The maximum value of the preset limit threshold; is the region enhancement degree of the v-th subregion; tanh() is the hyperbolic tangent function.

[0010] Preferably, the method for obtaining the region enhancement degree of each sub-region is as follows: Obtain the convex hull of the ID photo image that contains the skin connected component, and record all pixels within the convex hull as skin pixels; record all pixels within the watermark connected component of the ID photo image as watermark pixels. If any pixel is both a skin pixel and a watermark pixel, then that pixel is recorded as an occluded pixel. Calculate the percentage of occluded pixels in each sub-region; Calculate the ratio of the average gray value of each sub-region to the average gray value of the foreground region; The enhancement degree of each sub-region is positively correlated with the percentage and negatively correlated with the ratio.

[0011] Preferably, the method for obtaining the watermark-free ID photo image is as follows: all watermark connected components in the enhanced ID photo image are removed, and then the missing areas are repaired using a pre-acquired image restoration model to obtain the watermark-free ID photo image.

[0012] Preferably, the specific process for determining whether the current certificate verification process has passed is as follows: The verification image, along with the chip image and the watermark-free ID photo image, are used as inputs to a pre-acquired image recognition model to perform face verification and output the face verification distance value. The reciprocal of the mean of the two face verification distance values ​​is recorded as the face verification value. The Jaccard similarity between the certificate text vector and the chip text vector is recorded as the text verification value. The sum of the normalized face verification value and the normalized text verification value is recorded as the verification authenticity of the current certificate verification process. If the verification accuracy of the current certificate verification process is less than the preset pass threshold, the current certificate verification process is deemed to have failed; otherwise, the current certificate verification process is deemed to have passed.

[0013] Secondly, embodiments of this application also provide a multimodal information verification system for foreign affairs certificates based on a large model, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of any of the methods described above.

[0014] This application has at least the following beneficial effects: This application addresses the problem that existing technologies do not adequately consider the impact of anti-counterfeiting watermarks on image quality, leading to errors in the accuracy of document verification. By analyzing the features of skin regions in document portrait images and constructing skin confidence scores, the application can accurately locate the skin regions in the portrait images. Furthermore, by analyzing the distinguishing features between the watermark and the portrait in each detection connected region and constructing watermark deviation scores, the application can accurately assess the likelihood that a connected region is a watermark. Finally, by analyzing the degree of skin occlusion by the watermark in each sub-region and constructing region enhancement scores, the application adaptively obtains the limiting thresholds for each sub-region, thereby precisely adjusting the contrast enhancement level of each sub-region. This improves the restoration effect of the document portrait image after watermark removal, ultimately enhancing the accuracy and security of document verification. Attached Figure Description

[0015] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating the steps of a multimodal information verification method for foreign affairs documents based on a large model, provided in one embodiment of this application; Figure 2 This is a flowchart illustrating the process of obtaining optimized limitation thresholds for each sub-region in one embodiment of this application. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by this application to achieve the intended inventive purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of the multimodal information verification method and system for foreign affairs certificates based on a large model proposed in this application. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0019] The following, in conjunction with the accompanying drawings, details the specific scheme of the multimodal information verification method and system for foreign affairs certificates based on a large model provided in this application.

[0020] Please see Figure 1The diagram illustrates a flowchart of a method for verifying multimodal information of foreign affairs documents based on a large model, according to an embodiment of this application. The method includes the following steps: Step 1: Obtain the certificate image, verification image, chip image, certificate text vector, and chip text vector in the current certificate verification process, and extract the certificate portrait image from the certificate image.

[0021] Foreign affairs documents include ordinary passports and diplomatic passports. This example will be analyzed using an ordinary passport as an example.

[0022] During the document verification process, the paper document is first scanned using a passport scanner to obtain an overall image, and a high-definition camera captures the holder's facial image. Then, OCR technology is used to intelligently analyze the image, accurately extracting all text data. Regular expression matching is then used to identify various textual information such as the holder's name, gender, nationality, date of birth, and place of birth. Following the order of textual information recognition, a text vector is constructed based on all textual information, denoted as the document text vector. Simultaneously, a dedicated chip reader extracts the holder's facial image and basic textual information from the embedded chip in the document. Based on the method used to construct the document text feature vector, all textual information is then used to construct a chip text vector for subsequent basic information verification. The use of regular expression matching to extract document text information is a well-known technique and will not be elaborated upon here.

[0023] Next, a neural network model is used to extract the human image region from the certificate image. Specifically, the certificate images used for training in the verification system's database and their corresponding label data are one-hot encoded and then fed into the neural network for training, resulting in a trained neural network model capable of extracting the human image region from the certificate image. The label data for the certificate image is manually labeled; specifically, human image regions and non-human image regions are represented by the numbers 0 and 1, respectively. Neural networks include, but are not limited to, ResNet18, CNN, and YOLO; optimizers include, but are not limited to, Adam and SGD; loss functions include, but are not limited to, MSE, MAE, and cross-entropy loss. The training of the neural network is a well-known technique, and the specific process will not be elaborated further.

[0024] The certificate image captured during the current certificate verification process is input into a trained neural network model to extract the human face region and is recorded as the certificate human face image. Simultaneously, the on-site facial image of the certificate holder is recorded as the verification image, and the facial image of the certificate holder embedded in the chip is recorded as the chip image. It should be noted that the extracted certificate human face image is the smallest bounding rectangle within the certificate image that can contain the human face.

[0025] Step 2: Obtain the foreground region in the ID photo image based on the grayscale values, and divide the foreground region into several connected components; obtain the skin confidence of each connected component based on the area, tonal dispersion, and distance between the center of each connected component and the center of the ID photo image, thereby obtaining the skin connected components and detection connected components in the foreground region.

[0026] Because some regions add watermarks to passports to enhance anti-counterfeiting features, the personal photograph on the passport may be obscured by the watermark. Furthermore, passports are usually covered with a thin film to protect them from wear and moisture over long-term use. However, this film can also cause uneven lighting, interfering with verification. Therefore, to improve the accuracy of subsequent document information verification, image enhancement processing is necessary before inputting the passport image into a large model for facial recognition.

[0027] The CLAHE (Contrast Limited Adaptive Histogram Equalization) algorithm is a classic image enhancement algorithm, but it is significantly affected by the contrast limit threshold. A small threshold can suppress image noise but fails to adequately enhance facial features; a large threshold, while improving contrast, may amplify light spot noise and watermark features caused by the film, leading to false feature points and affecting subsequent image restoration and verification accuracy. Because the film and watermark processes vary across passports, a fixed threshold for the CLAHE algorithm is ill-suited to the complex and ever-changing characteristics of passport images. Therefore, it is necessary to combine the facial features of the passport to obtain a more precise limit threshold.

[0028] Since watermarks may obscure facial features, if an image contains a watermark and the obscuring effect is significant, it is necessary to enhance the image contrast to improve the accuracy of subsequent watermark removal and thus obtain a more accurate certificate verification result.

[0029] To assess the degree to which the watermark obscures the face, it is first necessary to accurately locate the skin area and the watermark area in the portrait. Since the foreground color and background color are usually the two parts with the largest color proportion in ID photo images, the grayscale values ​​of all pixels in the ID photo image are first used as input to Otsu's thresholding method to obtain the segmentation threshold, which is denoted as the preset segmentation threshold. The area composed of all pixels with values ​​less than the preset segmentation threshold is denoted as the foreground area, thus eliminating the interference of the background color.

[0030] The foreground region is then used as input to a connected component generation algorithm to obtain the connected components. The connected component generation algorithm is not limited to seed filling or region growing algorithms.

[0031] Taking the i-th connected component as an example, we will analyze whether it is possible for it to be a skin connected component.

[0032] Since the interference of background color has been eliminated, the area of ​​the skin connected component in the foreground region is usually larger than that of the hair, collar, watermark, etc. At the same time, because the skin color of a person is relatively uniform, the color difference between all pixels within the skin connected component is also smaller.

[0033] First, calculate the area of ​​each connected component. Then, calculate the ratio of the area of ​​the i-th connected component to the maximum area among all connected components; this ratio is denoted as the relative area proportion of the i-th connected component. The relative area proportion reflects the area ratio of the i-th connected component in the foreground region; the larger the value, the larger the area of ​​the i-th connected component, and the more likely it is to belong to the skin region of the portrait. It should be noted that in this embodiment, the area of ​​each connected component is the number of pixels within it.

[0034] The ID photo image is converted into an HSV image. Then, the dispersion of the hue values ​​of all pixels within the i-th connected component in the H channel is calculated, denoted as the color dispersion of the i-th connected component. Color dispersion reflects the degree of color difference within the i-th connected component; the smaller the value, the more uniform and consistent the color within the i-th connected component, and the more it matches the uniformity of skin color. The calculation of dispersion is not limited to variance, standard deviation, or squared deviation; this embodiment uses variance for calculation.

[0035] Furthermore, considering that the face is the most important core area in a photo ID image, it is usually located in the central region of the image to highlight facial features. Therefore, we can further evaluate whether the i-th connected component is a skin connected component by analyzing its position in the photo ID image.

[0036] Obtain the centroid coordinates of each connected component and the image of the person in the certificate, and calculate the Euclidean distance between the centroid coordinates of the i-th connected component and the centroid coordinates of the image of the person in the certificate, denoted as the center distance of the i-th connected component. The center distance reflects the distance between the center of the i-th connected component and the center of the image of the person in the certificate; the smaller the value, the closer the i-th connected component is to the central key area of ​​the image of the person in the certificate, and the more it matches the positional characteristics of the skin.

[0037] In a preferred embodiment, the skin confidence score of each connected region is obtained based on its area, tonal dispersion, and the distance between the center of each connected region and the center of the portrait image in the ID photo. This score characterizes the probability that each connected region belongs to a real skin region. The skin confidence score of each connected region is positively correlated with the relative proportion of its area and negatively correlated with its color dispersion and center distance. Specifically, a positive correlation means that the dependent variable increases (decreases) as the independent variable increases (decreases), and a negative correlation means that the dependent variable decreases (increases) as the independent variable increases (decreases).

[0038] Preferably, in this embodiment, the skin confidence score of the i-th connected component is denoted as... Its specific expression is: In the formula, Skin confidence of the i-th connected component; Let be the relative area ratio of the i-th connected component; , These are the color dispersion and center distance of the i-th connected component, respectively; As a preset constant, in order to avoid the denominator being 0, its value range is [0.005, 0.01]. The value has little impact on the calculation result and can be ignored. In this embodiment, it is taken as 0.008.

[0039] Skin confidence reflects the probability that the i-th connected component belongs to the real skin region; the larger the value, the more the i-th connected component matches the facial skin characteristics in terms of area proportion, color uniformity and spatial location, and the more likely it is to be the facial skin region in the ID photo image.

[0040] Furthermore, obtain the skin confidence scores of all connected components in the foreground region, and denote the connected component corresponding to the maximum skin confidence score as the skin connected component, and denote the other connected components as the detection connected components.

[0041] Step 3: Based on the degree of edge direction change and tonal dispersion of each detected connected component, obtain the watermark deviation of each detected connected component; obtain the watermark deviation and label data of all detected connected components in historical ID photo images, and use this to train a classification model to obtain the watermark connected components in the current ID photo image.

[0042] Furthermore, after obtaining the skin connected components, it is also necessary to analyze whether there is a watermark in the image and to assess whether the watermark will have a significant occlusion effect.

[0043] Let's take the u-th detected connected component as an example for analysis. Since watermarks usually have regular repetitive features, the direction of adjacent edge pixels in a small local area often shows significant consistency; while the edge direction of portrait features such as hair strands and eyebrows is curved and varied.

[0044] Therefore, an edge detection algorithm is used to obtain all edge pixels within the u-th connected component, taking the z-th edge pixel as an example. The edge detection algorithm is not limited to the Canny or Prewitt algorithms. A local window of size c×c is constructed centered on the z-th edge pixel; in this embodiment, c is set to 3. Then, the Sobel operator is used to calculate the gradient magnitudes of each pixel within the local window in the horizontal and vertical directions. Next, the arctan2() function is used to obtain the gradient direction values ​​of each pixel within the local window, reflecting the gradient change direction of each pixel. The dispersion of the gradient direction values ​​among all pixels within the local window centered on the z-th edge pixel is denoted as the local gradient dispersion of the z-th edge pixel. The larger the local gradient dispersion, the greater the difference in gradient direction among the edge pixels within the local area of ​​the z-th edge pixel, and the less consistent it is with the characteristics of a watermark.

[0045] The mean of the local gradient dispersion of all edge pixels in the u-th detected connected component is denoted as the local orientation difference of the u-th detected connected component. The local orientation difference reflects whether the gradient direction of the edge pixels inside the u-th detected connected component is consistent in the local region. The smaller the value, the more consistent the direction is, and the more consistent it is with the watermark feature.

[0046] Furthermore, since watermarks are printed and superimposed on the surface of certificates in the later stages, their textures will directly cover the original details of the portrait; and watermark patterns are usually designed uniformly and printed in batches, with ink evenly distributed. Therefore, within the watermark coverage area, the texture and color of the pixels often show a high degree of consistency.

[0047] Therefore, following the above method for calculating color dispersion, the color dispersion of the u-th detection connected component is calculated to reflect the degree of color difference within the u-th detection connected component.

[0048] Using the same method as calculating the local directional difference and color dispersion of the u-th detected connected component, the local directional difference and color dispersion of each detected connected component are calculated, and the maximum value of the local directional difference and color dispersion of the u-th detected connected component is normalized.

[0049] In a preferred embodiment, the watermark deviation of each detected connected region is obtained based on the degree of edge direction variation and color dispersion of each detected connected region. This watermark deviation is used to characterize the degree of deviation between the features in each detected connected region and the watermark features. The watermark deviation of each detected connected region is positively correlated with the local directional difference and color dispersion of each detected connected region.

[0050] Preferably, in this embodiment, the watermark deviation of the u-th detected connected component is denoted as... Its specific expression is: In the formula, Let the watermark deviation of the u-th detected connected component be denoted as . , These represent the local directional difference and color dispersion of the u-th detected connected component after normalization.

[0051] The watermark deviation can reflect the degree to which the features in the u-th detected connected component deviate from the watermark features; the smaller the value, the more consistent the edge direction and color distribution are in the local area of ​​the u-th connected component, the more consistent it is with the watermark features, and the more likely it is to be a watermark connected component.

[0052] Furthermore, the degree of watermark obstruction needs to be considered. If a watermark exists but does not obscure the skin, there is no need to significantly increase the contrast of the corresponding area to avoid additional errors.

[0053] First, a neural network classification model is used to analyze whether a watermark exists in the image. The watermark deviation is calculated for each detected connected component according to the watermark deviation calculation method. Then, professional personnel manually label the watermarks, setting the label of the watermark deviation corresponding to the real watermark area to 0 and the label of the watermark deviation for non-watermark areas to 1. In this way, T (300 in this example) ID photo images are obtained from the historical data of the verification system, and the watermark deviation and label data of all detected connected components in all ID photo images are obtained. The watermark deviation and label data of all detected connected components are then used as input to the neural network classification model to train the model, resulting in a trained classification model. The classification model is not limited to MLP, CNN, or LightGBM; the optimizer is not limited to Adam or SGD; the loss function is not limited to MSE or MAE; the training process supports early stopping and segmented decay of the learning rate; the training of the classification model is a well-known technique, and the specific process will not be elaborated further.

[0054] It should be noted that during the training of the classification model, if the watermark deviation is not calculated and the classification model is trained directly based on each detected connected component and its corresponding label data, then during the training process, the model needs to learn the visual features of the watermark in the image on its own. This would require a large amount of labeled data and a more complex model structure to evaluate whether the detected connected component might be a watermark. However, by calculating the watermark deviation of each detected connected component and combining it with the corresponding label data to train the classification model, during the training process, the model only needs to identify whether each detected connected component is a watermark based on specific data features, thereby improving the classification accuracy and efficiency of the classification model.

[0055] The watermark deviation of each detected connected component in the captured ID photo image is used as the input of the trained classification model, and the output is the binary label data corresponding to each detected connected component. If there is no detected connected component with label data of 0 in the ID photo image, it means that there is no watermark in the ID photo image; otherwise, the detected connected component with label data of 0 is recorded as the watermark connected component, thus obtaining the watermark region in the ID photo image.

[0056] Step 4: Divide the ID photo image into multiple sub-regions. Based on the intersection degree of the skin connected component and the watermark connected component in each sub-region, and the average gray level of each sub-region, obtain the optimized restriction threshold of each sub-region to obtain the enhanced ID photo image, and then obtain the watermark-free ID photo image. Based on the similarity between the verification image and the chip image and the watermark-free ID photo image, and the similarity between the ID text vector and the chip text vector, determine whether the current ID verification process has passed.

[0057] Furthermore, since the image enhancement process using the CLAHE algorithm first divides the image into several sub-regions, the SLIC segmentation algorithm is used to segment the ID photo image into N (N is 64 in this embodiment) sub-regions to better analyze the degree to which each sub-region needs enhancement. The SLIC segmentation algorithm is a well-known technique, and its specific process will not be described in detail here.

[0058] Taking the v-th sub-region as an example, the following analysis is performed. Since the connected components are currently generated using a connected component generation algorithm, it is difficult to identify whether the position of each pixel in the ID photo image belongs to the overlapping area of ​​the skin and the watermark. Therefore, a convex hull algorithm is used to obtain a convex hull that can contain the skin connected component, and all pixels within the convex hull are recorded as skin pixels. Then, all pixels within the watermark connected components in the image are recorded as watermark pixels. At this point, skin pixels contain watermark pixels. Therefore, if any pixel is both a skin pixel and a watermark pixel, it is recorded as an occluded pixel, indicating that the pixel's location is where the skin is occluded by the watermark. The proportion of occluded pixels in the v-th sub-region is calculated and recorded as the occlusion index of the v-th sub-region. It should be noted that if there are no occluded pixels in the v-th sub-region, the occlusion index of the v-th sub-region is set to 0, indicating that there is no skin occluded by the watermark in this sub-region.

[0059] The occlusion index reflects the degree to which the watermark occludes the skin in the v-th sub-region. The larger the value, the more severe the watermark occlusion of the skin. In this case, a stronger contrast is needed to enhance the sub-region, making the skin texture in the sub-region clearer, so as to improve the accuracy of subsequent image restoration after watermark removal.

[0060] Simultaneously, the ratio of the average gray value of the v-th sub-region to the average gray value of the foreground region is calculated and denoted as the gray-level ratio of the v-th sub-region. The gray-level ratio reflects the relative brightness of the v-th sub-region relative to the overall foreground; the smaller the value, the darker the v-th sub-region is, and the more it needs to be enhanced to improve contrast.

[0061] In a preferred embodiment, the region enhancement degree of each sub-region is obtained based on the degree of intersection between the skin connected region and the watermark connected region in each sub-region, as well as the average gray level of each sub-region. This region enhancement degree is used to characterize the degree of contrast enhancement of each sub-region. The region enhancement degree of each sub-region is positively correlated with the occlusion index of each sub-region and negatively correlated with the gray level ratio of each sub-region.

[0062] Preferably, in this embodiment, the region enhancement degree of the vth sub-region is denoted as... Its specific expression is: In the formula, Let v be the region enhancement degree of the v-th sub-region. Let be the occlusion index of the v-th sub-region; The grayscale ratio of the v-th sub-region; This is a preset constant; The purpose is to avoid the problem that other parameters are meaningless when the molecule is 0.

[0063] The region enhancement value reflects the degree of image enhancement in the v-th sub-region. The larger the value, the greater the degree of watermark occlusion on the skin and the lower the local brightness in the v-th sub-region, and the greater the need for enhancement to improve contrast.

[0064] Based on the above method for calculating regional enhancement, the regional enhancement of all sub-regions in the ID photo image is calculated.

[0065] Furthermore, the constraint thresholds for each sub-region in the CLAHE algorithm are optimized. Specifically, the range of the constraint threshold values ​​is set as follows: In this embodiment and If we take values ​​of 1 and 10 respectively, then the formula for calculating the optimized constraint threshold for each sub-region is: In the formula, The optimized constraint threshold for the v-th sub-region; This is the minimum value of the preset limit threshold; The maximum value of the preset limit threshold; is the region enhancement degree of the v-th sub-region; tanh() is the hyperbolic tangent function used to normalize the data.

[0066] The flowchart for obtaining the optimized limit threshold for each sub-region is as follows: Figure 2 As shown.

[0067] Using the ID photo image as input to the CLAHE algorithm, the image is divided into N sub-regions using the same sub-region division method described above. The optimized threshold values ​​for each sub-region are then used as their respective threshold values ​​for image enhancement, resulting in the enhanced ID photo image. This image enhancement method ensures that areas not obscured by the watermark and with high brightness receive minimal enhancement to avoid generating additional noise; conversely, areas heavily obscured by the watermark and with low brightness receive greater enhancement to improve contrast.

[0068] Furthermore, the watermark connected components identified by the classification model in the ID card portrait image are removed. Then, a pre-acquired image inpainting model is used to repair the missing areas, resulting in a watermark-free ID card portrait image. This avoids the watermark obscuring facial features and affecting subsequent verification accuracy. Before image inpainting, the ID card portrait image is adaptively enhanced using watermark distribution features, thereby improving skin details and enhancing the image inpainting effect after watermark removal. Image inpainting models include, but are not limited to, GANs and DF3Net. Image inpainting using image inpainting models is a well-known technique and will not be elaborated upon here.

[0069] Furthermore, the verification image, along with the chip image and the watermark-free ID photo, are used as inputs to a pre-acquired large-scale image recognition model for face verification, outputting the face verification distance value. The reciprocal of the mean of the two face verification distance values ​​is then recorded as the face verification value. Large-scale image recognition models include, but are not limited to, DeepFace and FaceNet. Using large-scale image recognition models for face recognition is a well-known technique and will not be elaborated upon here.

[0070] Finally, the similarity between the certificate text vector and the chip text vector is calculated and denoted as the text verification value. The sum of the normalized face verification value and the normalized text verification value is denoted as the verification authenticity of the current certificate verification process. The vector similarity calculation method is not limited to cosine similarity or Jaccard similarity; this embodiment uses Jaccard similarity for calculation. The normalization method is not limited to tanh function or sigmoid function; this embodiment uses tanh function for calculation.

[0071] The system retrieves the verification accuracy of T historical certificates from its database and uses these T verification accuracy data as input to the Otsu threshold method to obtain a segmentation threshold, which is then recorded as the preset qualified threshold. If the verification accuracy of the current certificate verification process is less than the preset qualified threshold, the current certificate verification process is deemed to have failed, indicating a mismatch between the certificate holder and the certificate information. In such cases, professional personnel are notified for specialized processing. Conversely, if the verification accuracy is greater than the preset qualified threshold, the current certificate verification process is deemed to have passed, and no further action is taken.

[0072] Thus, a multimodal information verification method for foreign affairs certificates based on a large model has been completed.

[0073] Based on the same inventive concept as the above methods, this application also provides a foreign affairs document multimodal information verification system based on a large model, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any one of the above-described foreign affairs document multimodal information verification methods based on a large model.

[0074] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments of this specification have been described above. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are possible or may be advantageous.

[0075] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0076] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for verifying foreign affairs certificate multi-modal information based on a large model, characterized in that, The method includes the following steps: Acquire the certificate image, verification image, chip image, certificate text vector, and chip text vector during the current certificate verification process, and extract the certificate portrait image from the certificate image; The foreground region in the ID photo image is obtained based on the grayscale values ​​in the ID photo image, and the foreground region is divided into several connected components. Based on the area, tonal dispersion, and distance between the center of each connected component and the center of the ID photo image, the skin confidence of each connected component is obtained, thereby obtaining the skin connected components and detection connected components in the foreground region. Based on the degree of edge direction change and tonal dispersion of each detected connected component, the watermark deviation of each detected connected component is obtained; the watermark deviation and label data of all detected connected components in historical ID photo images are obtained, and a classification model is trained to obtain the watermark connected components in the current ID photo image. The certificate image is divided into multiple sub-regions. Based on the intersection degree of the skin connected component and the watermark connected component in each sub-region, as well as the average gray level of each sub-region, the optimized restriction threshold of each sub-region is obtained, thereby obtaining the enhanced certificate image, and then obtaining the watermark-free certificate image. Based on the similarity between the verification image and the chip image and the watermark-free certificate image, as well as the similarity between the certificate text vector and the chip text vector, it is determined whether the current certificate verification process has passed. The formula for calculating the optimized restriction threshold of each sub-region is as follows: In the formula, The optimized constraint threshold for the v-th sub-region; This is the minimum value of the preset limit threshold; The maximum value of the preset limit threshold; The region enhancement degree of the v-th subregion; tanh() is the hyperbolic tangent function; The method for obtaining the region enhancement degree of each sub-region is as follows: Obtain the convex hull of the ID photo image that contains the skin connected component, and record all pixels within the convex hull as skin pixels; record all pixels within the watermark connected component of the ID photo image as watermark pixels. If any pixel is both a skin pixel and a watermark pixel, then that pixel is recorded as an occluded pixel. Calculate the percentage of occluded pixels in each sub-region; Calculate the ratio of the average gray value of each sub-region to the average gray value of the foreground region; The enhancement degree of each sub-region is positively correlated with the percentage and negatively correlated with the ratio.

2. The method for verifying multimodal information of foreign affairs certificates based on a large model as described in claim 1, characterized in that, The method for obtaining the skin confidence score of each connected component is as follows: Calculate the ratio between the area of ​​each connected component and the maximum area among all connected components; Convert the ID photo image into an HSV image, calculate the variance of the hue values ​​of all pixels in the H channel within each connected component, and denot it as the color dispersion of each connected component. Calculate the Euclidean distance between the centroid coordinates of each connected component and the centroid coordinates of the ID photo image; The skin confidence of each connected component is positively correlated with the ratio, and negatively correlated with the color dispersion and the Euclidean distance.

3. The method for verifying multimodal information of foreign affairs certificates based on a large model as described in claim 1, characterized in that, The skin connected component and the detection connected component in the foreground region refer to: the connected component corresponding to the maximum confidence value of the skin in the foreground region, and all other connected components in the foreground region other than the skin connected component, respectively.

4. The method for verifying multimodal information of foreign affairs certificates based on a large model as described in claim 2, characterized in that, The method for obtaining the watermark deviation of each detected connected component is as follows: Obtain all edge pixels within each detected connected region; calculate the variance of the gradient direction of all pixels within a local window centered on each edge pixel; The watermark deviation of each detected connected region is positively correlated with the variance of all edge pixels in each detected connected region and the color dispersion of each detected connected region.

5. The method for verifying multimodal information of foreign affairs certificates based on a large model as described in claim 1, characterized in that, The method for obtaining the watermark connected components in the current ID photo image is as follows: the watermark deviation of each detected connected component in the current ID photo image is used as the input of the trained classification model, and the output is the binary label data corresponding to each detected connected component; the detected connected component with label data of 0 is recorded as the watermark connected component.

6. The method for verifying multimodal information of foreign affairs certificates based on a large model as described in claim 1, characterized in that, The method for obtaining the watermark-free ID photo image is as follows: all watermark connected components in the enhanced ID photo image are removed, and then the missing areas are repaired using a pre-acquired image restoration model to obtain the watermark-free ID photo image.

7. The method for verifying multimodal information of foreign affairs certificates based on a large model as described in claim 1, characterized in that, The specific process for determining whether the current certificate verification process has passed is as follows: The verification image, along with the chip image and the watermark-free ID photo image, are used as inputs to a pre-acquired image recognition model to perform face verification and output the face verification distance value. The reciprocal of the mean of the two face verification distance values ​​is recorded as the face verification value. The Jaccard similarity between the certificate text vector and the chip text vector is recorded as the text verification value. The sum of the normalized face verification value and the normalized text verification value is recorded as the verification authenticity of the current certificate verification process. If the verification accuracy of the current certificate verification process is less than the preset pass threshold, the current certificate verification process is deemed to have failed; otherwise, the current certificate verification process is deemed to have passed.

8. A multimodal information verification system for foreign affairs certificates based on a large model, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method for verifying multimodal information of foreign affairs certificates based on a large model as described in any one of claims 1-7.

Images