Foreign affairs certificate recognition method and system based on multi-modal fusion

Abstract

This application relates to the field of image recognition technology, specifically to a method and system for recognizing diplomatic documents based on multimodal fusion. The method includes: using semantic segmentation to obtain text recognition regions and portrait regions from diplomatic document images; performing edge detection and rectangle fitting on the text recognition regions to determine the text ratio of each fitted rectangle and the text adhesion degree of each image block; matching pixels on both sides of the portrait region, analyzing the symmetry of each matching point with respect to the facial symmetry line of the portrait region, and determining a local enhancement factor; determining the image gain requirement based on the pixel ratio of the text recognition region to the portrait region in each image block, and correcting the contrast cropping threshold of the image enhancement algorithm to enhance the diplomatic document image; and recognizing the text content and portrait content in the enhanced diplomatic document image. This application aims to enhance diplomatic document images and increase the reliability of diplomatic document recognition.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, specifically to a method and system for recognizing foreign affairs certificates based on multimodal fusion. Background Technology

[0002] Multimodal fusion technology can collaboratively process data from different information sources, such as visual features of document images (layout, watermark, hologram), structured text content (obtained via OCR), biometric features (face, fingerprint, etc.), and digital signatures and encrypted data in embedded chips. By deeply fusing this heterogeneous information, the system can achieve a more comprehensive and reliable "identity verification" judgment, significantly improving recognition accuracy and anti-interference capabilities in complex scenarios.

[0003] In the process of extracting information from foreign affairs certificates, it is necessary to photograph the certificates and extract the image content using OCR technology. However, the effectiveness of OCR extraction is limited by image quality, making image enhancement crucial. Currently used image enhancement algorithms, such as Contrast-Limited Adaptive Histogram Equalization (CLAHE), rely on fixed parameters, such as the clip threshold. However, the same parameters can produce significantly different enhancement effects under different shooting angles and lighting conditions. In special cases, such as strong lighting or reflections, existing algorithms may fail to obtain high-quality images, leading to difficulties in OCR extraction or reducing the accuracy and reliability of information extraction. Summary of the Invention

[0004] In view of the above, it is necessary to provide a method and system for recognizing foreign affairs documents based on multimodal fusion to solve the above problems.

[0005] The first aspect of this application provides a method for recognizing foreign affairs documents based on multimodal fusion, the method comprising: The process involves acquiring images of diplomatic credentials, using semantic segmentation to obtain the text recognition region and the human image region, and then uniformly segmenting the diplomatic credentials image to obtain several image blocks. Edge detection and rectangle fitting are performed on the text recognition region to obtain each fitted rectangle; the text rate of each fitted rectangle is determined based on the width distribution characteristics of each fitted rectangle; and the text adhesion degree of each image block is determined based on the length distribution of all fitted rectangles in each image block. By analyzing the degree of symmetry of the human image region and combining it with the degree of dispersion of the gray values ​​of pixels in each image block, the local enhancement factor of each image block is determined. Based on the ratio of the number of pixels in the text recognition region and the portrait region in each image block, combined with the overall distribution of the text rate of the fitted rectangle in each image block, the text adhesion degree, and the local enhancement factor, the image gain requirement of each image block is determined, and the contrast cropping threshold of the image enhancement algorithm is corrected to enhance the foreign affairs certificate image. Recognize the text content and human image content in enhanced diplomatic document images.

[0006] Preferably, determining the text rate for each fitted rectangle specifically involves: For each image patch, the average width of all fitted rectangles is obtained, and the negative correlation mapping result of the difference between the width of each fitted rectangle and the average width is calculated. After normalization, the text rate of each fitted rectangle is obtained.

[0007] Preferably, the process of determining the text cohesion of each image block is as follows: normalize the length of the fitted rectangle of all edges contained in each image block and then take the average value.

[0008] Preferably, the step of determining the local enhancement factor for each image patch is as follows: The human portrait region is segmented based on the facial symmetry line, and the pixels of the two segments are matched. Analyze the distribution characteristics of the distance difference between each matching point pair and the facial symmetry line of the human image region to determine the regional blur of each matching point pair; The difference between the natural number 0.5 and the normalized value of the discreteness of each image block is obtained, and positively fused with the average regional blur of all portrait regions within the image block to obtain the local enhancement factor of each image block.

[0009] Preferably, the process of determining the regional ambiguity of each matching point pair specifically involves: The difference in distance from the two matched pixels to the midline is recorded as the midline deviation of the matched point pair; the mean of the midline deviations of all matched point pairs is taken as the portrait symmetry deviation. Subtract the human image symmetry deviation from the midline deviation of each matching point pair and normalize the result to obtain the region blur of each matching point pair.

[0010] Preferably, determining the image gain requirement for each image block specifically involves: For each image patch, calculate the percentage of pixels in the human face region, denoted as the first percentage; calculate the percentage of pixels in the text recognition region, denoted as the second percentage; The first proportion is used as the weight of the local enhancement factor of each image block; the average text rate of all fitted rectangles in each image block is multiplied by the text adhesion degree, and the second proportion is used as the weight of the obtained multiplication result. The local enhancement factor of each image block is weighted and summed with the obtained multiplication result to obtain the image gain requirement of each image block.

[0011] Preferably, the process of correcting the contrast cropping threshold of the image enhancement algorithm specifically involves: Calculate the sum of the natural number 1 and the image gain requirement, and then multiply it by the preset contrast clipping threshold to obtain the contrast clipping threshold correction value for each image block. The final contrast cropping threshold correction value is obtained by averaging the contrast cropping threshold correction values ​​of each image patch and all its neighboring image patches.

[0012] Preferably, the process of recognizing the text content and facial image content in the enhanced foreign affairs document image specifically involves: Obtain the database of foreign affairs certificates; extract the text recognition region from the enhanced foreign affairs certificate image, and obtain the text field matching rate between the structured fields in the text recognition region and the text in the database; The facial image region in the enhanced foreign affairs certificate image is matched with the facial images in the database to obtain the facial matching degree; The average of the text field matching rate and the image matching degree is used as the foreign affairs certificate recognition index. When the foreign affairs certificate recognition index is greater than a preset threshold, the content of the foreign affairs certificate is judged to be accurate; otherwise, the content of the foreign affairs certificate is judged to be inaccurate.

[0013] Preferably, the specific method for obtaining the text field matching rate is as follows: The average of the negative correlation mapping results of all minimum edit distances obtained from the foreign affairs certificate image is calculated by taking the minimum edit distance between each structured field in the text recognition region and the database text, and using this average as the text field matching rate.

[0014] Secondly, embodiments of this application also provide a foreign affairs document recognition system based on multimodal fusion, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of any of the methods described above.

[0015] This application has at least the following beneficial effects: This application addresses common issues in foreign affairs documents such as text adhesion, excessive ink, and blurred text due to wear. It quantifies the text adhesion of image blocks through edge detection and rectangle fitting analysis, and dynamically adjusts image enhancement parameters accordingly. Stronger sharpening and contrast enhancement are applied to highly adhered areas, effectively separating connected characters, restoring text details, and significantly reducing OCR misrecognition and missed recognition rates.

[0016] Secondly, leveraging the symmetry of facial structure and combining keypoint detection and matching algorithms, blurriness assessment is performed on the portrait region. For regions with significant keypoint matching deviations due to uneven lighting, motion blur, or low resolution, the enhancement intensity is adaptively increased to strengthen facial detail information, providing clearer and more reliable input images for face comparison and improving the accuracy of portrait verification.

[0017] Finally, based on image enhancement, highly reliable OCR text and facial features are extracted simultaneously. Cross-validation is performed using dual-channel information (text content matching + face comparison) to effectively resist the risks of forgery, occlusion, or tampering that may exist in single-modal recognition, and to greatly improve the overall credibility of foreign affairs certificate authenticity judgment and identity verification. Attached Figure Description

[0018] Figure 1 A flowchart illustrating the steps of a foreign affairs document recognition method based on multimodal fusion provided in one embodiment of this application; Figure 2 A flowchart illustrating the process of obtaining image gain requirement according to one embodiment of this application. Detailed Implementation

[0019] In the description of the embodiments of this application, the words "exemplary," "or," and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary," "or," and "for example" is intended to present the relevant concepts in a specific manner.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0021] It should also be noted that the terms "first" and "second" in this application and its accompanying drawings are used to distinguish similar objects, rather than to describe a specific order or sequence. The methods disclosed in the embodiments of this application or the methods shown in the flowcharts include one or more steps for implementing the method. Without departing from the scope of protection of this application, the execution order of multiple steps can be interchanged, and some steps can also be deleted.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0023] The following description, in conjunction with the accompanying drawings, details the specific scheme of the foreign affairs document recognition method and system based on multimodal fusion provided in this application.

[0024] Please see Figure 1 The diagram illustrates a flowchart of a method for recognizing foreign affairs documents based on multimodal fusion according to an embodiment of this application. The method includes the following steps: The first step is to acquire images of diplomatic credentials, use semantic segmentation to obtain the text recognition region and the human image region of the image, and then uniformly segment the diplomatic credentials image to obtain several image blocks.

[0025] Use a high-resolution scanner to acquire images of the front and back of the certificate, and then convert the obtained images to grayscale to obtain the foreign affairs certificate image, which is recorded as the image to be enhanced.

[0026] When enhancing the captured images, considering the different focus features of portrait recognition and text recognition, this application extracts the portrait and text regions in the image separately, and then performs adaptive image enhancement adjustments based on different recognition features, so that the image enhancement can increase both the accuracy of portrait region recognition and the reliability of OCR text extraction.

[0027] Based on this, the image to be enhanced is analyzed, and existing semantic segmentation algorithms are used to obtain the label information of the pixels. After obtaining the pixel-level labels of the semantic segmentation output, connected component analysis and morphological closing operations are performed on the "portrait" type pixels to extract the complete portrait mask. The same method is used to process the OCR text recognition region type pixels to finally obtain the portrait region and the OCR text recognition region. The segmentation algorithm used in this embodiment is the U-net semantic segmentation algorithm, which is a well-known existing technology and will not be described in detail in this application.

[0028] This application employs a contrast-limited adaptive histogram equalization (CLAHE) image enhancement algorithm, with a preset contrast clipping threshold (clip_limit) of 0.02, to enhance the image. Next, image feature analysis is performed, and the feature parameters are adjusted appropriately based on the analysis results.

[0029] The feature parameter `clip_limit` (contrast cropping threshold) controls the enhancement intensity of histogram equalization. A higher value results in a more significant enhancement, but may also introduce noise. Therefore, this parameter should be dynamically adjusted based on the specific image to achieve the best enhancement effect.

[0030] In this application, the algorithm divides the image into blocks and then performs image enhancement on the divided images. In this embodiment, the image is divided into N image blocks of the same size, where the value of N is 100. This value can be adjusted by the implementer according to the actual size of the image. When the image size increases, the value of N should increase accordingly; while when the image size is small, the value of N can be appropriately decreased.

[0031] The second step is to perform edge detection and rectangle fitting on the text recognition region to obtain each fitted rectangle; based on the width distribution characteristics of each fitted rectangle, determine the text rate of each fitted rectangle; and based on the length distribution of all fitted rectangles in each image block, determine the text adhesion degree of each image block.

[0032] Considering that some diplomatic documents may contain interconnected text (e.g., thickened font due to excessive ink during printing, or blurred text areas due to wear and tear), this can make the text difficult to recognize, increasing the challenge of OCR text recognition. Therefore, for these potentially connected areas, image sharpening is necessary to enhance text detail and facilitate the separation of adjacent characters. To this end, this application optimizes text recognition performance by performing adhesion analysis on the text recognition region and dynamically adjusting the image enhancement factor based on the analysis results.

[0033] Edge detection is performed on the obtained OCR text recognition region image, and the obtained edges are fitted using the minimum fitting rectangle. The length of the fitted rectangle can then be analyzed. The longer the fitted rectangle is, the more likely it is to contain multiple connected texts. Conversely, the shorter the fitted rectangle is, the more likely it is to indicate that the texts are clearly defined and not adjacent.

[0034] First, edge information is analyzed. In diplomatic documents, there may be areas such as text borders. These areas are easily misidentified as text recognition regions during rectangle fitting and recognition, leading to inaccurate identification of the actual text content and potentially incorrect text adhesion judgments. Considering that the edges of the rectangles formed by text borders in diplomatic documents are usually wider than the fitted rectangles of the text recognition regions, this characteristic is utilized to optimize the rectangle fitting algorithm, thereby improving the accuracy of text recognition regions and avoiding misidentification of text borders. Therefore, this application analyzes the obtained fitted rectangles to determine their likelihood of being text input regions.

[0035] Specifically, the average width of all fitted rectangles is obtained, and the negative correlation mapping result of the difference between the width of each fitted rectangle and the average width is calculated. After normalization, the text rate of each fitted rectangle is obtained. The larger the difference between the width of each fitted rectangle and the average width, the more likely the fitted edge of the rectangle is to be a text box, rather than an edge corresponding to text information. In this embodiment, the difference between the width of each fitted rectangle and the average width is calculated by the absolute value of the difference between the width of each fitted rectangle and the average width. In other embodiments, it can also be calculated by the square of the difference. The negative correlation mapping result of the variable is the reciprocal of the variable. It should be noted that if the variable is equal to 0, the text rate is meaningless, so a non-zero positive number needs to be added to the denominator. In this embodiment, the value is 0.01. In other embodiments, the negative correlation mapping of the variable can also be that the opposite of the variable is used as the independent variable of exp(), where exp() represents an exponential function with the natural constant as the base. It should be understood that the larger the value of the text rate, the more likely the edge information corresponding to the fitted rectangle is text edge information. The normalization method adopts the maximum-minimum normalization method.

[0036] Furthermore, based on the block segmentation results, the length of the fitted rectangle corresponding to the edge information contained in each image block is calculated. The text adhesion degree of the image block is then obtained based on the obtained length. Specifically, the lengths of the fitted rectangles for all edges contained within each image block are normalized and averaged to obtain the text adhesion degree of the image block. The normalization method used is min-max normalization. That is, the longer the length of the fitted rectangle corresponding to the edge information within the image block, the greater the possibility of text adhesion, and the greater the difficulty of text recognition in the corresponding region. This requires more enhancement of the region, meaning a larger `clip_limit` parameter is needed to better highlight text details and ensure the reliability of text recognition.

[0037] The third step is to analyze the degree of symmetry of the portrait area and, in combination with the degree of dispersion of the gray values ​​of pixels in each image block, determine the local enhancement factor of each image block.

[0038] Considering facial recognition features, ambiguity analysis is performed on the facial region. The greater the ambiguity of the obtained region, the more necessary it is to enhance that region. The specific calculation method is as follows: Considering that the human face is a roughly symmetrical structure, the facial symmetry line of the portrait area in foreign affairs documents can be determined. The facial symmetry line is obtained through the symmetry of the face. The portrait area is divided according to the facial symmetry line, and the matching degree of pixel pairs on both sides of the portrait is calculated. The image blur is analyzed based on the matching degree. That is, if symmetrical pixels can be matched, it means that the information on both sides of the portrait is relatively reliable. Otherwise, it means that there are relatively blurry areas, and analysis can be carried out based on these blurry areas. Specifically, in this embodiment, semantic segmentation is used to symmetrically divide the portrait region into two parts based on the facial symmetry line. Then, based on the symmetry of the portrait, the symmetry of the two parts after segmentation is calculated. When performing the analysis, the facial features are mainly considered, while other features such as hair may interfere with the symmetry. Therefore, the existing solvePnP algorithm is used on the portrait image to obtain 68 key points of the portrait region, and the obtained image key points are matched. The matching algorithm in this embodiment is the Hungarian matching algorithm to obtain the corresponding matching pixels on both sides of the portrait image. The matching algorithm is a well-known existing technology, and this application will not elaborate on it. In addition, it should be noted that the key points of the face are extracted by using the dlib 68-point face landmark detector. The dlib 68-point face landmark detector is a standardized facial landmark detection model trained by machine learning. This model uses 68 key points to represent different features of the face.

[0039] Based on the above analysis, the closer the difference between the matching pixel and the facial symmetry line of the portrait area, the more the portrait matching conforms to the portrait symmetry. If the obtained matching result deviates further from the midline, it indicates that there may be a certain area where the portrait recognition is inaccurate due to lighting or other reasons. Therefore, the lower the matching accuracy, the greater the ambiguity of the portrait image area in that area may be, and the more necessary it is to perform image enhancement on each image block containing the pixel to increase image details.

[0040] However, considering that although the human image has symmetry, there may be a slight deviation during the ID photo shooting process, causing the center of symmetry of key point pairs in the actual image to deviate from the image center, this application analyzes the deviation of each point pair from the center of the human image and calculates the human image symmetry deviation degree. The deviation of each point pair based on the symmetry deviation degree may be due to inaccurate key point recognition caused by external factors. The calculation method of human image symmetry deviation degree is as follows: First, the degree of image deviation of the matched keypoint pairs is calculated. The difference in distance from the two matched pixels to the midline is used. In this embodiment, the distance from the left pixel to the facial symmetry line is subtracted from the distance from the corresponding right pixel to the facial symmetry line. This difference is recorded as the midline deviation of the matched point pair. In other embodiments, the distance from the right pixel to the facial symmetry line can be subtracted from the distance from the corresponding left pixel to the facial symmetry line. For matched point pairs that are not disturbed, the midline deviation of the matched point pairs should be similar. However, for matched point pairs that are disturbed by external factors and whose keypoint extraction is inaccurate, the midline deviation should be significantly different from that of other matched point pairs. Therefore, the midline deviation of all matched point pairs can be obtained, and the average of the midline deviations of all matched point pairs can be used as the image symmetry deviation.

[0041] The pixel region blur of each matching point pair is obtained by subtracting the portrait symmetry deviation from the midline deviation and then normalizing the result. This involves matching pixels on both sides of the portrait. The greater the difference in distance between the matching pixel pair and the midline compared to other point pairs, the worse the matching result, indicating a higher likelihood of external factors causing the blur. For other portrait pixels in the image, the region blur corresponding to the feature pixel closest to its coordinates is selected as its own region blur.

[0042] Furthermore, analyzing the local contrast of the image reveals that when the blurriness of a pixel area is greater, the corresponding image block also has a higher contrast. This indicates that the blurriness may be caused by lighting factors, thus requiring a reduction in the cropping threshold for that area to limit the number of pixels at each gray level, thereby reducing image contrast. Conversely, if the contrast of the image block in that area is lower, it suggests that the blurriness may be due to factors such as document wear and tear. In this case, it is necessary to increase the cropping threshold for that area to increase the number of pixels at each gray level, thereby increasing image contrast and making the portrait clearer.

[0043] Therefore, for each image block, the standard deviation of the gray values ​​of all pixels in each image block is calculated to measure the dispersion of gray values. In other embodiments, the dispersion can also be calculated using variance and coefficient of variation. The standard deviations obtained from all image blocks are normalized using the minimum-maximum normalization method. After subtracting the normalized result from 0.5, it is positively fused with the average regional blur of the portrait pixels contained in the image block, and recorded as the local enhancement factor of each image block. In this embodiment, the positive fusion is performed by multiplication. The standard deviation is normalized... After normalization, the value range is [0,1]. Compared with the intermediate level of 0.5, when the contrast of the image patch is high, that is, greater than 0.5, it usually means that the blur may be caused by excessive external light. In this case, the difference between 0.5 and the normalization result is negative. At this time, the cropping threshold should be appropriately reduced to suppress the contrast. Conversely, if the contrast of the image patch is low, that is, less than 0.5, it means that the blur may be caused by wear or insufficient light. In this case, the difference between 0.5 and the normalization result is positive. At this time, the cropping threshold needs to be increased to improve the image contrast.

[0044] The fourth step: Based on the ratio of the number of pixels in the text recognition area and the portrait area in each image block, combined with the overall distribution of the text rate of the fitted rectangle in each image block, the text adhesion degree, and the local enhancement factor, the image gain requirement of each image block is determined, and the contrast cropping threshold of the image enhancement algorithm is corrected to enhance the foreign affairs certificate image.

[0045] Based on the text adhesion and region blur of each image patch in the image, the image gain requirement for each image patch is calculated. The specific calculation method is as follows: In the formula, This represents the image gain requirement of the q-th image block, with a value range of (-1, 1); This represents the number of pixels in the text recognition region contained in the q-th image block; This represents the number of human portrait pixels contained in the q-th image block. This represents the average text rate of the fitted rectangle contained in the q-th pixel block. This represents the local text adhesion degree of the q-th image patch. This represents the local enhancement factor calculated based on the portrait pixels contained in the q-th pixel block, with a value range of (-1, 1). For pixel blocks that do not contain text information or portrait pixels, the image gain requirement is recorded as 0; Recorded as the first percentage, This is recorded as the second percentage.

[0046] The flowchart for obtaining the image gain requirement is as follows: Figure 2 As shown.

[0047] The corresponding contrast clipping threshold correction method is as follows: In the formula, This represents the contrast cropping threshold correction value corresponding to the q-th image patch in the image. The default value for the cliplimit algorithm is q. This represents the image gain requirement of the q-th image block.

[0048] Furthermore, considering that different contrast cropping thresholds may be used for image blocks in each image, which could lead to unevenness in the enhanced image, this application performs a secondary correction based on the contrast cropping threshold correction value corresponding to the neighboring pixel blocks to avoid the above situation. The specific method is as follows: calculate the average of the contrast cropping threshold correction values ​​of each image block and all image blocks in its neighborhood to obtain the final correction value of the contrast cropping threshold; in this embodiment, the neighborhood of the image block is... Neighborhood; in other embodiments, the neighborhood may also be a four-neighborhood.

[0049] This method is used to obtain the contrast cropping threshold correction value corresponding to each pixel block in the image, and this value is used to complete the image enhancement.

[0050] The fifth step: Recognize the text and human figures in the enhanced foreign affairs document images.

[0051] Based on the enhanced image of the foreign affairs document, OCR technology is used to extract the text information of the foreign affairs document. This application recognizes the foreign affairs document by combining the text information and image information.

[0052] The structured fields extracted by OCR are compared with the corresponding text in the system records. The matching rate of the corresponding text is calculated using the edit distance. The negative correlation mapping results of all minimum edit distances obtained from the foreign affairs certificate images are normalized by using the minimum edit distance between each structured field in the text recognition area and the database text. The average of the normalized values ​​of all texts is used as the text field matching rate. In this embodiment, the normalization method adopts the maximum-minimum value normalization method. The negative correlation mapping of the minimum edit distance is specifically the reciprocal of the minimum edit distance. It should be noted that in order to prevent the denominator from being 0 when calculating the reciprocal, a preset positive number with a value of 0.01 is added to the denominator entropy in this embodiment.

[0053] Simultaneously, the portrait image is compared with the portrait images in the system using existing recognition algorithms (such as the Fisherfaces neural network recognition algorithm) to obtain the portrait matching degree.

[0054] Key identity information is obtained through OCR text recognition, and the enhanced portrait is compared with the portrait in the database: the average of the text field matching rate and the portrait matching degree is used as the foreign affairs certificate recognition index. When the foreign affairs certificate recognition index is greater than a preset threshold, the content of the foreign affairs certificate is judged to be accurate; otherwise, the content of the foreign affairs certificate is judged to be inaccurate. In this embodiment, the preset threshold is determined by the average of the foreign affairs certificate recognition index of historically manually annotated foreign affairs certificates.

[0055] Based on the same inventive concept as the above methods, this application also provides a foreign affairs document recognition system based on multimodal fusion, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any one of the above-described foreign affairs document recognition methods based on multimodal fusion.

[0056] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description; sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0057] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from its essential characteristics. Therefore, the embodiments described above should be considered exemplary and non-limiting in all respects; modifications to the technical solutions described in the foregoing embodiments, or equivalent substitutions of some technical features, without causing the essence of the corresponding technical solutions to deviate from the scope of the technical solutions in the embodiments of this application, should all be included within the protection scope of this application.

Claims

1. A method for recognizing foreign affairs documents based on multimodal fusion, characterized in that, The method includes the following steps: The process involves acquiring images of diplomatic credentials, using semantic segmentation to obtain the text recognition region and the human image region, and then uniformly segmenting the diplomatic credentials image to obtain several image blocks. Edge detection and rectangle fitting are performed on the text recognition region to obtain each fitted rectangle; the text rate of each fitted rectangle is determined based on the width distribution characteristics of each fitted rectangle; and the text adhesion degree of each image block is determined based on the length distribution of all fitted rectangles in each image block. By analyzing the degree of symmetry of the human image region and combining it with the degree of dispersion of the gray values ​​of pixels in each image block, the local enhancement factor of each image block is determined. Based on the ratio of the number of pixels in the text recognition region and the portrait region in each image block, combined with the overall distribution of the text rate of the fitted rectangle in each image block, the text adhesion degree, and the local enhancement factor, the image gain requirement of each image block is determined, and the contrast cropping threshold of the image enhancement algorithm is corrected to enhance the foreign affairs certificate image. Recognize the text content and human image content in enhanced diplomatic document images.

2. The method for recognizing foreign affairs documents based on multimodal fusion as described in claim 1, characterized in that, The determination of the text rate for each fitted rectangle is specifically as follows: For each image patch, the average width of all fitted rectangles is obtained, and the negative correlation mapping result of the difference between the width of each fitted rectangle and the average width is calculated. After normalization, the text rate of each fitted rectangle is obtained.

3. The method for recognizing foreign affairs documents based on multimodal fusion as described in claim 1, characterized in that, The process of determining the text adhesion degree of each image block is as follows: normalize the length of the fitted rectangle of all edges contained in each image block and then take the average value.

4. The method for recognizing foreign affairs documents based on multimodal fusion as described in claim 1, characterized in that, The step of determining the local enhancement factor for each image patch is as follows: The human portrait region is segmented based on the facial symmetry line, and the pixels of the two segments are matched. Analyze the distribution characteristics of the distance difference between each matching point pair and the facial symmetry line of the human image region to determine the regional blur of each matching point pair; The difference between the natural number 0.5 and the normalized value of the discreteness of each image block is obtained, and positively fused with the average regional blur of all portrait regions within the image block to obtain the local enhancement factor of each image block.

5. The method for recognizing foreign affairs documents based on multimodal fusion as described in claim 4, characterized in that, The process of determining the regional ambiguity of each matching point pair is as follows: The difference in distance from the two matched pixels to the midline is recorded as the midline deviation of the matched point pair; the mean of the midline deviations of all matched point pairs is taken as the portrait symmetry deviation. Subtract the human image symmetry deviation from the midline deviation of each matching point pair and normalize the result to obtain the region blur of each matching point pair.

6. The method for recognizing foreign affairs documents based on multimodal fusion as described in claim 1, characterized in that, The determination of the image gain requirement for each image block is specifically as follows: For each image patch, calculate the percentage of pixels in the human face region, denoted as the first percentage; calculate the percentage of pixels in the text recognition region, denoted as the second percentage; The first proportion is used as the weight of the local enhancement factor for each image patch; Calculate the product of the average text rate of all fitted rectangles in each image block and the text adhesion degree. Use the second proportion as the weight of the obtained product and perform a weighted summation of the local enhancement factor of each image block and the obtained product to obtain the image gain requirement of each image block.

7. The method for recognizing foreign affairs documents based on multimodal fusion as described in claim 1, characterized in that, The process of correcting the contrast cropping threshold of the image enhancement algorithm is as follows: Calculate the sum of the natural number 1 and the image gain requirement, and then multiply it by the preset contrast clipping threshold to obtain the contrast clipping threshold correction value for each image block. The final contrast cropping threshold correction value is obtained by averaging the contrast cropping threshold correction values ​​of each image patch and all its neighboring image patches.

8. The method for recognizing foreign affairs documents based on multimodal fusion as described in claim 1, characterized in that, The process of recognizing the text content and facial image content in the enhanced foreign affairs document image is as follows: Obtain the database of foreign affairs certificates; extract the text recognition region from the enhanced foreign affairs certificate image, and obtain the text field matching rate between the structured fields in the text recognition region and the text in the database; The facial image region in the enhanced foreign affairs certificate image is matched with the facial images in the database to obtain the facial matching degree; The average of the text field matching rate and the image matching degree is used as the foreign affairs certificate recognition index. When the foreign affairs certificate recognition index is greater than a preset threshold, the content of the foreign affairs certificate is judged to be accurate; otherwise, the content of the foreign affairs certificate is judged to be inaccurate.

9. The method for recognizing foreign affairs documents based on multimodal fusion as described in claim 8, characterized in that, The specific method for obtaining the text field matching rate is as follows: The average of the negative correlation mapping results of all minimum edit distances obtained from the foreign affairs certificate image is calculated by taking the minimum edit distance between each structured field in the text recognition region and the database text, and using this average as the text field matching rate.

10. A foreign affairs document recognition system based on multimodal fusion, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-9.

Images