Picture content recognition method and apparatus

Abstract

A method and apparatus for image content recognition are disclosed. The method includes: acquiring an image to be recognized; acquiring a first reference region of the image to be recognized based on the image to be recognized and a template image, wherein the first reference region corresponds to a second reference region pre-marked in the template image; calculating a first perspective transformation matrix; correcting the image to be recognized to obtain a first image; acquiring a second reference region in the first image based on the first reference region pre-marked in the template image; recognizing the content of the second reference region as the recognition output of the image to be recognized. This invention achieves template matching by identifying a reference region in the image to be recognized that matches a template image, performing an affine transformation based on the two reference regions, and finally determining the reference region in the image to be recognized based on the reference region marked in the template image, thereby accurately acquiring the content to be recognized in the image.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to an image content recognition method and apparatus. Background Technology

[0002] Tables, as a visual communication format and a common method for organizing and structuring data, are widely found in various documents such as newspapers, magazines, scientific literature, and financial statements, and are a common page element in all types of documents. As an important supplement and summary of textual descriptions and a vital carrier of data, tables have been extensively used across various industries.

[0003] With the development of internet technology and the advent of the big data era, tabular data is experiencing explosive growth. Manually processing tabular data presents problems such as numerous errors, high time costs, and significant difficulty. Therefore, efficiently extracting useful information from massive amounts of data is a crucial technology urgently needed across all industries to significantly improve office efficiency. Especially for image-based tables, how to efficiently and accurately identify table content is a pressing issue that needs to be addressed.

[0004] The information disclosed in this background section is intended only to enhance the understanding of the general background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] To address the problems existing in the prior art, the present invention provides an image content recognition method and apparatus.

[0006] The present invention provides an image content recognition method, the method comprising:

[0007] Obtain the image to be recognized;

[0008] Based on the image to be identified and the template image, a first reference region of the image to be identified is obtained, and the first reference region corresponds to a second reference region pre-marked in the template image.

[0009] The first perspective transformation matrix is ​​calculated based on the first coordinates of the first reference region and the second coordinates of the second reference region;

[0010] The image to be identified is corrected based on the first perspective transformation matrix to obtain the first image;

[0011] Based on the first region to be identified pre-marked on the template image, a second region to be identified in the first image is obtained, and the content of the second region to be identified is identified as the recognition output of the image to be identified.

[0012] Optionally, obtaining the image to be identified further includes:

[0013] Obtain the first original image;

[0014] Perform table line detection on the first original image to obtain the set of coordinates of the first corner point of the first original image;

[0015] Based on the first set of corner point coordinates and the second set of corner point coordinates of the template image, the second perspective transformation matrix is ​​calculated;

[0016] The first original image is corrected based on the second perspective transformation matrix to obtain a second image, which is used as the image to be identified.

[0017] Optionally, performing table line detection on the first original image to obtain the set of coordinates of the first corner points of the first original image further includes:

[0018] The first convolution kernel is used to perform erosion and dilation operations on the first original image to obtain the table lines;

[0019] The first original image is subjected to erosion and dilation operations using the second convolution kernel to obtain the table's vertical lines.

[0020] By performing a bitwise AND operation on the horizontal and vertical lines of the table, the coordinate set of the first corner point of the first original image is obtained.

[0021] Optionally, obtaining the first original image further includes:

[0022] Obtain the second original image;

[0023] The second original image is converted to grayscale and then binarized to obtain the first original image.

[0024] Optionally, obtaining a first reference region of the image to be identified based on the image to be identified and the template image further includes:

[0025] Input the image to be recognized into the structure recognition model, and obtain the confidence score output by the structure recognition model. The confidence score represents the probability that several text boxes in the image to be recognized correspond to the second reference region.

[0026] If the confidence level is greater than the first threshold, the combined distance between the plurality of text boxes and the second reference area is calculated as the first distance to be verified; if the confidence level is less than or equal to the first threshold, the combination of text boxes with the smallest combined distance to the second reference area is found among all combinations of the plurality of text boxes, and the smallest combined distance is taken as the second distance to be verified.

[0027] If the first distance or the second distance is greater than a preset second threshold, the verification fails, and a different template image is selected again; if the first distance or the second distance is less than or equal to the preset second threshold, the verification succeeds, and the plurality of text boxes or the combination of the text boxes is used as the first reference region output by the structure recognition model.

[0028] Optionally, the comprehensive distance includes at least one of the following: text similarity distance, line segment ratio distance, and vector angle distance.

[0029] Optionally, inputting the image to be recognized into the structure recognition model and obtaining the confidence level output by the structure recognition model further includes:

[0030] Input the image to be recognized into the optical character recognition unit in the structure recognition model, and obtain the text content and text box coordinates output by the optical character recognition unit;

[0031] Based on the coordinates of the text box, determine the region of interest on the image to be recognized;

[0032] Input the region of interest into the image feature extraction unit in the structure recognition model to obtain the feature map output by the image feature extraction unit;

[0033] Input the text content, the text box coordinates, and the feature map into the language representation unit in the structure recognition model, and obtain the confidence score output by the language representation unit.

[0034] Optionally, the second reference area includes multiple sub-reference areas;

[0035] Correspondingly, the calculation of the comprehensive distance includes:

[0036] In the plurality of text boxes, the nearest text box is matched for each of the sub-reference regions to form multiple matching pairs;

[0037] Calculate the sub-comprehensive distance for each of the matching pairs, and sum all the sub-comprehensive distances by weight to obtain the comprehensive distance.

[0038] Optionally, calculating the first perspective transformation matrix based on the first coordinates of the first reference region and the second coordinates of the second reference region further includes:

[0039] Based on the first coordinate and the second coordinate, the first perspective transformation matrix is ​​calculated using the least squares method.

[0040] The present invention also provides an image content recognition device, the device comprising:

[0041] The acquisition module is used to acquire the image to be recognized;

[0042] The structure recognition module is used to obtain a first reference region of the image to be recognized based on the image to be recognized and the template image, wherein the first reference region corresponds to a second reference region pre-marked in the template image;

[0043] The transformation calculation module is used to calculate the first perspective transformation matrix based on the first coordinates of the first reference region and the second coordinates of the second reference region;

[0044] The correction module is used to correct the image to be identified based on the first perspective transformation matrix to obtain the first image;

[0045] The recognition module is used to obtain a second region to be recognized in the first image based on a first region to be recognized pre-marked on the template image, and to recognize the content of the second region to be recognized as the recognition output of the image to be recognized.

[0046] The image content recognition method and apparatus provided by the present invention identify a reference region in the image to be recognized that matches a template image, form a template match, perform an affine transformation based on the two reference regions, and finally determine the region to be recognized in the image to be recognized based on the region to be recognized marked in the template image, so as to accurately obtain the content to be recognized in the image. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 This is a schematic flowchart of an image content recognition method provided in an embodiment of the present invention;

[0049] Figure 2 A flowchart for extracting table corners is provided in an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of a template image provided in an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of a process for obtaining the content of a region to be identified, provided as an embodiment of the present invention.

[0052] Figure 5 A schematic diagram of a reference area provided for an embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram of a structure recognition model disclosed in an embodiment of the present invention;

[0054] Figure 7 This is a schematic diagram of the structure of an image content recognition device provided in an embodiment of the present invention;

[0055] Figure 8 This is a schematic diagram of the physical structure of an electronic device provided by the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0057] The image content recognition method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0058] Figure 1 This is a schematic diagram of an image content recognition method provided in an embodiment of the present invention, such as... Figure 1 As shown, the technical solution of the present invention provides an image content recognition method, which includes the following steps.

[0059] S100: Obtain the image to be recognized.

[0060] In one embodiment of the present invention, the image to be identified includes a table, which includes both a pure table completely enclosed by table lines and tabular text whose main body is in table form but whose table lines are partially or completely hidden.

[0061] Optionally, obtaining the image to be identified further includes: obtaining a first original image; performing table line detection on the first original image to obtain a set of first corner coordinates of the first original image; calculating a second perspective transformation matrix based on the first corner coordinate set and the second corner coordinate set of the template image; and correcting the first original image based on the second perspective transformation matrix to obtain a second image, which is used as the image to be identified.

[0062] Optionally, performing table line detection on the first original image to obtain the first corner point coordinate set of the first original image further includes: performing erosion and dilation operations on the first original image through a first convolution kernel to obtain table horizontal lines; performing erosion and dilation operations on the first original image through a second convolution kernel to obtain table vertical lines; and performing a bitwise AND operation on the table horizontal lines and table vertical lines to obtain the first corner point coordinate set of the first original image.

[0063] Preferably, Figure 2 A flowchart for extracting table corners is provided as an embodiment of the present invention, such as... Figure 2 As shown, the first convolution kernel size is set to (w, 1). The image is first eroded, then dilated to obtain the horizontal lines of the table, where w is the width of the input image. The second convolution kernel size is set to (1, h). The image is first eroded, then dilated to obtain the vertical lines of the table. The intersection points of the obtained vertical and horizontal lines are obtained by bitwise AND operation. The top left, top right, bottom left, and bottom right points are taken as the table corner points. Similarly, the four table corner points of the template image can be obtained using the same method. A second perspective transformation matrix of size 3x3 can be calculated. Based on the obtained second perspective transformation matrix, the corrected image to be recognized can be obtained. This transformation can correct the tilted or angularly offset images to be recognized, improving the recognition accuracy of the general subsequent optical character recognition unit.

[0064] Optionally, obtaining the first original image further includes: obtaining a second original image; and converting the second original image to grayscale and binarization to obtain the first original image.

[0065] In one embodiment of the present invention, multiple template images are provided to match various types of documents to be identified. When preparing the template image, an image that is properly positioned, has clear text, is no larger than 4M, has a longest side of no more than 4096 pixels, and is in JPG, PNG, or JPEG format is selected as the template image. A reference area and a region to be identified are selected on the template image. Figure 3 This is a schematic diagram of a template image provided in an embodiment of the present invention, such as... Figure 3 As shown, the four text boxes selected with solid lines in the template image serve as the second reference area of ​​the template image, while the three text boxes selected with dashed lines serve as the areas to be recognized. For Figure 3 The template images in the document can be used to match images to be identified under the category of "product certificate of conformity" and to identify the specific content to be filled in for the three categories: "manufacturer", "product number", and "quantity".

[0066] S200. Based on the image to be identified and the template image, obtain the first reference region of the image to be identified, which corresponds to the second reference region pre-marked in the template image.

[0067] Optionally, inputting the image to be recognized into the structure recognition model to obtain the first reference region of the image to be recognized output by the structure recognition model further includes: inputting the image to be recognized into the structure recognition model to obtain the confidence score output by the structure recognition model, where the confidence score represents the probability that several text boxes in the image to be recognized correspond to the second reference region; if the confidence score is greater than a first threshold, calculating the comprehensive distance between the several text boxes and the second reference region as the first distance to be verified; if the confidence score is less than or equal to the first threshold, traversing all combinations of several text boxes to find the combination of text boxes with the smallest comprehensive distance to the second reference region, and using the smallest comprehensive distance as the second distance to be verified; if the first distance or the second distance is greater than a preset second threshold, the verification fails, and a different template image is selected again; if the first distance or the second distance is less than or equal to the preset second threshold, the verification succeeds, and the several text boxes or combinations of text boxes are used as the first reference region output by the structure recognition model.

[0068] In one embodiment of the present invention, Figure 4 This is a schematic diagram of a process for obtaining the content of a region to be identified, provided as an embodiment of the present invention. Figure 4 As shown, the structure recognition model provides the confidence score for each text box identified as a reference region. The average confidence score of multiple text boxes in the first reference region is defined as the score, and a first threshold is set. If the score is greater than the first threshold, the current output text box position is considered to be a reference region position in the image to be recognized. If the score is less than the first threshold, the current output text box position is considered not to meet the requirements, and it is necessary to traverse all recognized text boxes in the image to be recognized and calculate the comprehensive distance.

[0069] Optionally, the overall distance shall include at least one of the following: text similarity distance, line segment ratio distance, and vector angle distance.

[0070] It should be noted that in the comprehensive distance, the text similarity distance is used to measure the similarity of text content, the line segment ratio distance is used to determine whether the position distribution of the text box is consistent with the template image, and the vector angle distance is used to calculate the vector offset angle.

[0071] Figure 5 A reference area comparison diagram provided for an embodiment of the present invention, such as... Figure 5 As shown, the area to the left of the vertical line is the reference area labeled in the template image, and the area to the right of the vertical line is all the text boxes detected in the image to be recognized. Three text boxes are selected from the image to be recognized each time, for a total of [number] iterations. Each time, the combined distance between the three text boxes and the reference area text box in the template image is calculated and recorded. After the traversal is completed, the text box with the smallest combined distance is selected as the reference area on the image to be recognized. The calculation method for each distance is as follows.

[0072] The formula for calculating text similarity distance:

[0073] edit_distance(text1, text2) / max(len(text1), len(text2))

[0074] Where edit_distance is the edit distance, len() is the function to get the text length, max() is the function to get the maximum value, text1 is the text content of the reference area in the template image, text2 is the text box content in the image to be recognized, and the edit distance is normalized by dividing the maximum text length in the two text boxes.

[0075] Formula for calculating the proportional distance of a line segment:

[0076]

[0077] Among them, each line segment is as follows Figure 5 As shown, the corner points of the text boxes in the template image are connected pairwise, and the results are subtracted from the corresponding line segments in the image to be recognized. The difference is then divided by the larger value for normalization. The maximum proportional distance of all line segments is taken as the line segment distance of the currently selected 3 text boxes.

[0078] Formula for calculating the angular distance between vectors:

[0079]

[0080] in, Represents a vector Take the mold. Similarly, calculate the angle between the vector formed by the corner points of all text boxes in the template image and the corresponding vector of the image to be recognized, and normalize the vector angle by dividing it by π. Take the largest vector angle as the distance between the vectors of the three currently selected text boxes.

[0081] By using a comprehensive distance calculation method, the accuracy of the first reference region can be further determined and the matching verification between the template image and the image to be identified can be completed. This can not only improve the accuracy of obtaining the first reference region, but also automatically determine whether the image to be identified is of the same type as the template image.

[0082] Optionally, inputting the image to be recognized into the structure recognition model and obtaining the confidence level output by the structure recognition model further includes: inputting the image to be recognized into the optical character recognition unit in the structure recognition model and obtaining the text content and text box coordinates output by the optical character recognition unit; determining the region of interest on the image to be recognized based on the text box coordinates; inputting the region of interest into the image feature extraction unit in the structure recognition model and obtaining the feature map output by the image feature extraction unit; and inputting the text content, text box coordinates, and feature map into the language representation unit in the structure recognition model and obtaining the confidence level output by the language representation unit.

[0083] In one embodiment of the present invention, Figure 6 This is a schematic diagram of a structural recognition model disclosed in an embodiment of the present invention, such as... Figure 6 As shown, the optical character recognition unit uses a general optical character recognition algorithm to detect and recognize the image to be recognized. This algorithm specifically includes: a CRNN (Convolutional Recurrent Neural Network) for text content recognition and a DBNet (Differentiable Binarization Network) for text box detection. The CRNN obtains all text content in the image to be recognized, and the DBNet obtains the coordinates of the text boxes. Further, the text content and corresponding text box coordinates of the image to be recognized are input into the structural recognition model. These two inputs are encoded into vectors, and feature transformation is further performed through embedding. The region of interest (ROI), i.e., the image of the text box region, can be extracted from the original image using the text box coordinates. The image is then processed by Faster RCNN (Regions with CNN features) to extract a high-dimensional feature map. A fully connected layer maps the feature map to a fixed-size vector form. Next, the text box coordinates, text content, and corresponding feature maps are fused together. Then, a pre-trained BERT (Bidirectional Encoder Representation from Transformers) model is used to obtain the text layout information of the entire image to be recognized, namely the category and location of each text box and its corresponding confidence score. This yields the structural information of the image to be recognized and the category of each text box, essentially identifying which text boxes in the image belong to the reference region.

[0084] Optionally, the second reference region includes multiple sub-reference regions; correspondingly, the calculation of the overall distance includes: in several text boxes, matching the nearest text box to each sub-reference region to form multiple matching pairs; calculating the sub-overall distance of each matching pair, and weighted summing of all sub-overall distances to obtain the overall distance.

[0085] S300. Based on the first coordinates of the first reference region and the second coordinates of the second reference region, calculate the first perspective transformation matrix.

[0086] Optionally, the first perspective transformation matrix is ​​calculated based on the first coordinates of the first reference region and the second coordinates of the second reference region, further including: calculating the first perspective transformation matrix using the least squares method based on the first coordinates and the second coordinates.

[0087] S400. Correct the image to be recognized based on the first perspective transformation matrix to obtain the first image. By using the obtained first perspective transformation matrix, perspective transformation can be performed on the image to be recognized to further correct it. This can reduce text border offset and improve the accuracy of text recognition in the area to be recognized.

[0088] S500: Based on the first region to be identified pre-marked on the template image, obtain the second region to be identified in the first image, identify the content of the second region to be identified, and use it as the recognition output of the image to be identified.

[0089] In one embodiment, the region to be identified in the image can be obtained based on the coordinates of the region to be identified selected on the template image. The CRNN can then be used to identify the text content in the region, and the text content can be returned to complete the entire recognition process.

[0090] The grid image content recognition device provided by the present invention is described below. The grid image content recognition device described below can be referred to in correspondence with the grid image content recognition method described above.

[0091] Figure 7 This is a schematic diagram of the structure of an image content recognition device provided in an embodiment of the present invention, as shown below. Figure 7 As shown, the technical solution of the present invention also provides an image content recognition device, the device comprising:

[0092] The acquisition module 710 is used to acquire the image to be recognized; the structure recognition module 720 is used to input the image to be recognized into the structure recognition model to obtain the first reference region of the image to be recognized output by the structure recognition model, the first reference region corresponding to the second reference region pre-marked in the template image; the transformation calculation module 730 is used to calculate the first perspective transformation matrix based on the first coordinate of the first reference region and the second coordinate of the second reference region; the correction module 740 is used to correct the image to be recognized based on the first perspective transformation matrix to obtain the first image; the recognition module 750 is used to obtain the second region to be recognized in the first image based on the first region to be recognized pre-marked on the template image, recognize the content of the second region to be recognized, and use it as the recognition output of the image to be recognized.

[0093] This embodiment identifies a reference region in the image to be identified that matches the template image, forms a template match, performs an affine transformation based on the two reference regions, and finally determines the region to be identified in the image based on the region to be identified marked in the template image, so as to accurately obtain the content to be identified in the image.

[0094] Figure 8 A schematic diagram of the physical structure of an electronic device provided by the present invention, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute an image content recognition method, the method including:

[0095] Obtain an image to be recognized; based on the image to be recognized and a template image, obtain a first reference region of the image to be recognized, the first reference region corresponding to a second reference region pre-marked in the template image; calculate a first perspective transformation matrix based on the first coordinates of the first reference region and the second coordinates of the second reference region; correct the image to be recognized based on the first perspective transformation matrix to obtain a first image; obtain a second region to be recognized in the first image based on the first region to be recognized pre-marked in the template image, and identify the content of the second region to be recognized as the recognition output of the image to be recognized.

[0096] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the image content recognition method provided by the above methods, the method comprising:

[0098] Obtain an image to be recognized; based on the image to be recognized and a template image, obtain a first reference region of the image to be recognized, the first reference region corresponding to a second reference region pre-marked in the template image; calculate a first perspective transformation matrix based on the first coordinates of the first reference region and the second coordinates of the second reference region; correct the image to be recognized based on the first perspective transformation matrix to obtain a first image; obtain a second region to be recognized in the first image based on the first region to be recognized pre-marked in the template image, and identify the content of the second region to be recognized as the recognition output of the image to be recognized.

[0099] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the image content recognition methods provided above, the methods comprising:

[0100] Obtain an image to be recognized; based on the image to be recognized and a template image, obtain a first reference region of the image to be recognized, the first reference region corresponding to a second reference region pre-marked in the template image; calculate a first perspective transformation matrix based on the first coordinates of the first reference region and the second coordinates of the second reference region; correct the image to be recognized based on the first perspective transformation matrix to obtain a first image; obtain a second region to be recognized in the first image based on the first region to be recognized pre-marked in the template image, and identify the content of the second region to be recognized as the recognition output of the image to be recognized.

[0101] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for image content recognition, characterized in that, The method includes: Obtain the image to be recognized; Based on the image to be identified and the template image, a first reference region of the image to be identified is obtained, and the first reference region corresponds to a second reference region pre-marked in the template image. The first perspective transformation matrix is ​​calculated based on the first coordinates of the first reference region and the second coordinates of the second reference region; The image to be identified is corrected based on the first perspective transformation matrix to obtain the first image; Based on the first region to be identified pre-marked on the template image, the second region to be identified in the first image is obtained, and the content of the second region to be identified is identified as the recognition output of the image to be identified. Based on the image to be identified and the template image, obtaining a first reference region of the image to be identified further includes: Input the image to be recognized into the structure recognition model and obtain the confidence score output by the structure recognition model. The confidence score represents the probability that several text boxes in the image to be recognized correspond to the second reference region. If the confidence level is greater than the first threshold, the combined distance between the plurality of text boxes and the second reference area is calculated as the first distance to be verified; if the confidence level is less than or equal to the first threshold, the combination of text boxes with the smallest combined distance to the second reference area is found among all combinations of the plurality of text boxes, and the smallest combined distance is taken as the second distance to be verified. If the first distance or the second distance is greater than a preset second threshold, the verification fails, and a different template image is selected again; if the first distance or the second distance is less than or equal to the preset second threshold, the verification succeeds, and the plurality of text boxes or the combination of the text boxes is used as the first reference region output by the structure recognition model.

2. The image content recognition method according to claim 1, characterized in that, Obtain the image to be identified, further including: Obtain the first original image; Perform table line detection on the first original image to obtain the set of coordinates of the first corner point of the first original image; Based on the first set of corner point coordinates and the second set of corner point coordinates of the template image, the second perspective transformation matrix is ​​calculated; The first original image is corrected based on the second perspective transformation matrix to obtain a second image, which is used as the image to be identified.

3. The image content recognition method according to claim 2, characterized in that, Perform table line detection on the first original image to obtain the set of coordinates of the first corner points of the first original image, further including: The first convolution kernel is used to perform erosion and dilation operations on the first original image to obtain the table lines; The first original image is subjected to erosion and dilation operations using the second convolution kernel to obtain the table's vertical lines. By performing a bitwise AND operation on the horizontal and vertical lines of the table, the coordinate set of the first corner point of the first original image is obtained.

4. The image content recognition method according to claim 2, characterized in that, Obtain the first original image, further including: Obtain the second original image; The second original image is converted to grayscale and then binarized to obtain the first original image.

5. The image content recognition method according to claim 1, characterized in that, The comprehensive distance includes at least one of the following: text similarity distance, line segment ratio distance, and vector angle distance.

6. The image content recognition method according to claim 1, characterized in that, Inputting the image to be recognized into the structure recognition model and obtaining the confidence score output by the structure recognition model further includes: Input the image to be recognized into the optical character recognition unit in the structure recognition model, and obtain the text content and text box coordinates output by the optical character recognition unit; Based on the coordinates of the text box, determine the region of interest on the image to be recognized; Input the region of interest into the image feature extraction unit in the structure recognition model to obtain the feature map output by the image feature extraction unit; Input the text content, the text box coordinates, and the feature map into the language representation unit in the structure recognition model, and obtain the confidence score output by the language representation unit.

7. The image content recognition method according to claim 1, characterized in that, The second reference region includes multiple sub-reference regions; Correspondingly, the calculation of the comprehensive distance includes: In the plurality of text boxes, the nearest text box is matched for each of the sub-reference regions to form multiple matching pairs; Calculate the sub-comprehensive distance for each of the matching pairs, and sum all the sub-comprehensive distances by weight to obtain the comprehensive distance.

8. The image content recognition method according to claim 1, characterized in that, Based on the first coordinates of the first reference region and the second coordinates of the second reference region, a first perspective transformation matrix is ​​calculated, further including: Based on the first coordinate and the second coordinate, the first perspective transformation matrix is ​​calculated using the least squares method.

9. An image content recognition device, characterized in that, The device includes: The acquisition module is used to acquire the image to be recognized; The structure recognition module is used to obtain a first reference region of the image to be recognized based on the image to be recognized and the template image, wherein the first reference region corresponds to a second reference region pre-marked in the template image; The transformation calculation module is used to calculate the first perspective transformation matrix based on the first coordinates of the first reference region and the second coordinates of the second reference region; The correction module is used to correct the image to be identified based on the first perspective transformation matrix to obtain the first image; The recognition module is used to obtain a second region to be recognized in the first image based on a first region to be recognized pre-marked on the template image, and to recognize the content of the second region to be recognized as the recognition output of the image to be recognized; Based on the image to be identified and the template image, obtaining a first reference region of the image to be identified further includes: Input the image to be recognized into the structure recognition model and obtain the confidence score output by the structure recognition model. The confidence score represents the probability that several text boxes in the image to be recognized correspond to the second reference region. If the confidence level is greater than the first threshold, the combined distance between the plurality of text boxes and the second reference area is calculated as the first distance to be verified; if the confidence level is less than or equal to the first threshold, the combination of text boxes with the smallest combined distance to the second reference area is found among all combinations of the plurality of text boxes, and the smallest combined distance is taken as the second distance to be verified. If the first distance or the second distance is greater than a preset second threshold, the verification fails, and a different template image is selected again; if the first distance or the second distance is less than or equal to the preset second threshold, the verification succeeds, and the plurality of text boxes or the combination of the text boxes is used as the first reference region output by the structure recognition model.

Images