Intelligent identification self-learning method and device based on template matching, medium and equipment

Abstract

The invention relates to the technical field of intelligent identification, in particular to an intelligent identification self-learning method based on template matching, which comprises the following steps of: constructing a standard template; performing template matching on the standard template and the to-be-recognized image through a template matching classifier; performing structured processing on the to-be-recognized image according to the standard template to obtain key information extraction so as to perform information recognition on the to-be-recognized image; the key information comprises keywords and specific values. The standard template is constructed through the system, then the template is matched by using the template matching classifier, and then the to-be-recognized image and the standard template are mapped by using the structured processing mode so as to perform information recognition on the to-be-recognized image, so that offline manual input processing of paper materials is greatly reduced, and the handling efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent recognition technology, and in particular to an intelligent recognition self-learning method, device, medium and equipment based on template matching. Background Art

[0002] In the current information age, the rapid identification and structured processing of business materials has become a core requirement for governments and enterprises to improve office efficiency and service quality. Traditional business material identification and structuring mostly rely on manual review and input, which is not only time-consuming and labor-intensive, but also prone to errors, seriously affecting work efficiency and service quality.

[0003] In the field of human resources and social security, the processing of business materials also faces similar problems. For example, some early document management systems can only realize basic scanning and archiving functions, lack intelligent processing capabilities, and cannot automatically identify and extract key information. Other applications that attempt to introduce primary OCR technology can convert text in images, but when faced with complex and changing business materials of different types and formats and handwriting, the recognition accuracy and adaptability are still limited.

[0004] In recent years, with the development of intelligent image recognition technology, most document image recognition and understanding mainly use keyword matching methods to manually construct rules to extract key information. Based on the keyword matching method, key information in the document is extracted by manually constructing rules (such as regular expressions). However, this method requires reconstructing rules for different types of documents, and its universality is poor; once the keyword recognition is wrong, the matching rules are easily invalid; in addition, this method is also difficult to obtain accurate results when processing tilted documents or documents with perspective transformation interference.

[0005] In summary, although the existing technical solutions have solved the problems in business material identification and structuring to a certain extent, they still have the following obvious shortcomings: 1. Lack of flexibility: When faced with new types of business materials, the system often requires professional technicians to reprogram or adjust it, and the response to the newly added business material formats is not quick and flexible enough.

[0006] 2. High integration difficulty: The compatibility and interface standardization between different systems are not high, which makes it difficult to seamlessly integrate the overall solution into the existing business processes. Summary of the invention

[0007] In order to solve the deficiencies in the prior art, the present invention provides an intelligent recognition self-learning method based on template matching, comprising the following steps: S100, build standard template; S200, performing template matching between the standard template and the image to be identified through a template matching classifier; S300, structurally processing the image to be identified according to the standard template to obtain key information extraction to identify the image to be identified; the key information includes keywords and specific values.

[0008] The present invention also provides an intelligent recognition self-learning device based on template matching, comprising: Template building module, used to build standard templates; A template matching module is used to perform template matching between a standard template and an image to be identified through a template matching classifier; The result recognition module is used to perform structured processing on the image to be recognized according to the standard template to obtain key information extraction so as to perform information recognition on the image to be recognized; the key information includes keywords and specific values ​​corresponding to the keywords.

[0009] The present invention also provides a computer-readable storage medium, which stores computer instructions. When the computer is executed by a processor, the computer implements the above-mentioned intelligent recognition self-learning method based on template matching.

[0010] An electronic device provided by the present invention includes at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by at least one processor, and the instructions are executed by at least one processor so that the processor executes the intelligent recognition self-learning method based on template matching as described above.

[0011] Compared with the prior art, the intelligent recognition self-learning method based on template matching provided by the present invention can construct a standard template through the system if the system does not have a corresponding standard template, and then use the template matching classifier to match the template, and then use a structured processing method to map the image to be recognized and the standard template to perform information recognition on the image to be recognized, thereby greatly reducing the offline manual entry processing of paper materials and improving the efficiency of handling matters by the handling agencies.

[0012] The collection, use, storage, disclosure, and organization of personal information in the technical solution of this application comply with the provisions or requirements of relevant national laws and regulations, and do not violate public order and good customs, social morality, or harm the public interest. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1A flowchart of the intelligent recognition self-learning method based on template matching provided in the first embodiment of the present invention; Figure 2 A flowchart of a template construction process provided in the first embodiment of the present invention; Figure 3 A flow chart of template matching provided in the first embodiment of the present invention; Figure 4 A flow chart of a pure image feature classifier provided in Embodiment 1 of the present invention; Figure 5 A flow chart of a pure text feature classifier provided in Embodiment 1 of the present invention; Figure 6 A flowchart of key information extraction provided in the first embodiment of the present invention; Figure 7 A flowchart of the structured processing provided in the first embodiment of the present invention; Figure 8 A flowchart of calculating the intersection-over-union ratio provided in the first embodiment of the present invention; Fig. 9 A schematic diagram of selecting reference fields for a standard ID card template provided in the second embodiment of the present invention; Fig.10 A schematic diagram of a frame selection identification area of ​​a standard ID card template provided in the second embodiment of the present invention; Fig.11 A schematic diagram of an image to be recognized provided in Embodiment 2 of the present invention; Fig.12 A flowchart of the intelligent recognition self-learning platform based on template matching provided in the third embodiment of the present invention; Fig.13 A flowchart of a template matching intelligent recognition self-learning device provided in Embodiment 4 of the present invention; Fig.14 This is a schematic diagram of the structure of an electronic device provided in Embodiment 6 of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] In the description of the present invention, it should be noted that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0017] The present invention provides an intelligent recognition self-learning method based on template matching, such as Figure 1 As shown, the following steps are included: S100, build standard template; S200, performing template matching between the standard template and the image to be identified through a template matching classifier; S300, structurally processing the image to be identified according to the standard template to obtain key information extraction to identify the image to be identified; the key information includes keywords and specific values.

[0018] In one embodiment, if Figure 2 As shown, S100 constructs a standard template including the following steps: S110, upload an image of the standard template, identify all text boxes of the standard template, extract the reference fields of the standard template, and select the identification area containing the specific value to form an identification area frame, and then save the standard template. Specifically, firstly, the user, business handling personnel or management and operation personnel use the interface system to build a customized template for the business material image of the same format that needs to be identified, wherein the standard template is a standardized document image obtained by a certain image processing of a clear business material picture containing only the foreground, and is stored in the management module.

[0019] It should be noted that the keywords of key information and the specific values ​​corresponding to the keywords are the content required to be identified in template identification, which can be set according to the needs of the information to be collected, and can be increased or decreased accordingly. For example, the "passport number / PASSPORT NO.", "name / NAME", "nationality / NATIONALITY", "place of birth / PLACE OF BIRTH" that need to be collected in the passport are keywords; and the corresponding actual passport numbers such as "DE0000000", "ZHANG,SAN", "CHINESE" and "GUANG DONG" are the specific values ​​corresponding to the above keywords.

[0020] But in fact, the passport page also includes other keywords such as gender, date of birth, issuing authority and place, date of issuance, and validity period. In addition to keywords, the passport unified template also includes non-keyword words such as "People's Republic of China" and "Diplomatic Passport". All keywords and non-keywords in the standard template are unified as reference fields. The reference field is the text contained in every sheet of the same layout material, which is distinguished from the text that changes, such as the specific value. The specific value corresponding to the keyword and other personal information are not included in the reference field. The text box includes all reference fields and specific values. Other identity cards, Hong Kong and Macau passes and other certificates and materials are also divided in this way, which will not be repeated here.

[0021] When constructing a standard template, first extract the reference fields and set the text boxes for each reference field. Then identify the identification area corresponding to the specific value to form an identification area box. Through the above operations, all text boxes of the standard template are extracted.

[0022] S120 , input a preset number of training images to train the template matching classifier.

[0023] Specifically, users only need to create multiple standard templates that have been published in the preset standard template library. For commonly used standard templates, the system has prepared image classifiers or filled in classification keyword labels to assist in classification; each type of standard template of the image classifier is trained with no less than 30 training set images of the same layout. For new templates added by users, it is only necessary to upload no less than 30 training set images of the same layout or fill in classification keyword labels for each type of new standard template to create a classifier, and realize automatic classification and structured recognition of pictures of different layouts. This design abandons the traditional manual classification method, saves the manpower and time cost of manual classification, and realizes the automatic classification and structured recognition of business material pictures in one step through intelligent means, greatly reducing manpower costs and time consumption.

[0024] At the same time, the system provides a wealth of preset standard templates for users to choose from, covering a variety of common cards and certificates: such as identity cards, social security cards, bank cards, Hong Kong and Macao passes, household registers, passports, residence permits, etc., and a variety of common business materials: such as pension application forms for pension insurance participants, labor arbitration applications, unemployment insurance benefit application forms, unemployment insurance benefit payment decisions, etc., allowing users to directly select without uploading training sets or filling in classification keywords to classify and identify images with the same layout as preset templates.

[0025] In one embodiment, if Figure 3 As shown, the specific steps of S200 are: S210 , the template matching classifier extracts and analyzes the template according to the image to be identified, and determines a standard template corresponding to the image to be identified.

[0026] Specifically, after the classifier training is completed, the following two classification modes are supported, namely, image classification mode and text detection and recognition mode on the data stream: Image classification mode: Figure 4 As shown, the classification model relies on the training of the image training set; the classifier is a common deep learning image classification model, such as MobileNetV3, ResNet50, etc., which matches the image feature values ​​of the uploaded images and classifies the same layout images into the corresponding templates. If you want to use this classifier, you must ensure that all templates have uploaded more than 30 images of the same layout training set, but no classification keywords are filled in.

[0027] The image training set refers to a set of business data images of the same format provided by the user for a specific recognition template, which is intended to train the classifier to achieve automatic classification of subsequent images. For the best classification effect, the training set should ideally contain at least 30 non-repetitive images of the same format.

[0028] Text detection and recognition mode: Figure 5 As shown, users need to carefully set and fill in classification keywords for each template in the classifier. These keywords need to be closely related to business materials and should be able to accurately reflect the unique text content in the business material pictures. The uploaded pictures are fully recognized for text. If the recognized text information is highly matched with the filled classification keywords, the classifier will automatically classify the picture into the corresponding recognition template to achieve automatic classification under high matching. In this mode, there is no need to upload training set pictures, but classification keywords must be configured for all templates that need to be automatically classified.

[0029] Among them, the classification keywords are the unique text content that users need to use as the keyword text basis for template classification based on the business material pictures of the same layout. It is necessary to ensure that the filled-in keywords appear commonly in the pictures of the layout and have a high degree of recognition, which is sufficient to uniquely determine the template category to which the picture belongs based on the filled-in keywords. For example, the five fields of "Hong Kong and Macau Pass", "Date of Birth", "Validity Period", "Issuing Authority", and "Place of Issuance" of the Hong Kong and Macau Pass appear in every Hong Kong and Macau Pass. When most of these keywords are identified in the uploaded picture, such as when 3 or 4 of the 5 fields appear, the classifier can basically accurately determine that the picture is a Hong Kong and Macau Pass, and these 5 fields can be used as the classification keywords of the Hong Kong and Macau Pass.

[0030] Preferably, the template information extraction and parsing connects to Redis to obtain standard template information, and parses the standard template information JSON for subsequent structuring. Preferably, Redis has high performance and real-time data processing capabilities, and can obtain standard template information more accurately and quickly.

[0031] S220, correcting the image to be recognized; performing rotation correction and offset correction on the image.

[0032] S230, calling the text detection and recognition model to recognize the keywords and specific values ​​of the standard template and the image to be recognized; specifically, the text detection and recognition model is a commonly used text detection and recognition model, and its detailed implementation is not fully disclosed because it belongs to the public known technology.

[0033] In one embodiment, if Figure 6 As shown, the specific steps of S300 are: S310, performing structural processing on the image to be recognized, constructing a first frame and a second frame, and mapping the standard template and the image to be recognized according to the first frame and the second frame; S320, outputting the result of the structured processing to extract key information, so as to identify and output the key information of the image to be identified.

[0034] Furthermore, if Figure 7 As shown, the specific steps of S310 are: S311, initializing parameters of the standard template and the image to be recognized. Specifically, the parameters are basic parameter information of the standard template and the image to be recognized, such as image size and image pixels.

[0035] S312, establish a coordinate axis, extract the data required for structuring in the standard template and the image to be identified, convert the data format, and convert the position information into position data containing coordinates. Establish a coordinate axis, locate all text boxes in the standard template and the image to be identified, and calculate the position data of the coordinates corresponding to each text box to facilitate subsequent calculations.

[0036] S313, anchor point positioning is performed on each reference field of the standard template and the image to be identified, and the reference field box is determined according to the anchor point, and the reference field boxes of similar lines are merged. The logic of merging similar reference field boxes is to merge similar text boxes of similar lines within a preset threshold distance (and the reference field is not identified separately before merging, and can constitute a reference field after merging). This can solve the problem of separation of some reference fields. For example, taking "name" as an example, the surname and first name in some materials are too far apart, resulting in recognition failure. It is not in line with the requirements to identify the surname and first name separately, and merging the same lines can avoid this problem to a certain extent. For example, in the work injury recognition application form, due to the large spacing between the text boxes, the text detection of "employee surname" and "first name" is sometimes divided into two text boxes. The system will first merge the entire line of text to find out whether it contains which reference fields, and the line is "employee name". Then, each text box in this row is traversed. If it is found that it does not contain "Employee Name", a row merge will be performed. The text boxes in this row are traversed one by one from left to right, and each pair of adjacent text boxes on the left and right are tried to be spliced ​​together before determining whether it contains "Employee Name". If it does, it will be spliced ​​together. Finally, the system will merge the "Employee Surname" and "First Name" into a new text box "Employee Name".

[0037] S314, positioning the first anchor point according to the reference field frame of the standard template, connecting the determined first anchor point to form a first frame of the standard template, the first frame includes all the reference field frames of the standard template; positioning the second anchor point according to the reference field frame of the image to be identified, connecting the determined second anchor point to form a second frame of the image to be identified, the second frame includes all the reference field frames of the image to be identified.

[0038] Specifically, the coordinates of the upper left corner of the text box of the reference field in the recognition result of the image to be recognized are selected as anchor points, and these anchor points are combined into a new frame called the second frame. Similarly, the frame composed of anchor points in the template is called the first frame. It should be noted that the upper right corner, lower right corner, lower left corner or center point of the text box of the reference field can also be used as anchor points. There is no limitation here, as long as the two frames use the same anchor point setting scheme, which can be selected according to the specific situation.

[0039] It should be noted that if the recognition result cannot match the reference field in the template, such as the name on the ID card is far away, resulting in the text detection box being divided into "surname" and "name", and different rows cannot match the "name" in the reference field, all recognition results will be spliced ​​in the same row and matched after splicing. If the match is successful, the first word of the reference field will be matched with each text box in the row. At this time, the position of the upper left corner of the text box is extracted as the anchor point.

[0040] S315, calculating a first boundary distance between the first frame and the standard template; S316, calculating the second boundary distance of the second frame of the image to be identified, finding the main body area of ​​the image to be identified according to the second boundary distance, cropping the image to be identified, and scaling the cropped image to be identified according to the ratio. Fig.11 As shown, the uploaded image to be identified does not only contain the main area of ​​the document to be identified, but also includes other parts. Therefore, it is necessary to use structured processing to frame and crop the image to be identified so that it only contains the main area of ​​the document to be identified, so as to facilitate scaling to the standard template for mapping and identification.

[0041] Specifically, when calculating the second boundary distance of the second frame of the image to be recognized, the similarity ratio between the first frame and the second frame is calculated first; then the second boundary distance of the second frame of the image to be recognized is calculated according to the first boundary distance and the similarity ratio.

[0042] The formula for calculating the similarity ratio is:

[0043] in, For similar proportions, is the height of the second frame, The height of the first frame. max(1, template_anchor_high) is used to prevent division by zero. If template_anchor_high is zero, 1 is used as the divisor.

[0044] The calculation formula for the second boundary distance is:

[0045] in, is the second boundary distance, is the first boundary distance.

[0046] Specifically, the cropped image to be recognized is scaled according to the ratio; the ratio includes the horizontal ratio and the vertical ratio; the calculation formula of the horizontal ratio is:

[0047] in, is the horizontal ratio, is the width of the standard template, is the maximum horizontal coordinate value of the main area in the image to be identified, is the minimum horizontal coordinate value of the main area in the image to be identified; The vertical ratio is calculated as:

[0048] in, is the horizontal ratio, is the height of the standard template, is the maximum ordinate value of the main area in the image to be identified, y is the minimum vertical coordinate value of the main area in the image to be identified.

[0049] S317, mapping the cropped and scaled image to be recognized to a standard template, and performing an intersection over union (IoU) calculation on the logo area of ​​the standard template and all text boxes of the cropped and scaled image to be recognized.

[0050] In one embodiment, if Figure 8 As shown, the specific steps of S317 are: S3171, traverse each identification area frame of the standard template; S3172, calculating the intersection and union ratio of the identification area frame of the standard template and all the text frames of the image to be recognized that are cropped and mapped to the standard template one by one; S3173. When the calculated intersection-union ratio is greater than a preset value, the text box matches successfully.

[0051] Specifically, the text box of the image to be identified, which has been cropped and mapped to the standard template, is subjected to the intersection over union (‌IoU) calculation with the identification area box in the standard template. All text boxes of the image to be identified are sequentially subjected to the intersection over union calculation with the identification area box of the standard template. After the text box of the image to be identified is mapped to the standard template, the text box of the image to be identified that overlaps with the identification area box of the standard template is likely to be the identification area box of the image to be identified, and the text box that does not overlap and has an intersection over union value of 0 is definitely not the identification area box of the image to be identified. The above operation can be used to find the identification area box of the image to be identified and determine the specific value.

[0052] A reasonable intersection over union (IoU) matching threshold, preferably IoU>0.1, is set as the standard for successful text box matching. In particular, for information displayed in multiple lines such as ID card addresses, the system intelligently merges the text content to ensure information integrity. All successfully matched text information is then integrated and output in a structured form, thus efficiently completing the conversion process from the original image to be identified to structured data.

[0053] S320: Output the result of structured processing to extract key information to perform information recognition on the image to be recognized. Fig.14 As shown, preferably, the present invention can not only provide key information in row and column output for users to view, but also provide key information output in JSON format for technical personnel in the field to view and use.

[0054] The present invention provides an intelligent recognition self-learning method based on template matching to solve at least one of the shortcomings of low efficiency and low quality of manual entry of business materials in public services in the prior art. It is particularly aimed at business needs in the human resources and social security field, such as intelligent reimbursement of work-related injury invoices, certification of qualifications for receiving benefits, and application for labor and personnel dispute arbitration, aiming to improve the efficiency and quality of business material entry.

[0055] Embodiment 2 In a specific embodiment, the present invention covers three core steps: template construction, template matching, and key information extraction. Taking ID card recognition as an example, the ID card standard template is used as the template: S100, constructing a standard template; specifically, constructing a template is divided into two stages: The first stage is to select reference fields. The interface guides users to build a customized template for the business material images of the same format that need to be identified. For example, upload a standard template of the ID card portrait, and select the reference field areas such as name, gender, ethnicity, address, etc. one by one. Fig. 9 The red text box in the ID card on the left, and record the coordinates of each text box in the standard template, Fig. 9 The right side is the recognition result output of the reference field. Among them, the standard template only contains the ID card foreground, and the standard template size is: width 640 pixels, width marked as template_w; height 415 pixels, height marked as template_h. For example, the text box of the reference field name 'address' is a rectangle, and the coordinates of its four vertices are [[37.23, 208.98], [94.49,208.98], [94.49, 242.04], [37.23, 242.04]], marked as template_box, preferably, starting from the upper left corner coordinate [37.23, 208.98], and sorted clockwise.

[0056] The coordinates of the five text box fields other than the field name 'Address' correspond to: Name [[38.41, 60.21], [93.9,60.21], [93.9, 85.59], [38.41, 85.59]]; Gender [[39.59, 110.39], [95.08, 110.39], [95.08, 132.82], [39.59,132.82]]; Citizen ID Numbers [[40.18, 337.08], [185.4, 337.08], [185.4, 379.58], [40.18, 379.58]]; Birth [[39.84, 160.93], [96.62, 160.93], [96.62, 186.83], [39.84,186.83]]; Ethnicity [[182.79, 111.13], [238.57, 111.13], [238.57, 138.02], [182.79,138.02]].

[0057] It should be noted that, in order to simplify the calculation and length, the decimals in the present invention are rounded to two decimal places using the round() function in Python, and the pixel unit of the coordinates is also omitted, and only the coordinate values ​​are recorded. In order to more clearly illustrate that the reference field is a field that must be included in each of the same materials, such as the national emblem side of the identity card, its reference field can be ["People's Republic of China", "Resident Identity Card", "Issuing Authority", "Validity Period"], and the identification area is the specific value corresponding to "Issuing Authority" and "Validity Period".

[0058] The second stage is to select the identification area to form an identification area frame, that is, to determine which areas will be used to extract the specific value corresponding to the structured field. Specifically, the identification area frame is as follows Fig.10 As shown in the green text box on the left side of the ID card, the identification area is a rectangle, and the coordinate information of the four vertices must also be recorded, such as Fig.10 As shown in the figure, the result of selecting the identification area is output. In addition, if the standard template lacks field name prompts, such as Fig.10 The blue text box is the field name of the keyword key. Users need to manually add it, such as entering the field name 'address' and assigning the variable name 'address'. The variable name is used to return the structure JSON field, and the field name is the keyword key used in the green marked area box mapped to the business material image to be identified in the key information extraction stage of S300 structured processing. Assume that Fig.11 When the position of the "Ningxia xx" text detection box fails to be completely mapped to the green box "Shanxi xx" and a large offset occurs, the blue text box will be referenced for position correction.

[0059] S200 , performing template matching between the standard template and the image to be identified through a template matching classifier.

[0060] Specifically, the user uploads the ID card image file to be identified through the interface system, such as Fig.11The ID card image to be identified is shown on the left. For ease of distinction, the ID card standard template is referred to as the "standard template" and the ID card image to be identified is referred to as the "image to be identified" in the following text. After uploading, the system background calls the classifier of the image recognition engine, and the classifier finds the standard template corresponding to the ID card in the management module through image classification. After the match is successful, the system performs text detection and text recognition operations to obtain the text content in the image, and extracts specific value segment structured information based on template matching. Before extracting key information, the system will first sort the text boxes of text recognition from top to bottom and from left to right by row. In scenes where the rows and columns are relatively clear, this is a common preprocessing technology in the OCR industry for sorting text boxes. There are many technical documents on the Internet, such as Baidu PaddleOCR's PaddleOCR, which has a set of sorting rules after recognition, which will not be repeated here.

[0061] S300, structurally processing the image to be identified according to the standard template to obtain key information extraction to identify the image to be identified; the key information includes keywords and specific values.

[0062] Specifically, after finding the corresponding ID card standard template through template matching technology, the system can map all text boxes in the image to be identified to the corresponding positions of the standard template through a series of mathematical transformations using the reference field box, and then traverse each identification area box in the standard template and calculate the intersection over union (IoU) of each mapped text box in the image to be identified. If it is greater than a preset threshold, it is considered a successful match. Then, through structured processing, all successfully matched text information is integrated and output in a structured form, thereby achieving structured extraction of key information.

[0063] Specifically, the mapping process is as follows: Anchor point positioning and box construction,First, traverse all text boxes in the image to be recognized, and find the text box whose text completely matches the reference field in the standard template, such as Fig. 9 All the red text boxes are shown, and then the number of text box matches, that is, anchor box pairs, is counted. Each pair of anchor box pairs is a reference field text box in the template that matches the text box of the image to be recognized. In the ID card of the standard template, 6 fields are expected to be matched. It is worth noting that if the number of matches is less than 6 due to text recognition errors, even so, the processing flow remains unchanged. Although the main area defined by the anchor box pairs is reduced, the similarity ratio corresponding to the main body of the ID card in the standard template and the image to be recognized remains the same.

[0064] Calculate the similarity ratio between the foreground part of the image and the template, and pass in a normal and regular image to be identified such as Fig.11 , the image size is 784 pixels in width, the width of the image to be identified is marked as image_w and 1046 pixels in height, the height of the image to be identified is marked as image_h.

[0065] On the image to be recognized, the upper left corner of the text box of the reference field in the recognition result is selected as the second anchor point. The upper left corner coordinates (x, y) of the six fields are as follows: [('Address', (135, 574)), ('Citizen ID number', (137,694)), ('Birth', (137, 527)), ('Name', (138, 434)), ('Gender', (136, 479)), ('Nationality', (268, 481))], and the second anchor points are combined into a new anchor point frame called the second frame. The width of the second frame is 133, marked as image_anchor_width. In the coordinates, the maximum value of the horizontal axis x of the second anchor point of the second frame is 268, and the minimum value is 135; the height of the second frame is 260, marked as image_anchor_high, and the maximum value of the vertical axis y of the second anchor point of the second frame is 694, and the minimum value is 434.

[0066] Similarly, the box formed by the first anchor point of the standard template is called the first box, the width of the first box is 145.56, and the width of the first box is marked as template_anchor_width, the maximum value of the horizontal axis x of the first anchor point of the first box is 182.79, and the minimum value is 37.23; the height of the first box is 276.87, and the first box is marked as template_anchor_high, the maximum value of the vertical axis y of the first anchor point of the first box is 337.08, and the minimum value is 60.21.

[0067] Finally, the similarity ratio is: scale=image_anchor_high / max(1,template_anchor_high) Substituting it into the calculation, the result is 0.94. In the present invention, the y value is used by default for calculation. When the y values ​​are too close, that is, the y values ​​are all in the same row, the width of the x value is used for calculation; The width using the x value is calculated as: scale=image_anchor_width / max(1,template_anchor_width) Boundary distance calculation: Calculate the boundary distance from the first anchor point of the upper left corner (or upper right corner, lower right corner, lower left corner or center point, etc.) of the first frame in the standard template to the document body. Boundary distance refers to positioning a point (upper left corner, upper right corner, lower right corner, lower left corner or center point, etc.) of a reference field text box as an anchor point, and calculating the upper, lower, left and right boundary distances from the anchor point to the edge of the image.

[0068] Steps: Traverse the 6 matching anchor point frame pairs of the ID card and calculate the boundary distance of each reference field on the standard template. For example, take the upper left corner of each reference field on the template as an anchor point and calculate the distance from the anchor point to the left, top, right, and bottom boundaries of the template.

[0069] Specifically, substitute the coordinates (x, y) of the upper left corner of 'address' as [37.23, 208.98], the left, upper, right, and lower boundary distances are (37.23, 208.98, 602.77, 206.02), the left boundary distance template_left_distance and the upper boundary distance template_top_distance are the coordinates (x, y) of the upper left corner, which is (37.23, 208.98).

[0070] The right boundary distance is: template_right_distance=template_w - x=640-37.23=602.77.

[0071] The lower boundary distance is: template_down_distance=template_h - y=415-208.98=206.02.

[0072] Calculate image boundaries and scale operations: The boundary distance of the image to be identified is derived by calculating the similarity ratio of the length or width of the first frame and the second frame and the ratio of the boundary distance. The document body of the image to be identified is restored and cropped using this boundary distance. At the same time, the position information in the image is scaled according to the ratio of the template.

[0073] The distance from the anchor point of each corresponding matching reference field of the image to be identified to the boundary of the image to be identified is calculated. The main area of ​​the ID card in the standard template and the image to be identified is similar, and the abstract first frame and the second frame are also similar. The similarity ratio is marked as scale.

[0074]

[0075] Since the standard template only contains the foreground of the ID card, the distance to the boundary of the main area of ​​the ID card is exactly the boundary distance to the edge of the image.

[0076] 'Address' upper left corner (x i住址 ,y i住址 ) is (135, 574), the distance from the left edge of the 'address' in the image to be identified to the main area of ​​the ID card is: image_left_distance =template_left_distance * scale =37.23*0.94 ≈35.00 Similarly, the upper, right and lower boundary distances are calculated by multiplying the upper, right and lower boundary distances calculated in the standard template by scale. Therefore, the left, upper, right and lower boundary distances of the 'address' in the image to be recognized are image_left_distance is 35.0, image_top_distance is 196.44, image_right_distance is 566.6, and image_down_distance) is 193.66.

[0077] Use this boundary distance to restore the main area of ​​the ID card in the image to be identified and then crop it: the upper left corner coordinates of the main area of ​​the ID card in the image to be identified are marked as (x0, y0) and the lower right corner coordinates are marked as (x1, y1).

[0078] Specifically, x0=x i住址 -image_left_distance; y0=y i住址 -image_top_distance; x1=x i住址 +image_right_distance; y1=y i住址 +image_down_distance; From 'address', we get (x0, y0, x1, y1), which are [100.0, 377.56, 701.6, 767.66] respectively. Similarly, we can get the other 5 fields corresponding to the main area of ​​the ID card in the image to be identified. Among them, 'citizen ID number' is [99.23, 377.14, 700.83, 767.24]); 'date of birth' is [99.55, 375.73, 701.15, 765.83]); 'name' is [101.89, 377.4, 703.49, 767.5]); 'sex' is [98.79, 375.23, 700.39,765.33]; 'ethnicity' is [96.18, 376.54, 697.78, 766.64]; There are many ways to calculate the final ID card main area, such as finding the average, sorting and taking the middle value, etc. This example uses a simple method to directly take the minimum value of x0 and y0 in the upper left corner of the main area derived from the six fields, and the maximum value of x1 and y1 in the lower right corner, so the range of the main area is as follows: (image_main_min_x, image_main_min_y, image_main_max_x, image_main_max_y) is (96.18, 375.23, 703.49, 767.66).

[0079] The main body of the document to be identified is cropped, and the position information in the image is scaled according to the ratio of the standard template. The inverse of the similarity ratio scale, x_ratio and y_ratio, need to be calculated first. Usually, the two are equal. Because of the impact of image quality such as affine transformation or perspective transformation, x_ratio and y_ratio are sometimes not exactly the same, so they are divided into two variables, respectively, from the perspective of width or height: x_ratio=template_w / (image_main_max_x-image_main_min_x) y_ratio=template_h / (image_main_max_y-image_main_min_y) Substituting in x_ratio, we get 1.05, and substituting in y_ratio, we get 1.06. For example, here, because the values ​​are rounded to two decimal places during the entire calculation process, this accumulated error ultimately makes the values ​​of x_ratio and y_ratio different.

[0080] Traverse all text boxes in the image to be recognized in turn, and scale them to the template according to the proportion. This article still takes "address" as an example. The coordinates of the four points in the text box are [[135, 574], [190, 574], [190, 600], [135,600]], and for each point coordinate (x, y) in the image to be recognized Fig.11 The upper left point of the main body of the ID card is used as the reference point for mapping. Substituting ((x-image_main_min_x)*x_ratio, (y-image_main_min_y)*y_ratio) into the four mapped points, the coordinates are [[40.76, 210.7], [98.51, 210.7], [98.51, 238.26], [40.76, 238.26]]. In the same way, all the text box position information in the image to be recognized is mapped to the template according to the proportion.

[0081] ‌IoU calculation and structured output: Traverse each identification area frame in the standard template, and calculate the intersection over union (‌IoU) of each identification area frame with all text boxes of the image to be identified after cropping and mapping to the corresponding position of the standard template. Set a reasonable matching threshold, preferably IoU>0.1, as the standard for successful text box matching. Single-line identification area can basically be found in actual scenes by successful ‌IoU matching, but multiple lines are more complicated, so an example is given here to illustrate. In addition, if two rectangles have overlapping parts, in addition to calculating ‌IoU, ‌IoU_max will also be calculated. ‌IoU_max is the ratio of the overlapping area to the smaller of the two rectangle areas. Finally, the conditions for judging successful matching are ‌IoU>0.1, or ‌IoU>0.001 and ‌IoU_max>0.3. ‌IoU_max is used for complex situations with multiple lines.

[0082] first, Fig.10 The coordinates of the four points of the green box 'address' identification area "Shanxi xx" in the standard template are [[103.11, 180.94], [392.61, 180.94], [392.61, 283.93], [103.11, 283.93]], and the area is S green. In order to facilitate the description of the multi-line merging operation, this article assumes Fig.11 The first line of the address is "Ningxia xxx", and the coordinates of the four points of its text box are [[119.51, 213.88], [373.61, 213.88], [373.61, 237.2], [119.51,237.2]], and the area of ​​the text box is recorded as S1. Its IoU with the 'address' identification area is 0.2, indicating that the two have a certain degree of intersection. The IoU_max is 1.0, which means that the line is almost completely located in the corresponding identification area; assuming that the second line is "xx village xx number", the coordinates of the four points of the text box are [[116.36, 246.74], [235.01, 246.74], [235.01, 272.18], [116.36, 272.18]], the area is recorded as S2, and its ‌IoU with the 'address' identification area is 0.1, ‌IoU_max is 1.0. Because it has been sorted by row, these two rows meet the conditions for successful matching, ‌IoU>0.1, or ‌IoU>0.001 and ‌IoU_max>0.3, so the result of the two rows of splicing "Ningxia xxxxx Village xx" is obtained as the value of the address. Similarly, the text information of all successfully matched identification areas can be obtained, and then it can be integrated and output in a structured form.

[0083] To summarize the above process, a normal and regular image is input, and after operations such as template matching, foreground extraction, and IoU matching structured extraction, the value corresponding to the field that needs structured extraction is finally obtained, thereby efficiently completing the conversion process from the original image to be identified to structured data.

[0084] Embodiment 3 The present invention proposes an intelligent recognition self-learning platform based on template matching. The platform uses the intelligent recognition self-learning method based on template matching as in the embodiment. The architecture layout of the self-learning platform is as follows: Fig.12 , divided into the following modules: user end, self-learning platform interface system (application end, hereinafter referred to as interface system), self-learning platform image recognition engine (algorithm end, hereinafter referred to as image recognition engine), self-learning platform management module (application end, hereinafter referred to as management module).

[0085] In the human resources and social security field, business processing personnel and management and operation personnel collect image information generated during the business processing process and upload historical images of the same type to the self-learning platform. Then, the image structured field definition, data annotation, and sample division are performed on the platform, and the image classifier model is trained through convenient operations to view the test results of the training model and evaluate the model performance. When the model effect meets the business needs, the API service of the corresponding business material recognition structured reasoning algorithm can be released with one click and used on the model business material intelligent recognition system.

[0086] Embodiment 4 The present invention also provides an intelligent recognition self-learning device based on template matching, such as Fig.13 As shown, including: Template building module, used to build standard templates; A template matching module is used to perform template matching between a standard template and an image to be identified through a template matching classifier; The result recognition module performs structured processing on the image to be recognized according to the standard template to obtain key information extraction to perform information recognition on the image to be recognized; the key information includes keywords and specific values.

[0087] Embodiment 5 The present invention also provides a computer-readable storage medium, which stores computer instructions. When the computer is executed by a processor, the computer implements the intelligent recognition self-learning method based on template matching as in the first embodiment.

[0088] In a specific implementation, the computer-readable storage medium is a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (HDD) or a solid-state drive (SSD), etc.; the computer-readable storage medium may also include a combination of the above-mentioned types of memory.

[0089] Embodiment 6 The present invention also provides an electronic device, such as Figure 5 As shown, it includes at least one processor and a memory connected to the processor in communication, wherein the memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the processor to perform the intelligent recognition self-learning method based on template matching as in Example 1.

[0090] In specific implementation, the number of processors may be one or more, and the processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned chips. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0091] The memory and the processor can be communicatively connected via a bus or other means. The memory stores instructions that can be executed by at least one processor. The instructions are executed by at least one processor so that the processor executes the intelligent recognition self-learning method based on template matching introduced in the above method embodiment.

[0092] The present invention has the following advantages: Generalization capability: Provide a self-learning platform that can adapt to a variety of business materials, reduce dependence on specific rules, and improve the system's generalization capability and adaptability.

[0093] Simplify the operation process: By providing a user-friendly interface and tools, business operators and management and operation personnel can easily complete tasks such as adding new templates, image data annotation, model training and deployment, thereby lowering the usage threshold.

[0094] Improve user experience: Through automated identification and processing processes, reduce the time spent on manual operations, improve the efficiency and accuracy of business processing, and enhance user satisfaction.

[0095] The business material self-learning platform of the present invention can identify business materials very well. For business materials with fixed templates, such as bank cards, identity cards, social security cards, household registers, passports, Hong Kong and Macao passes, Taiwan compatriots' home return permits, pension application forms for pension insurance participants, labor arbitration applications, etc., they can be directly identified and structured stored.

[0096] For business materials that are not fixed templates preset by the system, the existing system data can be provided for the system to learn and identify autonomously. Users (such as business handling personnel) are supported to train classifiers (image data upload, annotation, sample division and model training) and annotate templates. Finally, the model after multiple rounds of iterative training can be verified with pre-set test samples. Models that meet the accuracy requirements can be deployed with one click and API services can be released.

[0097] It greatly reduces the manual input of paper materials offline, improves the efficiency of handling matters by handling agencies, truly realizes the electronicization of materials, efficient handling, standardized handling procedures, and standardized quality supervision. In order to improve the efficiency of handling related business by handling personnel, when accepting corporate pension insurance and work-related injury insurance business, it eliminates the need for handling personnel to manually enter a large number of business material information items, reduces the input error rate, and improves business handling efficiency. It is then applied to business handling to solve problems such as low model applicability caused by the diverse formats of human resources and social security business materials, and improves the convenience, intelligence, and scientificity of the current human resources and social security business application system.

[0098] Although the terms such as standard template, image to be recognized, key information, keyword and specific value are used more frequently in this article, the possibility of using other terms is not excluded. The use of these terms is only for more convenient description and explanation of the essence of the present invention; interpreting them as any additional limitation is contrary to the spirit of the present invention.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent recognition self-learning method based on template matching, characterized in that: The steps include: S100, build standard template; S200, performing template matching between the standard template and the image to be identified through a template matching classifier; S300, performing structured processing on the image to be identified according to the standard template to obtain the key information extraction, so as to perform information identification on the image to be identified; the key information includes keywords and specific values.

2. The intelligent recognition self-learning method based on template matching according to claim 1 is characterized in that: S100 builds a standard template including the following steps: S110, uploading an image of the standard template, identifying all text boxes of the standard template, extracting reference fields of the standard template, and selecting an identification area containing the specific value to form an identification area frame, and then saving the standard template; S120: Input a preset number of training images to train the template matching classifier.

3. The intelligent recognition self-learning method based on template matching according to claim 1 is characterized in that: The specific steps of S200 are: S210, the template matching classifier extracts and analyzes the template according to the image to be identified, and determines the standard template corresponding to the image to be identified; S220, correcting the image to be recognized; S230, calling a text detection and recognition model to perform text recognition on all text boxes of the standard template and the image to be recognized.

4. The intelligent recognition self-learning method based on template matching according to claim 1 is characterized in that: S310, performing structural processing on the image to be recognized, constructing the first frame and the second frame, and mapping the standard template and the image to be recognized according to the first frame and the second frame; S320, output the result of the structured processing to extract the key information, so as to identify and output the key information of the image to be identified.

5. The intelligent recognition self-learning method based on template matching according to claim 4 is characterized in that: The specific steps of S310 are: S311, initializing the parameters of the standard template and the image to be recognized; S312, establishing a coordinate axis, extracting the data required for structuring in the standard template and the image to be identified and performing data format conversion, and converting the position information into position data containing coordinates; S313, finding each reference field of the standard template and the image to be recognized, and determining a reference field frame, while merging the same reference field of text frames that are similar in line and are respectively detected as at least two; S314, performing first anchor point positioning according to the reference field frame of the standard template, connecting the determined first anchor point to form a first frame of the standard template, the first frame including all the reference field anchor points of the standard template; performing second anchor point positioning according to the reference field frame of the image to be recognized, connecting the determined second anchor point to form a second frame of the image to be recognized, the second frame including all the reference field anchor points of the image to be recognized; S315, calculating a first boundary distance between the first frame and the standard template; S316, calculating a second boundary distance of a second frame of the image to be recognized, calculating a document main body area of ​​the image to be recognized according to the second boundary distance, cropping the image to be recognized, and scaling the cropped image to be recognized according to a ratio; S317: Map the cropped and scaled image to be recognized to the standard template, and perform Intersection over Union (IoU) calculation on the identification area of ​​the standard template and all the text boxes of the cropped and scaled image to be recognized.

6. The intelligent recognition self-learning method based on template matching according to claim 5 is characterized in that: Calculating a similarity ratio between the first frame and the second frame; calculating a second boundary distance of the second frame of the image to be recognized according to the first boundary distance and the similarity ratio; The formula for calculating the similarity ratio is: in, For similar proportions, is the height of the first frame, is the height of the second frame. max(1, template_anchor_high) is used to prevent division by zero. The calculation formula for the second boundary distance is: in, is the second boundary distance, is the first boundary distance.

7. The intelligent recognition self-learning method based on template matching according to claim 5 is characterized in that: The specific steps of S317 are: S3171, traversing each identification area frame of the standard template; S3172, calculating the intersection and union ratio of the identification area frame of the standard template and all the text frames of the image to be recognized that are cropped and mapped to the standard template one by one; S3173. When the calculated intersection-union ratio is greater than a preset value, the text box matches successfully.

8. An intelligent recognition self-learning device based on template matching, characterized in that: include: Template building module, used to build standard templates; A template matching module, used for performing template matching between the standard template and the image to be identified through a template matching classifier; The result recognition module is used to perform structured processing on the image to be recognized according to the standard template to obtain the key information extraction, so as to perform information recognition on the image to be recognized; the key information includes keywords and specific values.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer is executed by a processor, the intelligent recognition self-learning method based on template matching as described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: It includes at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by at least one processor, and the instructions are executed by at least one processor so that the processor executes the intelligent recognition self-learning method based on template matching as described in any one of claims 1-7.

Images