Document image recognition method and device

Abstract

The invention discloses a document image recognition method and device. The method comprises the steps that a to-be-recognized document image including a to-be-recognized document is acquired, a template image matched with the to-be-recognized document image is determined according to the type of the to-be-recognized document, and anchor point fields of the template image are the same as anchor point fields of the to-be-recognized document; based on the position information of the anchor point field in the template image, determining a to-be-identified area corresponding to the anchor point field of the to-be-identified document image; and identifying the to-be-identified area to obtain text content contained in the to-be-identified document image.

Description

Document image recognition method and device Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a document image recognition method and device. Background Art

[0002] In some scenarios, documents such as bills and spreadsheets are displayed as images. To facilitate document management, the document content can be digitized. For example, the text content in the document can be converted to a spreadsheet or electronic document. This allows subsequent staff to modify, adjust, or organize the text content using electronic devices.

[0003] Summary of the Invention

[0004] In one aspect, a document image recognition method is provided. The method includes obtaining a document image to be recognized that includes a document to be recognized, and determining a template image that matches the document image to be recognized based on the type of the document to be recognized. The template image has an anchor field that is identical to the anchor field of the document image to be recognized. Based on positional information of the anchor field in the template image, a region to be recognized corresponding to the anchor field of the document image to be recognized is determined, and the region to be recognized is recognized to obtain text content contained in the document image to be recognized.

[0005] In some embodiments, the above-mentioned method of "obtaining a document image to be identified" may specifically include: obtaining a target document image, determining a detection frame of the target document image, and obtaining the document image to be identified from the target document image based on the detection frame.

[0006] In some embodiments, the aforementioned method of "determining a detection frame for a target document image" may specifically include: obtaining a first-direction line segment set and a second-direction line segment set for the target document image based on a line segment detection algorithm. The first-direction line segment set includes a plurality of lines whose angles with the first direction are less than a first preset angle, and the second-direction line segment set includes a plurality of lines whose angles with the second direction are less than the first preset angle, with the first direction and the second direction being perpendicular. The lines in the first-direction line segment set are merged to obtain a plurality of merged lines in the first direction. The angle between any two of the plurality of lines in the first direction is greater than or equal to the second preset angle, and the distance between them is greater than or equal to a preset distance. The lines in the second-direction line segment set are merged to obtain a plurality of merged lines in the second direction. The angle between any two of the plurality of lines in the second direction is greater than or equal to the preset angle, and the distance between them is greater than or equal to a preset distance. The detection frame for the target document image is determined based on the merged plurality of lines in the first direction and the merged plurality of lines in the second direction.

[0007] In some embodiments, the specific method of "determining a detection frame for the target document image based on the merged multiple straight lines in the first direction and the merged multiple straight lines in the second direction" may include: determining multiple quadrilaterals based on the merged multiple straight lines in the first direction and the merged multiple straight lines in the second direction, wherein one quadrilateral is formed by two straight lines from the merged multiple straight lines in the first direction and two straight lines from the merged multiple straight lines in the second direction. Selecting a detection frame for the target document image from the multiple quadrilaterals based on parameters of the multiple quadrilaterals, where the parameters of the quadrilaterals include one or more of parallelism, area, and matching degree of the quadrilaterals.

[0008] In some embodiments, the method of "determining the to-be-recognized area corresponding to the anchor field of the document image to be recognized based on the position information of the anchor field in the template image" may specifically include: determining the position information of the anchor field in the document image to be recognized based on the position information of the anchor field in the template image, and determining the text recognition area corresponding to the anchor field based on the position information of the anchor field in the document image to be recognized. Determining the to-be-recognized area within the text recognition area based on the mapping relationship corresponding to the anchor field.

[0009] In some embodiments, the mapping relationship corresponding to the anchor field includes a correspondence between the position information of the anchor field in the template image and the position information of the anchor field in the document image to be recognized. The method of "determining the area to be recognized within the text recognition area based on the mapping relationship corresponding to the anchor field" specifically includes: determining the position information of the anchor field within the area to be recognized based on the position information of the anchor field in the template image, and determining the area to be recognized based on the correspondence between the anchor field and the corresponding text content.

[0010] In some embodiments, the above-mentioned method of "identifying the area to be identified and obtaining the text content of the document image to be identified" may specifically include: if the anchor field in the template image matches the anchor field of the document image to be identified, then identifying the area to be identified and obtaining the text content of the document image to be identified.

[0011] In some embodiments, the method further includes: determining whether the template image matches the document image to be identified based on the text content included in the anchor field of the template image and the text content included in the anchor field of the document image to be identified. If the cosine similarity between the text content included in the anchor field of the template image and the text content included in the anchor field of the document image to be identified is greater than or equal to a threshold, then the template image matches the document image to be identified. If the cosine similarity between the text content included in the anchor field of the template image and the text content included in the anchor field of the document image to be identified is less than the threshold, then the template image does not match the document image to be identified.

[0012] In some embodiments, the method of "determining a text recognition area corresponding to an anchor field based on the position information of the anchor field in the document image to be recognized" specifically includes: determining a first coordinate point and a second coordinate point of the document image to be recognized based on the position information of the anchor field in the document image to be recognized, wherein the position information of the first coordinate point includes the minimum horizontal coordinate and the minimum vertical coordinate of the anchor field in a preset coordinate system, and the position information of the second coordinate point includes the maximum horizontal coordinate and the maximum vertical coordinate of the anchor field in the preset coordinate system. The area in the document image to be recognized where a matrix constructed with the first coordinate point and the second coordinate point as diagonals is located is determined as the text recognition area.

[0013] In some embodiments, the method of "determining the area to be identified corresponding to the anchor field of the document image to be identified based on the position information of the anchor field in the template image" specifically includes: inputting the document image to be identified into a single-word text detection model to obtain the position information of each field in the document image to be identified, and determining the anchor field among multiple fields of the document image to be identified based on the position information of each field in the document image to be identified and the position information of the anchor field of the template image. Based on the position information of the anchor field of the document image to be identified, determining the area to be identified corresponding to the anchor field of the document image to be identified.

[0014] In some embodiments, the method further includes: inputting the image of the document to be identified into a document type determination model to obtain the type of the document to be identified. The aforementioned method of "determining a template image that matches the image of the document to be identified based on the type of the document to be identified" specifically includes: selecting, based on the type of the document to be identified, a template image of the same type as the document to be identified from a plurality of calibrated template images. The calibrated target image refers to an image with the position information of the anchor field and the recognition area corresponding to the anchor field calibrated.

[0015] In some embodiments, the method further includes: acquiring multiple types of template images, and determining, in response to a calibration operation, position information of an anchor point field of each type of template image and a recognition area corresponding to the anchor point field.

[0016] On the other hand, a document image recognition device is provided, which includes an acquisition unit, a determination unit, and a recognition unit.

[0017] The acquiring unit is configured to acquire a document image to be recognized including the document to be recognized.

[0018] The determining unit is configured to: determine a template image that matches the image of the document to be identified according to the type of the document to be identified, wherein the anchor point field of the template image is the same as the anchor point field of the image of the document to be identified.

[0019] The determining unit is further configured to determine the to-be-recognized area corresponding to the anchor field of the to-be-recognized document image based on the position information of the anchor field in the template image.

[0020] The recognition unit is configured to: recognize the area to be recognized and obtain the text content contained in the document image to be recognized.

[0021] In some embodiments, the acquisition unit is specifically configured to: acquire a target document image, determine a detection frame of the target document image, and acquire a document image to be identified from the target document image based on the detection frame.

[0022] In some embodiments, the determination unit is specifically configured to: obtain a first-direction line segment set and a second-direction line segment set of the target document image based on a straight line segment detection algorithm. The first-direction line segment set includes multiple straight lines whose angles with the first direction are less than a first preset angle, and the second-direction line segment set includes multiple straight lines whose angles with the second direction are less than the first preset angle, and the first direction is perpendicular to the second direction. The straight lines in the first line segment set are merged to obtain a plurality of merged straight lines in the first direction. The angle between any two of the multiple horizontal straight lines is greater than or equal to the second preset angle, and the distance between them is greater than or equal to a preset distance. The straight lines in the second-direction line segment set are merged to obtain a plurality of merged straight lines in the second direction. The angle between any two of the multiple second-direction straight lines is greater than or equal to the second preset angle, and the distance between them is greater than or equal to a preset distance. The detection frame of the target document image is determined based on the merged plurality of straight lines in the first direction and the merged plurality of straight lines in the second direction.

[0023] In some embodiments, the determining unit is specifically configured to: determine a plurality of quadrilaterals based on the merged plurality of straight lines in the first direction and the merged plurality of straight lines in the second direction, where a quadrilateral is formed by two straight lines from the merged plurality of straight lines in the first direction and two straight lines from the merged plurality of straight lines in the second direction; and select a detection frame for the target document image from the plurality of quadrilaterals based on parameters of the plurality of quadrilaterals, where the parameters of the quadrilaterals include one or more of parallelism, area, and matching degree of the quadrilaterals.

[0024] In some embodiments, the determining unit is specifically configured to: determine the position information of the anchor field in the document image to be recognized based on the position information of the anchor field in the template image, determine the text recognition area corresponding to the anchor field based on the position information of the anchor field in the document image to be recognized, and determine the area to be recognized within the text recognition area based on the mapping relationship corresponding to the anchor field.

[0025] In some embodiments, the mapping relationship corresponding to the anchor field includes a correspondence between position information of the anchor field in the template image and position information of the anchor field in the document image to be recognized. The determination unit is specifically configured to: determine the position information of the anchor field in the area to be recognized based on the position information of the anchor field in the template image, and determine the area to be recognized based on the correspondence between the anchor field and the corresponding text content.

[0026] In some embodiments, the recognition unit is specifically configured to: if the anchor field in the template image matches the anchor field of the document image to be recognized, recognize the area to be recognized to obtain the text content of the document image to be recognized.

[0027] In some embodiments, the determination unit is further configured to determine whether the template image matches the document image to be identified based on the text content included in the anchor field of the template image and the text content included in the anchor field of the document image to be identified. If the cosine similarity between the text content included in the anchor field of the template image and the text content included in the anchor field of the document image to be identified is greater than or equal to a threshold, then the template image matches the document image to be identified. If the cosine similarity between the text content included in the anchor field of the template image and the text content included in the anchor field of the document image to be identified is less than the threshold, then the template image does not match the document image to be identified.

[0028] In some embodiments, the determination unit is specifically configured to: determine a first coordinate point and a second coordinate point of the document image to be recognized based on position information of an anchor point field in the document image to be recognized, where the position information of the first coordinate point includes the minimum horizontal coordinate and the minimum vertical coordinate of the anchor point field in a preset coordinate system, and the position information of the second coordinate point includes the maximum horizontal coordinate and the maximum vertical coordinate of the anchor point field in the preset coordinate system. Determine an area in the document image to be recognized where a matrix constructed with the first coordinate point and the second coordinate point as diagonals is located as the text recognition area.

[0029] In some embodiments, the determination unit is specifically configured to: input the document image to be recognized into the single-word text detection model, obtain position information of each field in the document image to be recognized, and determine an anchor field among multiple fields of the document image to be recognized based on the position information of each field in the document image to be recognized and the position information of the anchor field of the template image. Based on the position information of the anchor field of the document image to be recognized, determine the to-be-recognized area corresponding to the anchor field of the document image to be recognized.

[0030] In some embodiments, the determination unit is further configured to input the image of the document to be identified into a document type determination model to obtain the type of the document to be identified. The determination unit is specifically configured to select, based on the type of the document to be identified, a template image of the same type as the document to be identified from a plurality of calibrated template images. The calibrated target image refers to an image in which the position information of the anchor field and the recognition area corresponding to the anchor field are calibrated.

[0031] In some embodiments, the acquiring unit is further configured to acquire multiple types of template images. The determining unit is further configured to determine, in response to the calibration operation, the position information of the anchor field of each type of template image and the recognition area corresponding to the anchor field.

[0032] In another aspect, a document image recognition device is provided, comprising a processor and a communication interface. The communication interface is coupled to the processor. The processor is configured to execute a computer program or instruction to implement the document image recognition method of any of the above embodiments.

[0033] In another aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer program instructions, which, when executed on a computer (eg, a server), causes the computer to execute the document image recognition method as described in any of the above embodiments.

[0034] In another aspect, a computer program product is provided, comprising computer program instructions, which, when executed on a computer (eg, a server), cause the computer to execute the document image recognition method according to any one of the above embodiments.

[0035] In another aspect, a computer program is provided. When the computer program is executed on a computer (eg, a server), the computer program enables the computer to execute the document image recognition method according to any one of the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] To more clearly illustrate the technical solutions of the present disclosure, the following briefly introduces the drawings required for use in some embodiments of the present disclosure. Obviously, the drawings described below are only drawings of some embodiments of the present disclosure, and those skilled in the art can also derive other drawings based on these drawings. Furthermore, the drawings described below are schematic diagrams and are not intended to limit the actual dimensions of the products, actual processes of the methods, actual timing of signals, and the like involved in the embodiments of the present disclosure.

[0037] FIG1 is a schematic diagram of document image recognition according to some embodiments;

[0038] FIG2 is a schematic diagram of a calibrated template image according to some embodiments;

[0039] FIG3 is a schematic diagram of a recognition result of a document image according to some embodiments;

[0040] FIG4 is a schematic diagram of a method for calibrating a template image according to some embodiments;

[0041] FIG5 is a schematic diagram of an upload interface according to some embodiments;

[0042] FIG6 is a schematic diagram of an uploaded template image according to some embodiments;

[0043] FIG7 is a schematic diagram of an editing interface according to some embodiments;

[0044] FIG8 is a schematic diagram of a calibrated template image according to some embodiments;

[0045] FIG9 is a schematic diagram of a calibrated template image according to some embodiments;

[0046] FIG10 is a schematic diagram of a document image recognition method according to some embodiments;

[0047] FIG11 is a schematic diagram of an upload interface according to some embodiments;

[0048] FIG12 is a schematic diagram of a region to be recognized of a document image to be recognized according to some embodiments;

[0049] 13 is a schematic diagram of a collection of first directional line segments and second directional line segments of a document image according to some embodiments;

[0050] FIG14 is a schematic diagram of line segment merging according to some embodiments;

[0051] FIG15 is a schematic diagram of a document image recognition method according to some embodiments;

[0052] FIG16 is a schematic diagram of a document image to be recognized according to some embodiments;

[0053] FIG17 is a schematic diagram of a detection frame of a document image to be recognized according to some embodiments;

[0054] FIG18 is a schematic diagram of a text recognition area of ​​a document image to be recognized according to some embodiments;

[0055] FIG19 is a schematic diagram of a region to be recognized of a document image to be recognized according to some embodiments;

[0056] FIG20 is a structural diagram of a document image recognition device according to some embodiments;

[0057] FIG21 is a structural diagram of a document image recognition device according to some embodiments. DETAILED DESCRIPTION

[0058] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions in some embodiments of the present disclosure. Obviously, the embodiments described are only some embodiments of the present disclosure, not all embodiments. Based on the embodiments provided by the present disclosure, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of the present disclosure.

[0059] Unless the context requires otherwise, throughout the specification and claims, the term "comprise" and its other forms, such as the third person singular form "comprises" and the present participle form "comprising", are to be interpreted as open, inclusive, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "example", "specific example" or "some examples" are intended to indicate that a particular feature or characteristic associated with the embodiment or example is included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features or characteristics may be included in any one or more embodiments or examples in any appropriate manner.

[0060] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, "plurality" means two or more.

[0061] When describing some embodiments, the expressions "coupled" and "connected" and their derivatives may be used. For example, when describing some embodiments, the term "connected" may be used to indicate that two or more components are in direct physical or electrical contact with each other. For another example, when describing some embodiments, the term "coupled" may be used to indicate that two or more components are in direct physical or electrical contact. However, the term "coupled" or "communicatively coupled" may also refer to two or more components that are not in direct contact with each other, but still cooperate or interact with each other. The embodiments disclosed herein are not necessarily limited to the contents of this document.

[0062] “At least one of A, B and C” has the same meaning as “at least one of A, B or C” and both include the following combinations of A, B and C: A only, B only, C only, the combination of A and B, the combination of A and C, the combination of B and C, and the combination of A, B and C.

[0063] “A and / or B” includes the following three combinations: A only, B only, and a combination of A and B.

[0064] As used herein, the term "if" is optionally interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that" or "if [stated condition or event] is detected" are optionally interpreted to mean "upon determining" or "in response to determining" or "upon detecting [stated condition or event]" or "in response to detecting [stated condition or event]," depending on the context.

[0065] The use of "adapted to" or "configured to" herein is intended to be open and inclusive language that does not exclude devices adapted or configured to perform additional tasks or steps.

[0066] Additionally, the use of “based on” is meant to be open and inclusive, as a process, step, calculation, or other action “based on” one or more stated conditions or values ​​may, in practice, be based on additional conditions or values ​​beyond those stated.

[0067] In some scenarios, to facilitate the management of documents (such as receipts, forms, cards, etc.), paper documents can be converted to electronic documents, or the text content contained in paper documents can be recognized to obtain the fields in the document. In this way, subsequent staff can manage the electronic documents or the fields in the documents.

[0068] In some scenarios, documents are stored in image format (i.e., document images in the embodiments of the present application). However, in some cases, the text content contained in the document image needs to be summarized. For example, to summarize the total amount on multiple bills, it is necessary to determine the amount on each bill and then summarize them.

[0069] Typically, the text content contained in a document image can be identified manually. However, when multiple types or quantities of documents need to be identified, the manual method is not only inefficient but also prone to errors.

[0070] In view of this, an embodiment of the present application provides a document image recognition method for recognizing text content contained in a document image, which can effectively improve the recognition efficiency of the document image and meet the needs of personalized text recognition.

[0071] For example, after acquiring a document image to be recognized, a matching template image can be selected based on the type of document in the document image to be recognized. Based on the anchor field in the template image and the location of the anchor field in the template image, text detection and recognition can be performed on the document image to be recognized, thereby obtaining a recognition result for the document image to be recognized. This recognition result may include the anchor field in the document image to be recognized and the text content corresponding to the anchor field. For more details, please refer to the technical solution shown in FIG10 below.

[0072] Furthermore, the present invention also provides a template image calibration method for calibrating anchor fields in a template image and key fields to be identified corresponding to the anchor fields.

[0073] It should be noted that, in the embodiment of the present application, the anchor field may refer to the same field in the document image of the same type, and may also be called a reference field, an identification field, or other names without limitation.

[0074] The following will describe in detail the implementation of the embodiment of the present application in conjunction with the accompanying drawings.

[0075] As shown in Figure 1, Figure 1 is an application scenario of the document image recognition method provided by an embodiment of the present application. In response to an input document image, the recognition device can perform a recognition operation on the text content in the document image.

[0076] Furthermore, the recognition device can also perform a calibration operation on the template image. The calibration operation on the template image can refer to calibrating the anchor field and the key field to be recognized in the template image. The template image refers to an image that is clean and has no tilt or deformation.

[0077] For example, as shown in Figure 2, a template image is provided in an embodiment of the present application. In Figure 2, the solid-line box is the anchor field of the template image, and the dotted-line box is the key field to be identified.

[0078] In one example, the recognition device can obtain a template image in response to an input operation, and determine the anchor field and the key field to be identified of the template image in response to a calibration operation. After determining the anchor field and the key field to be identified of the template image, the recognition device can create a corresponding text content recognition service for the template image. In this way, the recognition device can create corresponding types of text content recognition services for different types of template images. That is, one text content recognition service can be used to identify the text content in one type of document image. For example, the recognition device can call a corresponding type of text content recognition service based on the content of the document to be identified, and identify the key field to be identified corresponding to the anchor field in the document image to be identified.

[0079] In a possible implementation, the document image recognition operation may refer to identifying a key field to be recognized corresponding to an anchor field in the document image to be recognized based on a template image.

[0080] For example, as shown in FIG3 , an embodiment of the present application provides a document image to be recognized and a corresponding recognition result.

[0081] In one example, after acquiring the document image to be recognized, the recognition device can perform document orientation correction, template recognition, and template anchor point matching on the document image, and then perform text detection and recognition, and input the recognition result.

[0082] The document orientation correction may refer to correcting the orientation of the document image to be recognized to a calibration orientation, which may be the orientation of the template image.

[0083] Template recognition may refer to determining a template image that matches a document image to be recognized. For example, the recognition device may determine a corresponding template image based on the document type in the document image to be recognized.

[0084] Template anchor matching may refer to matching a field in a document image to be recognized with an anchor field in a template image or a template image to determine the anchor field in the document image to be recognized.

[0085] Text detection and recognition may refer to identifying text content corresponding to an anchor field in a document image to be recognized, and obtaining a recognition result.

[0086] In the embodiment of the present application, the identification device can be an electronic device, a computer, or a server. The server can be a physical server or a cloud server, and the server can be a single server or a cluster server, without limitation.

[0087] In some examples, the recognition device may be provided with an input unit (e.g., a scanner, keyboard, mouse) and an output unit (e.g., a display or screen). The input unit may be used to obtain a template document and a document image to be recognized. The output unit may be used to input a recognition result of the document image to be recognized.

[0088] In some examples, the recognition device can also be connected to other devices. For example, the recognition device can be used to capture a document image and upload the document image to another device (such as a server) for text content recognition. After obtaining the recognition result, the recognition device can input the recognition result. In this way, the recognition device does not need to perform text content recognition, which reduces the capability requirements of the recognition device.

[0089] The document image recognition method provided in the embodiment of the present application may include: a process of constructing a text content recognition service based on multiple types of template images (referred to as the "text content recognition service construction process") and a process of identifying the text content in the document image to be identified based on the text content recognition service (referred to as the "document image recognition process").

[0090] The following first describes the "construction process of text content recognition service".

[0091] As shown in FIG. 4 , the “construction process of text content recognition service” may include: S401 to S403 .

[0092] S401: In response to an upload operation, obtain a template image.

[0093] The uploading operation may refer to an operation of uploading a template image. For example, the uploading operation may refer to an operation of a staff member inputting a template image through an input unit of the recognition device.

[0094] In one example, the recognition device may be provided with an image editing interface, which may be used to perform calibration and recognition operations on the uploaded image.

[0095] As shown in FIG5 , a template image upload interface of the recognition device is shown. The upload interface can respond to clicking “click to upload image” to obtain a template image, resulting in the interface shown in FIG6 .

[0096] The template image may be an image that meets preset conditions. For example, the preset conditions may include one or more of image size, number of pixels, image format, and image angle. For example, the image size may not exceed 4 megabytes (M), the longest side of the image may not exceed 4096 pixels, and the image format may include Joint Photographic Experts Group (JPG) and Portable Network Graphics (PNG).

[0097] For example, to ensure users clearly understand template image requirements before uploading, the upload interface can also display template image prompts, as shown in Figure 5. This prompt can be used to indicate template image requirements. For example, the prompts can include preset conditions and image quality requirements. Image quality requirements may include: the uploaded image must have clear handwriting and be properly positioned. Thus, high-quality, correctly angled images can improve image recognition.

[0098] In another example, as shown in FIG5 , the upload interface may further include an input box for setting a template name. The input box may display the name of the template image in response to an input operation. The number of characters in the template name may not exceed a preset number, such as 30.

[0099] S402: In response to the calibration operation, obtain the anchor point field of the template image.

[0100] The calibration operation may refer to an operation of selecting an anchor field in the acquired template image. For example, the calibration operation may refer to an operation of a staff member calibrating a field in the template image through an image editing interface of the recognition device.

[0101] In one example, in response to clicking “Next” in the upload interface shown in FIG6 , the recognition device may display an editing interface as shown in FIG7 .

[0102] As shown in FIG7 , the coding interface is provided with an editing area and a field display area.

[0103] The editing area can be used to calibrate the uploaded template image, and the field display area can be used to display the calibrated fields.

[0104] As shown in FIG7 , the editing area may be provided with editing tools. The editing tools are used to adjust and edit the template image within the editing area. For example, the editing tools in FIG7 may include, from top to bottom, “Drag,” “Frame Selection,” “Image Rotation,” “Perspective Crop,” “Zoom In,” “Zoom Out,” and “Fit to Canvas.”

[0105] In a possible implementation, in response to the frame selection template image, the recognition device may recognize text content in the frame selection area and display the text information in the frame selection area in the field display area.

[0106] In one example, as shown in Figure 8, the results of a selection operation on the template image in Figure 7 and the results of identifying the fields within the selected area are shown. The identification results are anchor fields in the template image. For example, the anchor fields in the template image in Figure 7 may include: College English Test Band 6, transcript number, exam time, admission ticket number, total score, name, and ID number.

[0107] Furthermore, the recognition device can also edit / modify the text within the field display area in response to a modification operation. For example, when the text within the field display area is consistent with the text within the area corresponding to the template image, the recognition device can obtain the modified text content in response to the user's modification operation on the text within the field display area.

[0108] When the anchor field in the template image is marked, the recognition device can save the information of the marked anchor field in the template image in response to clicking "Save", for example, the text content and position information included in the anchor field.

[0109] Furthermore, the identification device may also execute S403.

[0110] S403: In response to the recognition operation, obtain the area to be recognized corresponding to each anchor point field.

[0111] The recognition operation may include a box selection operation.

[0112] In one example, in response to clicking “Step 2: Select the recognition area”, the recognition device may display an editing interface as shown in FIG9 .

[0113] For example, in combination with the editing interface shown in FIG9 , the recognition device can obtain multiple areas to be recognized in response to a frame selection operation.

[0114] In the embodiment of the present application, to improve recognition accuracy, as shown in the interface of Figure 9, each area to be recognized may also be provided with one or more setting columns. A setting column can be used to set the field name, field type, etc. within the corresponding area to be recognized. Field types may include general, pure numbers, dates, handwritten Chinese characters, pure English / characters, custom organizations, custom work areas, etc.

[0115] In one example, the number of setting columns is consistent with the number of areas to be identified, and the order of the setting columns is consistent with the order of the areas to be identified. In this way, it can be ensured that one setting column can set the field name, character length, field type, etc. in the area to be identified at the corresponding position.

[0116] In another example, to more accurately select the area to be recognized, the editing area may also be provided with editing tools, as shown in Figure 9. These editing tools can be used to adjust the template image within the editing area. For example, the editing tools in Figure 9, from top to bottom, may include: "Drag," "Select," "Perspective Crop," "Zoom In," "Zoom Out," "Fit to Canvas," etc.

[0117] Drag to adjust the image's shape. Select a box to select fields within an image. Perspective crop to crop an image. Zoom in and out to adjust the image's size. Fit to canvas to adjust the ratio between the image and the editing area.

[0118] In an embodiment of the present application, after obtaining the anchor point field of the template image and the corresponding area to be identified, the identification device can create a corresponding text content recognition service for this type of template image in response to clicking "Publish".

[0119] Based on the embodiment described in FIG4 , the recognition device can calibrate the template image through the editing interface to obtain the anchor field of the template image and the corresponding area to be recognized. In this way, for each type of document, the recognition device can create a corresponding text content recognition service to facilitate subsequent recognition of the text content of the document image.

[0120] After explaining the “construction process of the text content recognition service”, the “document image recognition process” in the embodiment of the present application is explained below.

[0121] In one example, as shown in FIG10 , in an embodiment of the present application, the “document image recognition process” may include S1001 to S1004 .

[0122] S1001: Obtain a document image to be recognized.

[0123] The image of the document to be identified includes the document to be identified. For example, the image of the document to be identified may be an image obtained by photographing the document to be identified using a camera, or may be an image obtained by scanning the document to be identified using a scanner.

[0124] In one example, after obtaining multiple types of text content recognition services, in conjunction with the editing interface shown in FIG9 , in response to clicking "Test," an upload interface as shown in FIG11 may be displayed. Based on the interface shown in FIG11 , in response to clicking "Please Upload Image," the recognition device may obtain an image to be recognized. Similarly, the uploaded image to be recognized also meets the aforementioned pre-set conditions.

[0125] S1002: Determine a template image that matches the document to be identified based on the type of the document to be identified.

[0126] The template image that matches the document to be identified may be a template document of the same type as the document to be identified, for example, both are College English Test Band 6 transcripts.

[0127] In one example, the recognition device can determine the type of the document to be recognized based on a document type determination model. The document type determination model takes a document image as input and outputs the document type. The document type determination model is obtained by training training data based on a preset algorithm. The preset algorithm can be set as needed, for example, a neural network algorithm, a machine learning algorithm, etc. The training data set can include template images of multiple types. For example, the number of template images of a given type can be set as needed, for example, greater than or equal to 30.

[0128] Furthermore, in order to more accurately determine the type of document to be identified, the template images in the training data may be oriented. For example, training data may be constructed based on the document type and the direction of the document may be marked (only one direction may be marked).

[0129] During data augmentation, document orientation can be annotated based on rotation, flipping, and other operations performed during the augmentation process. For example, for a document rotated 90° counterclockwise, the document image's orientation can be annotated as 90° counterclockwise. In this way, a document type determination model trained on training data consisting of orientation-annotated template images can more accurately determine document types.

[0130] In an embodiment of the present application, after determining the type of the document image to be identified, the recognition device can select a text content recognition service of the same type as the document image to be identified from a plurality of pre-built types of text content recognition services. That is, a calibrated template image is obtained. The position information between the anchor field and the corresponding recognition area in the calibrated template image is fixed, or the relative distance between the position information between the anchor field and the corresponding recognition area in the calibrated template image is constant. Based on this, subsequently, the server can determine the positional relationship between the anchor field and the corresponding recognition area of ​​the document image to be identified based on the positional relationship between the anchor field and the corresponding recognition area. Specifically, please refer to the description of the subsequent embodiments, which will not be repeated here.

[0131] S1003 : Determine the area to be recognized corresponding to the anchor field of the document image to be recognized based on the position information of the anchor field in the template image.

[0132] The position information of the anchor point field in the template image is the same as the position information of the anchor point field in the document image to be recognized. The position information may include coordinate data.

[0133] In this way, the recognition device can determine the position information of the anchor field in the document image to be recognized based on the position information of the anchor field in the template image. Because the position information of the anchor field and the position information of the area to be recognized are fixed, the recognition device can determine the position information of the area to be recognized based on the position information of the anchor field. In other words, the area to be recognized of the document image to be recognized is obtained.

[0134] In one example, as shown in FIG12 , the area to be recognized of the document image to be recognized may be the area indicated by the dotted box in the figure.

[0135] For details, please refer to the description of the subsequent embodiments, which will not be described in detail here.

[0136] S1004: Identify the area to be identified to obtain text content contained in the document image to be identified.

[0137] The text content contained in the area to be identified can refer to the text content corresponding to the anchor field. For example, in conjunction with the aforementioned College English Test Band 6 transcript, the anchor fields for this type of document may include name, school, department, admission ticket number, ID number, total score, etc. The text content in the area to be identified can include the field corresponding to the name, the field corresponding to the school, the value corresponding to the admission ticket number, the value corresponding to the ID number, the value corresponding to the total score, etc.

[0138] In one example, the recognition results in the embodiments of the present application can be displayed in a structured form.

[0139] In one possible implementation, to improve recognition efficiency, the recognition device may use a preset algorithm to identify the target area. The preset algorithm can be set as needed, for example, a progressive scale expansion network based on a deep learning algorithm and an end-to-end image sequence neural network.

[0140] In the embodiment of the present application, the recognition process for the region to be recognized may include two stages: a feature learning stage and a text alignment stage. The feature learning stage may refer to the stage of extracting labels of the feature sequence of the region to be recognized, and the text alignment stage may refer to the stage of integrating the labels of the feature sequence obtained in the feature learning stage to obtain the recognition result.

[0141] In one example, the above preset algorithm may include three layers: a convolutional layer, a recurrent layer, and a transcription layer.

[0142] The convolutional layer can be used to extract feature sequences from an image. For example, the convolutional layer can use a convolutional neural network (CNN) to extract feature sequences of text content from a document image.

[0143] The recurrent layer can be used to predict the label (true value) distribution of the feature sequence obtained from the convolutional layer. For example, the recurrent layer can use the recurrent neural network (RNN) layer to obtain the label distribution of the feature sequence of the text content.

[0144] The transcription layer can be used to convert the label distribution obtained from the recurrent layer into the final recognition result through operations such as deduplication and integration. For example, the transcription layer can use the connectionist temporal classification (CTC) criterion to dedupe and integrate the label distribution of the feature sequence of the text content.

[0145] Based on the technical solution of Figure 10, after acquiring a document image to be recognized, a template image can be obtained based on the type of document to be recognized. Because the position of the anchor field and the position of the area to be recognized are fixed, the area to be recognized in the document image can be accurately determined based on the position information of the anchor field in the template image. In this way, the area to be recognized can be recognized and the required text content can be obtained.

[0146] In some embodiments, after obtaining the image of the document to be identified, in order to improve the recognition accuracy, the background in the image of the document to be identified can be removed and the image of the document to be identified can be corrected to obtain an image of the document to be identified with less noise and less distortion.

[0147] In one possible scenario, the document image uploaded by the user may be tilted or flipped. This significantly affects the text content recognition results. Therefore, to improve text recognition accuracy, text edge detection (to remove the background of the document image) and correction can be performed on the uploaded document image.

[0148] The processes of background removal and image correction are described below.

[0149] 1. Background removal.

[0150] Background removal can refer to removing noise from the image of a document to be recognized, excluding the document to be recognized. For example, if the image of the document to be recognized is an image of the document to be recognized placed on a table, the table may be present in the image. Therefore, to improve recognition accuracy, the table can be removed from the image of the document to be recognized.

[0151] In a possible implementation, after acquiring a target document image including a document to be recognized, the recognition device may determine a detection frame of the target document image, and acquire an image of the document to be recognized from the target document image based on the detection frame.

[0152] The detection frame can be used to detect the document to be identified in the image of the document to be identified. For example, the image within the detection frame is the image of the document to be identified.

[0153] In an embodiment of the present application, the recognition device can perform line detection on a target document image to obtain multiple lines of the document image to be recognized. The recognition device can merge approximately parallel lines from the multiple lines to obtain merged vertical and horizontal lines. The recognition device can then determine a boundary line from a quadrilateral formed by the vertical and horizontal lines, and further, based on the selected boundary line, obtain a detection frame for the target document image.

[0154] For example, the recognition device may perform line detection on the document image to be recognized using a line segment detector (LSD) method to obtain a plurality of lines (or line segments), among which there may be interference lines.

[0155] In one example, as shown in Figure 13, multiple straight lines or line segments are detected in a document image to be recognized. Based on the angles between the straight lines and a first direction, the multiple straight lines can be divided into a first-direction line segment set and a second-direction line segment set. The angle between each straight line in the first-direction line segment set and the first direction is within a range of 0 to 45 degrees, and the angle between each straight line in the second-direction line segment set and the first direction is within a range of 45 to 90 degrees. The straight lines in the first-direction line segment set do not intersect, and the straight lines in the second-direction line segment set do not intersect.

[0156] The angle between the first direction and the second direction is 90 degrees. For example, the first direction is horizontal and the second direction is vertical.

[0157] As shown in Figure 14, taking the first direction as the horizontal direction and the second direction as the vertical direction as an example, the first direction line segment set can be referred to as a horizontal line segment set, and the second direction line segment set can be referred to as a vertical line segment set. For any two straight lines in the plurality of straight lines within the horizontal line segment set, the two straight lines whose included angle is less than a preset angle and whose distance is less than a preset distance are merged into one. For example, the two straight lines can be represented by the longer of the two straight lines to obtain a plurality of merged straight lines. If there are still two straight lines in the plurality of merged straight lines whose included angle is less than the preset angle and whose distance is less than the preset distance, these two straight lines are merged. In this way, after multiple merging, a plurality of horizontal straight lines can be obtained, and the included angle of any two straight lines in the plurality of horizontal straight lines is greater than or equal to the preset angle, and the distance between any two straight lines is greater than or equal to the preset distance.

[0158] The distance between two straight lines may refer to the minimum distance between the two straight lines. Alternatively, the distance between two straight lines may refer to the average of the maximum distance and the minimum distance between the two straight lines. The preset angle may be set as needed, for example, 15 degrees, 20 degrees, etc., without limitation.

[0159] Similarly, for any two lines in the set of vertical line segments, the two lines whose included angle is less than a preset angle and whose distance is less than a preset distance are merged into one. After multiple merging steps, multiple vertical lines can be obtained, and any two lines in the set of horizontal lines have an included angle greater than or equal to the preset angle and a distance greater than or equal to the preset distance.

[0160] In the embodiment of the present application, for the merged first direction line segment set and the merged second direction line segment set, the optimal boundary line is determined from the merged first direction line segment set and the merged second direction line segment set respectively.

[0161] In one example, the merged first-direction line segment set can be divided into a top line segment set (topLineSet) and a bottom line segment set (bottomLineSet) with the image as the center. The merged second-direction line segment set can be divided into a left line segment set (leftLineSet) and a right line segment set (rightLineSet) with the image as the center.

[0162] For example, the recognition device can select a straight line from each of the top, bottom, left, and right line segment sets, and determine the score of the quadrilateral formed by the four selected straight lines based on the parameter values ​​of the quadrilateral. In this way, the recognition device can use the quadrilateral with the highest score among the multiple quadrilaterals as the detection frame of the document image to be recognized.

[0163] Among them, the parameters of the quadrilateral may include one or more of parallelism, area, and matching degree. Parallelism can be used to characterize the relationship between the length of the longest diagonal and the length of the shortest diagonal of a quadrilateral. For example, parallelism = 1-(length of the longest diagonal / length of the shortest diagonal), and matching degree can be used to characterize the relationship between the corresponding sides of a quadrilateral and the sum of the side lengths. For example, matching degree = sum of corresponding side lengths / sum of all side lengths. Corresponding sides may refer to sides with the same positional relationship in a quadrilateral. For example, if the upper side of a quadrilateral corresponds to the lower side of the quadrilateral, then these two sides are corresponding sides.

[0164] In one example, to more accurately determine the detection frame to be identified, the parameters of the quadrilateral may include parallelism, area, and matching. That is, the detection frame to be identified is the quadrilateral with the greatest parallelism, area, and matching among the multiple quadrilaterals.

[0165] In another example, if a parameter is shared among multiple quadrilaterals, these parameters can be prioritized. For example, area can have a higher priority than parallelism, which in turn has a higher priority than matching. Thus, when a parameter is shared, a quadrilateral can be selected based on the parameter with the next highest priority.

[0166] Of course, the above-mentioned multiple parameters are only exemplary, and other parameters can also be used to select the detection frame of the document image to be detected from multiple quadrilaterals. For example, when the horizontal line segment set and the vertical line segment set are merged as mentioned above, one line segment can correspond to one score. For example, the shorter the line segment, the lower the score. For two line segments that need to be merged, the score of the merged line segment is the score of the line segment with the largest score among the two line segments that need to be merged. In this way, when selecting the detection library of the document image to be identified from the quadrilaterals, the sum of the scores of the four line segments can be used as the score of the quadrilateral. The detection library of the document image to be identified can be the quadrilateral with the highest score among the multiple quadrilaterals.

[0167] In the embodiment of the present application, each of the multiple parameters of the quadrilateral is provided with a corresponding weight or score. Thus, after determining the values ​​of the multiple parameters of each quadrilateral, the recognition device can calculate the score of each quadrilateral based on the weight or score corresponding to each parameter.

[0168] 2. Image correction.

[0169] In some scenarios, the document image to be recognized is captured against a complex background. For example, the document image to be recognized may be placed on an uneven surface, resulting in a distorted image. Consequently, the detection frame may not fully cover the document image. Therefore, correction of the document image is necessary to improve recognition accuracy.

[0170] In one example, the recognition device can correct the document image to be recognized in response to the operation of adjusting the vertices of the detection frame. For example, the distance between the upper left vertex and the upper right vertex of the document image to be recognized before adjustment is W, and the height (height) of the document image to be recognized after correction is R*W. The coordinates of the four vertices of the document image to be recognized after adjustment are: (0, 0), (0, height), (W, 0), (W, height). Among them, R is the adjustment coefficient, for example, it can be the aspect ratio of the template image. In this way, the recognition device can construct a transformation matrix (or perspective transformation matrix) according to the changes between the coordinate data of the vertices before and after the detection frame is adjusted. Based on the transformation matrix, the recognition device can correct the image to be recognized to obtain the corrected image to be recognized.

[0171] Based on the technical solution of this embodiment, by performing background removal and correction on the document image to be recognized, the influence of noise on image recognition can be reduced, thereby improving the accuracy of image recognition.

[0172] In some embodiments, in an embodiment of the present application, as shown in FIG15 , in the above S1003, the specific implementation method of determining the area to be identified corresponding to the anchor field in the document image to be identified based on the position information of the anchor field in the template image may include the following S1501 to S1503.

[0173] S1501. Determine the position information of the anchor field in the document image to be recognized based on the position information of the anchor field in the template image.

[0174] The position information of the anchor field refers to the area occupied by the anchor field box in the image.

[0175] In one possible implementation, the recognition device may input the document image to be recognized after background correction and removal into a single-word text detection model to obtain coordinate data of each field in the document image to be recognized. The single-word text detection model may have the function of detecting the coordinate data of text in the image.

[0176] After obtaining the coordinate data for each field in the document image to be recognized, the recognition device can calculate the distance between each field and the anchor field and use the field with the closest distance as the anchor field for the document image to be recognized. In this way, the recognition device can construct an anchor field frame based on the coordinate data of the anchor field in the document image to be recognized.

[0177] The distance between the field and the anchor field can be the Euclidean distance. For example, the distance x1 and y1 represent the coordinate data of the x1 field. x2 and y2 represent the coordinate data of the anchor field. The coordinate data of the anchor field can be the coordinate data of the first field in the anchor field, or the coordinate data of the middle field of the anchor field.

[0178] It should be noted that in the embodiments of the present application, the position information of a field refers to coordinate data in a preset rectangular coordinate system. The preset rectangular coordinate system can be established based on the document image to be recognized. For example, the origin (0, 0) of the preset rectangular coordinate system can be the lower left corner vertex of the detection box of the document image to be recognized, the x-axis is the horizontal direction, and the y-axis is the vertical direction.

[0179] S1502: Determine a text recognition area based on position information of an anchor point field in the document image to be recognized.

[0180] The text recognition area may refer to an area including fields in the document image to be recognized.

[0181] In one example, the recognition device can sort the coordinate data of the anchor field box in the document image to be recognized, obtain the coordinate data of the two points with the farthest distance, namely (minimum horizontal coordinate Xmin, minimum vertical coordinate Ymin), (maximum horizontal coordinate Xmax, maximum vertical coordinate Ymax), and use the minimum circumscribed rectangle determined by the two points as the text recognition area of ​​the document image to be recognized.

[0182] It should be noted that the aforementioned minimum horizontal coordinate Xmin, minimum vertical coordinate Ymin, maximum horizontal coordinate Xmax, and maximum vertical coordinate Ymax refer to the values ​​in the coordinate data of multiple fields recognized by the recognition device. For example, if the recognition device recognizes 10 fields, each field corresponding to one piece of coordinate data, then the aforementioned Xmin is the minimum horizontal coordinate value of the coordinate data in the 10 fields, Ymin is the minimum vertical coordinate value of the coordinate data in the 10 fields, Xmax is the maximum horizontal coordinate value of the coordinate data in the 10 fields, and Ymax is the maximum vertical coordinate value of the coordinate data in the 10 fields.

[0183] The coordinates of the four vertices of the rectangle are: (Xmin, Ymin), (Xmin, Ymax), (Xmax, Ymin), (Xmax, Ymax).

[0184] For example, as shown in Figure 16, an embodiment of the present application provides a document image to be identified. The solid-line box in Figure 16 represents the anchor point field frame. The coordinate system within this anchor point field frame is a rectangular coordinate system established with the lower left corner of the document image to be identified as the origin, the horizontal axis as the horizontal direction, and the vertical axis as the vertical direction. Figure 16 shows the minimum horizontal coordinate, minimum vertical coordinate, maximum horizontal coordinate, and maximum horizontal coordinate of the anchor point field frame in the document image to be identified.

[0185] Combined with the minimum horizontal coordinate, minimum vertical coordinate, maximum horizontal coordinate and maximum horizontal coordinate of the anchor field box in FIG16 , the text recognition area of ​​the document image to be recognized may be the rectangular box shown in FIG17 .

[0186] Furthermore, in order to improve recognition accuracy, the recognition device may also calibrate the anchor field frame.

[0187] In one example, the recognition device can determine the multiple fields of the document image to be recognized that are closest to the anchor field box based on the distance between the coordinate data of the anchor field box of the template image and the coordinate data of each field in the document image to be recognized. For example, as shown in Figure 18, there are multiple text recognition areas of the document image to be recognized (the areas occupied by the solid line boxes in Figure 18).

[0188] S1503: Determine the area to be recognized in the text recognition area according to the mapping relationship corresponding to the anchor point field.

[0189] The mapping relationship corresponding to the anchor field may refer to the corresponding relationship between the position information of the anchor field in the template image and the position information of the anchor field in the document image to be recognized.

[0190] In one example, in conjunction with the text recognition area shown in FIG18, for each solid-line frame, the recognition device can determine the area to be recognized in the solid-line frame based on the position information of the anchor field within the solid-line frame. For example, FIG19 shows the area to be recognized (the area occupied by the dotted-line frame in FIG19) determined based on the text recognition area in FIG18.

[0191] For example, the recognition device may determine the position information of the anchor field in the area to be recognized based on the mapping relationship, and use the area in the solid-line frame except the area where the anchor field frame is located as the area to be recognized.

[0192] For another example, the recognition device can determine the position information of the anchor field in the to-be-recognized area based on the mapping relationship, and determine the area where the recognition area in the solid-line frame is located based on the correspondence between the anchor field and the corresponding text content area (the correspondence can be the distance between the two). That is, the to-be-recognized area of ​​the to-be-recognized document image.

[0193] Based on the technical solution shown in FIG15 , the text detection area of ​​the document image to be recognized can be determined based on the position information of the anchor field in the template image. This narrows the area of ​​the document image to be recognized. Then, based on the position information of the anchor field and the mapping relationship corresponding to the anchor field, the area of ​​the document image to be recognized can be accurately determined.

[0194] In some embodiments, after obtaining the anchor field frame in the document image to be recognized, it may be detected whether the anchor field in the template image matches the anchor field in the document image to be recognized.

[0195] If the anchor field in the template image matches the anchor field in the document image to be recognized, the recognition device may perform text recognition on the to-be-recognized area of ​​the document image to be recognized, thereby determining the text content of the document image to be recognized. If the anchor field in the template image does not match the anchor field in the document image to be recognized, the recognition device may input a message indicating that the template image and the document image to be recognized do not match. For example, a message indicating that the uploaded image cannot be recognized may be input.

[0196] In one example, the recognition device may determine whether the template image matches the document image to be recognized based on the cosine similarity between the text content included in the anchor field of the template image and the text content included in the anchor field in the document image to be recognized.

[0197] For example, if the cosine similarity between the text content included in the anchor field of the template image and the text content included in the anchor field of the document image to be identified is greater than or equal to a threshold, it indicates that the template image matches the document image to be identified, and the recognition device can perform text recognition on the text area to be identified to obtain the text content of the document image to be identified. If the cosine similarity between the text content included in the anchor field of the template image and the text content included in the anchor field of the document image to be identified is less than a threshold, it indicates that the template image does not match the text image to be identified, and the recognition device can output information indicating that the template image does not match the document image to be identified.

[0198] The threshold value can be set as needed, for example, it can be 0.9 or other values ​​without limitation.

[0199] It should be noted that in the embodiment of the present application, the anchor field frame has been calibrated. Since the text content included in the anchor field is fixed, the recognition device can calculate the cosine similarity between the text content included in the anchor field of the template image and the text content included in the anchor field in the document image to be recognized based on the number of fields of text content included in the anchor field.

[0200] For example, cosine similarity can be used to measure the size of the difference between two individuals by using the cosine value of the angle between two vectors in a vector space. If the cosine similarity is 1, it means that the two vectors are the same. If the cosine similarity is 0, it means that the two vectors are different. Based on this principle, the closer the cosine similarity between the text content included in the anchor field of the template image and the text content included in the anchor field in the document image to be identified, the closer the angle is to 0, the more similar the text content included in the anchor field of the template image and the text content included in the anchor field in the document image to be identified are.

[0201] In one example, the recognition device may separate the text content included in the anchor field of the document image to be recognized and the text content included in the anchor field of the template image using single words to obtain respective corresponding vectors.

[0202] For example, if the text content of the anchor field includes five fields, the recognition device can perform vectorization based on the position information of the five fields to obtain the vector corresponding to the anchor field. For example, the vector corresponding to the text content included in the anchor field of the template image can be (x1, x2, x3, x4, x5), and the vector corresponding to the text content included in the anchor field of the document image to be recognized can be (y1, y2, y3, y4, y5).

[0203] Based on the above two vectors, the cosine similarity D between the text content included in the anchor field of the template image and the text content included in the anchor field of the document image to be identified is mn It can be shown as the following formula 1.

[0204] Based on this embodiment, by checking whether the anchor field of the template image matches the anchor field of the document image to be recognized, it is possible to detect whether the template image and the document image to be recognized match. In this way, the problem of mismatch between the template image and the document image to be recognized, which leads to recognition anomalies, is avoided.

[0205] In some other embodiments, since the document image to be identified may include multiple anchor fields, for any of the multiple anchor fields, if the number of fields in the recognition result output by the recognition area corresponding to the anchor field is inconsistent with the number of key fields to be identified corresponding to the anchor field in the template image, the recognition device may recognize the recognition area corresponding to the anchor field in the document image to be identified again. If the number of fields included in the text content corresponding to the anchor field in the recognition result is still inconsistent with the number of key fields to be identified corresponding to the anchor field in the template image, the recognition device may input the unrecognizable anchor field. Alternatively, the recognition device may input the result and mark the result to remind that there is an abnormality in the recognition result. For example, the recognition device may bold the font of the recognition result or set it to a font of another color (such as red).

[0206] Based on this embodiment, for abnormal recognition results, the recognition device can re-recognize to obtain a correct recognition result, or input prompt information to prompt the user.

[0207] In the embodiment of the present application, the document image recognition device can be divided into functional modules or functional units according to the above method example. For example, each functional module or functional unit can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules or functional units. Among them, the division of modules or units in the embodiment of the present application is schematic and is only a logical functional division. In actual implementation, there may be other division methods.

[0208] As shown in Figure 20, it is a structural diagram of a document image recognition device provided in an embodiment of the present application. The device includes: an acquisition unit 2001, a determination unit 2002, and a recognition unit 2003.

[0209] The acquiring unit 2001 is configured to acquire a document image to be recognized including a document to be recognized.

[0210] The determining unit 2002 is configured to: determine a template image that matches the image of the document to be identified according to the type of the document to be identified, wherein the anchor point field of the template image is the same as the anchor point field of the image of the document to be identified.

[0211] The determining unit 2002 is further configured to determine the to-be-recognized area corresponding to the anchor field of the to-be-recognized document image based on the position information of the anchor field in the template image.

[0212] The recognition unit 2003 is configured to: recognize the area to be recognized and obtain the text content of the document image to be recognized.

[0213] In some embodiments, the acquisition unit 2001 is specifically configured to: acquire a target document image, determine a detection frame of the target document image, and acquire a document image to be identified from the target document image based on the detection frame.

[0214] In some embodiments, the determination unit 2002 is specifically configured to: obtain a horizontal line segment set and a vertical line segment set of the target document image according to a straight line segment detection algorithm. Merge the horizontal straight lines in the horizontal line segment set to obtain a plurality of merged horizontal straight lines. The angle between any two of the plurality of horizontal straight lines is greater than or equal to a preset angle, and the distance between them is greater than or equal to a preset distance. Merge the vertical straight lines in the vertical line segment set to obtain a plurality of merged vertical straight lines. The angle between any two of the plurality of vertical straight lines is greater than or equal to a preset angle, and the distance between them is greater than or equal to a preset distance. Determine the detection frame of the target document image based on the merged plurality of horizontal straight lines and the merged plurality of vertical straight lines.

[0215] In some embodiments, the determination unit 2002 is specifically configured to: determine a plurality of quadrilaterals based on the merged plurality of horizontal lines and the merged plurality of vertical lines, where a quadrilateral is formed by two lines from the merged plurality of horizontal lines and two lines from the merged plurality of vertical lines; and select a detection frame for the target document image from the plurality of quadrilaterals based on parameters of the plurality of quadrilaterals, where the parameters of the quadrilaterals include one or more of parallelism, area, and matching degree of the quadrilaterals.

[0216] In some embodiments, the determining unit 2002 is specifically configured to: determine the position information of the anchor field in the document image to be recognized based on the position information of the anchor field in the template image, determine the text recognition area corresponding to the anchor field based on the position information of the anchor field in the document image to be recognized, and determine the area to be recognized within the text recognition area based on the mapping relationship corresponding to the anchor field.

[0217] In some embodiments, the mapping relationship corresponding to the anchor field includes a correspondence between the position information of the anchor field in the template image and the position information of the anchor field in the document image to be recognized. Determination unit 2002 is specifically configured to: determine the position information of the anchor field in the area to be recognized based on the position information of the anchor field in the template image, and determine the area to be recognized based on the correspondence between the anchor field and the corresponding text content.

[0218] In some embodiments, the recognition unit 2003 is specifically configured to: if the anchor field in the template image matches the anchor field of the document image to be recognized, recognize the area to be recognized to obtain the text content of the document image to be recognized.

[0219] In some embodiments, the determining unit 2002 is further configured to determine whether the template image matches the document image to be identified based on the text content included in the anchor field of the template image and the text content included in the anchor field of the document image to be identified. If the cosine similarity between the text content included in the anchor field of the template image and the text content included in the anchor field of the document image to be identified is greater than or equal to a threshold, then the template image matches the document image to be identified. If the cosine similarity between the text content included in the anchor field of the template image and the text content included in the anchor field of the document image to be identified is less than the threshold, then the template image does not match the document image to be identified.

[0220] In some embodiments, the determination unit 2002 is specifically configured to: determine a first coordinate point and a second coordinate point of the document image to be recognized based on position information of an anchor point field in the document image to be recognized, where the position information of the first coordinate point includes the minimum horizontal coordinate and the minimum vertical coordinate of the anchor point field in a preset coordinate system, and the position information of the second coordinate point includes the maximum horizontal coordinate and the maximum vertical coordinate of the anchor point field in the preset coordinate system. Determine an area in the document image to be recognized where a matrix constructed with the first coordinate point and the second coordinate point as diagonals is located as a text recognition area.

[0221] In some embodiments, the determination unit 2002 is specifically configured to: input the document image to be recognized into the single-word text detection model, obtain position information of each field in the document image to be recognized, and determine the anchor field among the multiple fields of the document image to be recognized based on the position information of each field in the document image to be recognized and the position information of the anchor field of the template image. Based on the position information of the anchor field of the document image to be recognized, determine the to-be-recognized area corresponding to the anchor field of the document image to be recognized.

[0222] In some embodiments, the determination unit 2002 is further configured to input the image of the document to be identified into a document type determination model to obtain the type of the document to be identified. The determination unit is specifically configured to select, based on the type of the document to be identified, a template image of the same type as the document to be identified from a plurality of calibrated template images. The calibrated target image refers to an image in which the position information of the anchor field and the recognition area corresponding to the anchor field are calibrated.

[0223] In some embodiments, the acquisition unit 2001 is further configured to: acquire multiple types of template images. The determination unit 2002 is further configured to: determine the position information of the anchor field of each type of template image and the recognition area corresponding to the anchor field in response to the calibration operation.

[0224] When implemented through hardware, the acquisition unit in the embodiment of the present application can be integrated into the communication interface, and the processing unit can be integrated into the processor. The specific implementation is shown in Figure 21.

[0225] FIG21 shows a processing device involved in the above-mentioned embodiment, which can be or be applied to the above-mentioned document image recognition device. The device includes a processor 2102 and a communication interface 2103. The processor 2102 is used to control and manage the actions of the device, for example, executing the steps performed by the above-mentioned determination unit 2002 or recognition unit 2003, and / or executing other processes of the technology described herein. The communication interface 2103 is used to support communication between the device and other network entities, for example, executing the steps performed by the above-mentioned acquisition unit 2001. The device may also include a memory 2101 and a bus 2104. The memory 2101 is used to store program code and data of the device.

[0226] Among them, the memory 2101 can be a memory in the device, etc., and the memory can include a volatile memory, such as a random access memory; the memory can also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory can also include a combination of the above types of memory.

[0227] The processor 2102 may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array, or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. The processor may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, or a combination of a DSP and a microprocessor.

[0228] Bus 2104 may be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 2104 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, FIG21 shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0229] The device in FIG21 may also be a chip, which includes one or more (including two) processors 2102 and a communication interface 2103 .

[0230] Optionally, the chip further includes a memory 2105, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor 2102. A portion of the memory 2105 may also include a non-volatile random access memory (NVRAM).

[0231] In some embodiments, the memory 2105 stores the following elements, execution modules or data structures, or a subset thereof, or an extended set thereof.

[0232] In the embodiment of the present application, the corresponding operation is performed by calling the operation instruction stored in the memory 2105 (the operation instruction may be stored in the operating system).

[0233] Some embodiments of the present disclosure provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium), which stores computer program instructions. When the computer program instructions are executed on a computer (e.g., a server), the computer executes the document image recognition method as described in any of the above embodiments.

[0234] Exemplarily, the above-mentioned computer-readable storage media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes, etc.), optical disks (e.g., CDs (Compact Disks), DVDs (Digital Versatile Disks), etc.), smart cards, and flash memory devices (e.g., EPROMs (Erasable Programmable Read-Only Memory), cards, sticks, or key drives, etc.). The various computer-readable storage media described in the present disclosure may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0235] Some embodiments of the present disclosure further provide a computer program product, for example, stored on a non-transitory computer-readable storage medium. The computer program product includes computer program instructions that, when executed on a computer (e.g., a server), cause the computer to perform the document image recognition method described in the above embodiments.

[0236] Some embodiments of the present disclosure further provide a computer program. When the computer program is executed on a computer (eg, a server), the computer program enables the computer to execute the document image recognition method described in the above embodiments.

[0237] The beneficial effects of the above-mentioned computer-readable storage medium, computer program product and computer program are the same as the beneficial effects of the document image recognition method described in some of the above-mentioned embodiments, and will not be repeated here.

[0238] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0239] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0240] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0241] The above description is merely a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that a person skilled in the art can conceive within the technical scope disclosed in the present disclosure should be included within the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A method for recognizing a document image, wherein: The method comprises: Acquire a document image to be identified, wherein the document image to be identified includes a document to be identified; Determining a template image that matches the image of the document to be identified according to the type of the document to be identified; the anchor point field of the template image is the same as the anchor point field of the image of the document to be identified; Based on the position information of the anchor field in the template image, determining the to-be-recognized area corresponding to the anchor field of the to-be-recognized document image; The area to be identified is identified to obtain text content contained in the document image to be identified.

2. The method according to claim 1, wherein: The step of obtaining the document image to be identified comprises: Acquire a target document image, and determine a detection frame of the target document image; According to the detection frame, the document image to be identified is acquired from the target document image.

3. The method according to claim 2, wherein: Determining a detection frame of the target document image includes: According to the straight line segment detection algorithm, a first direction line segment set and a second direction line segment set of the target document image are obtained, wherein the first direction line segment set includes a plurality of first straight lines, and the angles between the plurality of first straight lines and the first direction are less than a first preset angle, and the second direction line segment set includes a plurality of second straight lines, and the angles between the plurality of second straight lines and the second direction are less than the first preset angle, and the angle between the first direction and the second direction is 90 degrees; Merging a plurality of first straight lines in the first direction line segment set to obtain a plurality of merged straight lines in the first direction, wherein an angle between any two straight lines in the plurality of first direction straight lines is greater than or equal to a second preset angle, and a distance between any two straight lines is greater than or equal to a preset distance; Merging multiple second straight lines in the second direction line segment set to obtain a plurality of merged second direction straight lines, wherein the included angle between any two straight lines in the plurality of second direction straight lines is greater than or equal to the second preset angle, and the distance between any two straight lines is greater than or equal to the preset distance; A detection frame of the target document image is determined according to the merged multiple straight lines in the first direction and the merged multiple straight lines in the second direction.

4. The method according to claim 3, wherein: The step of determining the detection frame of the target document image according to the merged plurality of straight lines in the first direction and the merged plurality of straight lines in the second direction comprises: Determine a plurality of quadrilaterals according to the merged plurality of straight lines in the first direction and the merged plurality of straight lines in the second direction, wherein one of the quadrilaterals is formed by two straight lines in the merged plurality of straight lines in the first direction and two straight lines in the merged plurality of straight lines in the second direction; A detection frame of the target document image is selected from the plurality of quadrilaterals according to parameters of the plurality of quadrilaterals, wherein the parameters include one or more of parallelism, area, and matching degree of the quadrilaterals.

5. The method according to any one of claims 1 to 4, wherein: Determining the to-be-recognized area corresponding to the anchor field of the to-be-recognized document image based on the position information of the anchor field in the template image includes: Determining the position information of the anchor field in the to-be-recognized document image based on the position information of the anchor field in the template image; Determining a text recognition area corresponding to the anchor field according to position information of the anchor field in the document image to be recognized; According to the mapping relationship corresponding to the anchor point field, the area to be recognized in the text recognition area is determined.

6. The method according to claim 5, wherein: The mapping relationship corresponding to the anchor point field includes a corresponding relationship between the position information of the anchor point field in the template image and the position information of the anchor point field in the document image to be recognized; The determining the area to be recognized in the text recognition area according to the mapping relationship corresponding to the anchor point field includes: The position information of the anchor field in the area to be identified is determined according to the position information of the anchor field in the template image, and the area to be identified is determined according to the correspondence between the anchor field and the corresponding text content area.

7. The method according to any one of claims 1 to 6, wherein: The step of identifying the area to be identified to obtain text content of the document image to be identified includes: If the anchor field in the template image matches the anchor field of the document image to be identified, the area to be identified is identified to obtain the text content of the document image to be identified.

8. The method according to claim 7, wherein: The method further comprises: Determining whether the template image matches the document image to be identified based on the text content included in the anchor field of the template image and the text content included in the anchor field of the document image to be identified; Among them, if the cosine similarity between the text content included in the anchor field of the template image and the text content included in the anchor field of the document image to be identified is greater than or equal to a threshold, then the template image matches the document image to be identified; if the cosine similarity between the text content included in the anchor field of the template image and the text content included in the anchor field of the document image to be identified is less than the threshold, then the template image does not match the document image to be identified.

9. The method according to claim 5, wherein: The step of determining the text recognition area corresponding to the anchor field according to the position information of the anchor field in the document image to be recognized includes: According to the position information of the anchor point field in the document image to be identified, a first coordinate point and a second coordinate point of the document image to be identified are determined, wherein the position information of the first coordinate point includes the anchor point field. the minimum horizontal coordinate and the minimum vertical coordinate of the point field in the preset coordinate system, and the position information of the second coordinate point includes the maximum horizontal coordinate and the maximum vertical coordinate of the anchor point field in the preset coordinate system; An area in the document image to be recognized where a matrix constructed with the first coordinate point and the second coordinate point as a diagonal line is located is determined as the text recognition area.

10. The method according to any one of claims 1 to 9, wherein: The determining, based on the position information of the anchor field in the template image, the area to be identified corresponding to the anchor field of the document image to be identified includes: Inputting the document image to be recognized into a single-word text detection model to obtain position information of each field in the document image to be recognized; Determining an anchor field among multiple fields of the document image to be identified according to position information of each field in the document image to be identified and position information of the anchor field of the template image; According to the position information of the anchor point field of the document image to be recognized, the to-be-recognized area corresponding to the anchor point field of the document image to be recognized is determined.

11. The method according to any one of claims 1 to 10, wherein: The method further comprises: Inputting the to-be-recognized document image into a document type determination model to obtain the type of the to-be-recognized document; The step of determining a template image that matches the image of the document to be identified according to the type of the document to be identified includes: According to the type of the document to be identified, a template image of the same type as the document to be identified is selected from a plurality of calibrated template images; wherein the calibrated template image refers to an image in which the position information of the anchor field and the identification area corresponding to the anchor field are calibrated.

12. The method according to any one of claims 1 to 11, wherein: The method further comprises: Get multiple types of template images; In response to the calibration operation, the position information of the anchor field of each type of template image and the recognition area corresponding to the anchor field are determined.

13. A document image recognition device, wherein: The identification device comprises: An acquisition unit is configured to acquire a document image to be recognized, wherein the document image to be recognized includes a document to be recognized; A determination unit configured to determine, according to the type of the document to be identified, a template image that matches the image of the document to be identified; the anchor point field of the template image is the same as the anchor point field of the image of the document to be identified; The determining unit is further configured to determine the to-be-recognized area corresponding to the anchor field of the to-be-recognized document image based on the position information of the anchor field in the template image; The recognition unit is configured to recognize the area to be recognized and obtain the text content of the document image to be recognized.

14. A processing device, wherein: include: Processor and memory; The memory stores computer instructions, and the processor is used to run the computer program to implement the document image recognition method according to any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium, wherein: The computer-readable storage medium stores instructions. When a computer executes the instructions, the computer executes the document image recognition method according to any one of claims 1 to 12.

Images