ID card layout analysis method based on target detection and key point detection

Abstract

This invention discloses a document layout analysis method based on object detection and keypoint detection, comprising: Step A: acquiring a color document image and labeling the document image; Step B: extracting feature information from the color document image to obtain coded features; Step C: adding a coordinate width loss function to the loss function to calculate the loss of image features and corresponding label information; Step D: predicting the result of the document image using object detection and keypoint detection algorithms, including title coordinates, content coordinates, and the coordinates of keypoint information corresponding to the content coordinates, to obtain the text line boxes of the content column and the title column, as well as the keypoint positions corresponding to the content column; Step E: using a character recognition algorithm to perform full-page text line recognition, and outputting full-page recognition information in a structured manner based on the relationship between the title column and the corresponding content column obtained in Step D.

Description

Technical Field

[0001] The present invention relates to the technical field of certificate recognition, and in particular to a certificate layout analysis method based on target detection and key point detection. Background Art

[0002] With the trend of globalization, more and more foreign friends will come to my country for study and tourism. Passports are essential personal documents. For full-page passport recognition, the overall layout distribution of the passport is an important link for the final recognition result. The existing passport layout analysis algorithms are mainly based on the transformer large model, or by collecting existing document layouts for layout analysis. Among them, the large model-based method has limited usage scenarios and cannot meet the needs of small embedded platforms. The method based on collecting existing document layouts for layout analysis or the template matching method, considering that there are hundreds of countries in the world, each country uses different passports according to specific circumstances and the nature of departure. In addition, due to the update and iteration of passport anti-counterfeiting technology, passport templates not only have the characteristics of a large number, but also new passport templates will continue to emerge due to the migration of time. This brings many immeasurable potential risks to the collection of passport templates, and the system with pictures as input is often accompanied by problems such as deformation, blur, and interference information. Each template requires a large number of steps for tuning, and its scalability is very poor. Summary of the invention

[0003] One object of the present invention is to provide a document layout analysis method based on target detection and key point detection, which does not require the use of a large model to classify the content area and the title area, saving a large amount of data collection time, hardware resources and computing power costs. In the case of a large number of document layouts, there is no need to identify text lines one by one. Instead, the overall layout content relationship can be determined by identifying specific content, thereby extracting effective content and quickly identifying the entire page information of the document.

[0004] Other advantages and features of the present invention will become more apparent from the following detailed description and may be realized by means of the instrumentalities and combinations particularly pointed out in the appended claims.

[0005] According to one aspect of the present invention, a document layout analysis method based on target detection and key point detection of the present invention, which can achieve the aforementioned objectives and other objectives and advantages, includes:

[0006] Step A: Collect a color document image and label the document image;

[0007] Step B: extracting feature information of the color document image and obtaining coding features for the document image;

[0008] Step C: Add the coordinate width loss function to the loss function and calculate the loss of image features and corresponding label information;

[0009] Step D: Through the target detection and key point detection algorithms, the result prediction is performed on the document image, including the title coordinates and the content coordinates, as well as the coordinates of the key point information corresponding to the content coordinates, and the result prediction is performed to obtain the text line frame where the content column is located and the text line frame of the title column, as well as the key point position corresponding to the content column;

[0010] Step E: Use a character recognition algorithm to perform full-page text line recognition, obtain the relationship between the title bar and the corresponding content bar according to step D, and output the full-page recognition information in a structured manner.

[0011] According to an embodiment of the present invention, the step A includes a sub-step A1: the requirement for labeling the document image is: classifying the document image according to the title bar and the content bar.

[0012] According to one embodiment of the present invention, step A includes sub-step A2: the content bar is a content text line corresponding to each title bar, and the title bar and the content bar are both marked in the form of a rectangular frame, and the center point of the title bar corresponding to each content bar is marked as the key point corresponding to each content bar.

[0013] According to one embodiment of the present invention, step B includes sub-step B1: sending the document image I into the darknet network backbone, performing feature encoding to obtain image feature I1, and using the deep learning network structure to expand and optimize the feature, the formula of the image feature is I1=encode(I).

[0014] According to an embodiment of the present invention, step C includes sub-step C1: inputting the image feature I1 and label information into the loss function to perform loss calculation, and performing back propagation optimization algorithm effect on the obtained loss result.

[0015] According to an embodiment of the present invention, the step D includes a sub-step D1: distinguishing the title bar from the content bar, and clarifying the corresponding relationship between the title bar and the content bar through the corresponding title bar pointed to by the key point of the content bar.

[0016] According to one embodiment of the present invention, step E includes sub-step E1: using a character recognition algorithm to identify the text line information of the entire page, and distinguishing the title bar and the corresponding content bar of the text line box through the positions of the title bar and content bar obtained by step D1.

[0017] According to an embodiment of the present invention, step E includes sub-step E2: simultaneously judging the title bar corresponding to the content bar by the location of the key point, obtaining the layout analysis result when performing layout analysis, and outputting the recognition content result in a structured manner.

[0018] According to one embodiment of the present invention, step C includes sub-step C2: the loss function is calculated as follows:

[0019] loss=loss box +loss pose +loss dw +loss cls +loss obj

[0020] loss box is the coordinate box regression loss, and the SIOU loss function is used. The formula of the SIOU loss function is:

[0021]

[0022] in,

[0023]

[0024] Among them, B represents the predicted coordinate frame, B GT Represents the coordinate frame of the label annotation, ∩ represents the intersection of two coordinate frames, ∪ represents the union of two coordinate frames, |.| represents taking the absolute value, Δ represents the angle loss between the predicted label and the annotated label, Ω represents the shape loss,

[0025] loss pose is the key point position loss, and the cross entropy loss function is used to optimize the classification results of the key points. The formula of the key point position loss function is:

[0026]

[0027] Among them, n is the number of detected targets, x i Represents the category confidence, y i is the corresponding label, i represents the feature map index, and the value range of i is between 1 and n.

[0028] loss dw The coordinate width loss is calculated by using the L1 loss function. The distance loss is calculated for the width of the coordinate boxes of the target result and the predicted result. The coordinate width loss function of the L1 loss function is:

[0029]

[0030] Among them, m represents the number of coordinate frames, W gtRepresents the width of the label coordinate frame, W pre represents the width of the predicted coordinate frame, |.| represents the absolute value result, i represents the coordinate frame index, and the value range of i is between 1 and m.

[0031] loss cls is the category loss function, and the cross entropy loss function is used to determine the target category. The formula is

[0032]

[0033] Among them, n is the number of detected targets, y i is the corresponding label, x i Represents category confidence, i represents the feature map index, and the value range of i is between 1 and n.

[0034] loss obj The key point target loss is calculated using the key point loss prediction result and the actual label result. The cross entropy loss function is

[0035]

[0036] Among them, n is the number of detected targets, y i is the corresponding label, x i Represents category confidence, i represents the feature map index, and the value range of i is between 1 and n.

[0037] According to an embodiment of the present invention, the step C includes a sub-step C3: the width values ​​of the label coordinate frame and the predicted coordinate frame are obtained through coordinate calculation.

[0038] The beneficial effects of the present invention are as follows: based on the target detection algorithm, a key point detection branch is added to classify and locate the certificate title and identification content, optimize the overall layout analysis algorithm, identify and locate the overall information of the certificate, and be able to simultaneously confirm the coordinate information of the certificate title and certificate content area, and form a one-to-one correspondence between the title bar and the content bar, and be able to form an overall layout logical relationship, realize structured output of information on the certificate page, save storage space of the operating platform, and speed up the overall testing time. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flowchart of the steps of a method for analyzing a certificate layout based on target detection and key point detection according to an embodiment of the present invention.

[0040] Figure 2 This is an algorithm flow chart of the document layout analysis method based on target detection and key point detection in the above embodiment of the present invention.

[0041] Figure 3Schematic diagram of network connection of the document layout analysis method based on target detection and key point detection in the above embodiment of the present invention.

[0042] Figure 4 It is a test schematic diagram of the document layout analysis method based on target detection and key point detection of the above embodiment of the present invention.

[0043] Figure 5 It is a schematic diagram of text line box detection results of the document layout analysis method based on target detection and key point detection in the above embodiment of the present invention.

[0044] Figure 6 This is a schematic diagram of the algorithms of steps 2 to 4 of the document layout analysis method based on target detection and key point detection in the above embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.

[0046] Those skilled in the art should understand that, in the disclosure of the present invention, the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the above terms should not be understood as limiting the present invention.

[0047] It is understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "one" cannot be understood as a limitation on the number. In addition, in the description of the present invention, unless otherwise specified, the meaning of "multiple" is two or more than two.

[0048] In the description of the present invention, it should be understood that "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through a medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0049] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0050] like Figures 1 to 6 As shown, a document layout analysis method based on target detection and key point detection of an embodiment of the present invention is disclosed. The document layout analysis method based on target detection and key point detection can be applied to various types of documents, such as passports, to identify and locate the overall information of the document. Specifically, the document title and identification content are classified and located, and the coordinate information of the document title and the document content area can be confirmed at the same time, and a one-to-one correspondence between the title bar and the content bar is formed, thereby generating a layout logic relationship of the entire document, realizing structured output of information on the document page, saving storage space of the operating platform, accelerating the overall test time, and improving recognition accuracy. The document layout analysis method based on target detection and key point detection includes the following steps:

[0051] Step 1: Use imaging devices, such as mobile phones and document readers, to collect and annotate the image of the document, collect a color document image, and label the document image. It is worth noting that the requirements for labeling the document image are: classify the document image according to the title bar and content bar, which can be divided into two categories, including but not limited to the title bar and the content bar. The title bar can include but is not limited to: name, nationality, date of birth, etc. The content bar is a content text line corresponding to each title bar. The title bar and the content bar are both annotated in the form of a rectangular frame, and the center point of each content bar corresponding to the title bar is used as the key point corresponding to each content bar for annotation;

[0052] Step 2: Extract the feature information of the color ID image and obtain the encoding feature of the ID image. Specifically, send the ID image I to the darknet network backbone for feature encoding to obtain the image feature I1, and use the deep learning network structure, such as yolov8n-p2 and PAN (Path Aggregation Network)-FPN (Feature PyramidNetwork feature pyramid network) is used to reduce missed detection of text lines, and the FPN and PAN structures are used to perform multi-scale fusion of features. The upper feature map has a deeper network layer and contains stronger semantic information of text lines, while the lower feature has less position information loss of text lines because it passes through fewer convolutional layers. The FPN result is upsampled by self-directional down-sampling, so that the bottom feature map contains stronger semantic information of text lines. The PAN mechanism samples from low to high, so that the top feature contains more position information of text lines. The two features are finally fused, so that different sizes ensure accurate prediction of text lines. The formula of image features is I1=encode(I), which is used for subsequent target detection and key point detection. In order to reduce the missed detection problem of short text lines (including title columns and content columns) in the present invention, the backbone network structure uses the yolov8n-p2 network structure. Yolov8n-p2 adds one upsampling and merges with other features to obtain feature output for small targets. The network connection diagram of PAN and FPN is as follows Figure 3 As shown:

[0053] Step 3: Add the coordinate width loss function to the loss function, calculate the loss of the image feature I1 and the corresponding label information, send the image feature I1 and the label information into the loss function for loss calculation, and perform the back propagation optimization algorithm effect on the obtained loss result. In detail, the calculation method of the loss function is

[0054] loss=loss box +loss pose +loss dw +losscls +loss obj

[0055] loss box is the coordinate frame regression loss. Preferably, the SIOU loss function is used. The formula of the SIOU loss function is:

[0056]

[0057] in,

[0058]

[0059] Among them, B represents the predicted coordinate frame, B GT Represents the coordinate frame of the label annotation, ∩ represents the intersection of two coordinate frames, ∪ represents the union of two coordinate frames, |.| represents taking the absolute value, Δ represents the angle loss between the predicted label and the annotated label, Ω represents the shape loss,

[0060] loss pose is the key point position loss, and the cross entropy loss function is used to optimize the classification results of the key points. The formula of the key point position loss function is:

[0061]

[0062] Among them, n is the number of detected targets, x i Represents the category confidence, y i is the corresponding label, i represents the feature map index, and the value range of i is between 1 and n.

[0063] loss dw It is the coordinate width loss. Preferably, the L1 loss function is used to calculate the distance loss for the width of the coordinate boxes of the target result and the predicted result. The coordinate width loss function of the L1 loss function is:

[0064]

[0065] Among them, m represents the number of coordinate frames, W gt Represents the width of the label coordinate frame, W pre It represents the width of the predicted coordinate frame. Furthermore, the width values ​​of the label coordinate frame and the predicted coordinate frame are calculated through coordinates. |.| represents the absolute value result. i represents the coordinate frame index. The value range of i is between 1 and m.

[0066] loss cls is the category loss function, and the cross entropy loss function is used to determine the target category. The formula is

[0067]

[0068] Among them, n is the number of detected targets, y i is the corresponding label, x i Represents category confidence, i represents the feature map index, and the value range of i is between 1 and n.

[0069] loss obj The key point target loss is calculated using the key point loss prediction result and the actual label result. The cross entropy loss function is

[0070]

[0071] Among them, n is the number of detected targets, y i is the corresponding label, x i Represents category confidence, i represents the feature map index, and the value range of i is between 1 and n;

[0072] Step 4: Use the object detection and key point detection algorithms learned in training to predict the results of the document image, including the title coordinates and content coordinates, as well as the coordinates of the key point information corresponding to the content coordinates. Predict the results and obtain the text line box where the content column is located and the text line box of the title column, as well as the key point position corresponding to the content column. Test image I, such as Figure 4 As shown, it is input into the target detection and key point detection algorithm, and the resulting image is as follows Figure 5 As shown, '0' represents the title bar class in the prediction result, '1' represents the content bar class in the prediction result, '●' represents the predicted key point class, and represents the association relationship between the content bar and each title bar, that is, in the result image, the title bar and the content bar can be directly distinguished, and the corresponding relationship between the title bar and the content bar can be clarified through the corresponding title bar pointed to by the key point of the content bar. During character recognition, it is more intuitive and clear to provide users with the character information of the title and the character content information corresponding to the title, that is, the relationship between the title bar and the content bar is confirmed;

[0073] Step 5: Use a character recognition algorithm to identify the entire page of text lines. According to the relationship between the title bar and the corresponding content bar obtained in step 4, output the entire page recognition information in a structured manner. Specifically, use a character recognition algorithm, such as CRNN, to identify the entire page of text line information. Through the positions of the title bar and content bar obtained in step 4, distinguish the title bar of the text line box and the corresponding content bar. At the same time, through the location of the key points, determine the title bar corresponding to the content bar. When performing layout analysis, the layout analysis results can be quickly obtained. Finally, the recognition content results are output in a structured manner, such as type: X, country code: Y, document number: AA, etc., and transmitted to the inspection operator to complete the identity verification work.

Claims

1. A document layout analysis method based on target detection and key point detection, characterized in that: include: Step A: Collect a color ID image and label the ID image; Step B: extracting feature information of the color document image and obtaining coding features for the document image; Step C: Add the coordinate width loss function to the loss function and calculate the loss of image features and corresponding label information; The step C includes sub-step C2: the loss function is calculated as follows: loss=loss box +loss pose +loss dw +loss cls +loss obj loss box is the coordinate box regression loss, and the SIOU loss function is used. The formula of the SIOU loss function is: in, Among them, B represents the predicted coordinate frame, B GT Represents the coordinate frame of the label annotation, ∩ represents the intersection of two coordinate frames, ∪ represents the union of two coordinate frames, |.| represents the absolute value result, Δ represents the angle loss between the predicted label and the annotated label, Ω represents the shape loss, loss pose is the key point position loss, and the cross entropy loss function is used to optimize the classification results of the key points. The formula of the key point position loss function is: Among them, n is the number of detected targets, x i Represents the category confidence, y i is the corresponding label, i represents the feature map index, and the value range of i is between 1 and n. loss dw The coordinate width loss is calculated by using the L1 loss function. The distance loss is calculated for the width of the coordinate boxes of the target result and the predicted result. The coordinate width loss function of the L1 loss function is: Among them, m represents the number of coordinate frames, W gt Represents the width of the label coordinate frame, W pre represents the width of the predicted coordinate frame, |.| represents the absolute value result, i represents the coordinate frame index, and the value range of i is between 1 and m. loss cls is the category loss function, and the cross entropy loss function is used to determine the target category. The formula is Among them, n is the number of detected targets, y i is the corresponding label, x i Represents category confidence, i represents the feature map index, and the value range of i is between 1 and n. loss obj The key point target loss is calculated using the key point loss prediction result and the actual label result. The cross entropy loss function is Among them, n is the number of detected targets, y i is the corresponding label, x i Represents category confidence, i represents the feature map index, and the value range of i is between 1 and n; Step D: Through the target detection and key point detection algorithms, the result prediction is performed on the document image, including the title coordinates and the content coordinates, as well as the coordinates of the key point information corresponding to the content coordinates, and the result prediction is performed to obtain the text line frame where the content column is located and the text line frame of the title column, as well as the key point position corresponding to the content column; Step E: Use a character recognition algorithm to perform full-page text line recognition, obtain the relationship between the title bar and the corresponding content bar according to step D, and output the full-page recognition information in a structured manner.

2. The document layout analysis method according to claim 1, characterized in that: The step A includes sub-step A1: the requirement for labeling the document image is: classifying the document image according to the title column and content column.

3. The document layout analysis method according to claim 2, characterized in that: The step A includes sub-step A2: the content bar is a content text line corresponding to each title bar, the title bar and the content bar are both marked in the form of a rectangular frame, and the center point of the title bar corresponding to each content bar is marked as the key point corresponding to each content bar.

4. The document layout analysis method according to claim 3, characterized in that: The step B includes sub-step B1: sending the document image I into the darknet network backbone, performing feature encoding, obtaining the image feature I1, and using the deep learning network structure to expand and optimize the feature, the formula of the image feature is I1=encode(I).

5. The document layout analysis method according to claim 4, characterized in that: The step C includes sub-step C1: inputting the image feature I1 and label information into the loss function to perform loss calculation, and performing back propagation optimization algorithm effect on the obtained loss result.

6. The document layout analysis method according to claim 5, characterized in that: The step D includes a sub-step D1: distinguishing the title bar from the content bar, and clarifying the corresponding relationship between the title bar and the content bar through the corresponding title bar pointed to by the key point of the content bar.

7. The document layout analysis method according to claim 6, characterized in that: The step E includes sub-step E1: using a character recognition algorithm to recognize the text line information of the entire page, and distinguishing the title bar and the corresponding content bar of the text line frame through the positions of the title bar and the content bar obtained in step D1.

8. The document layout analysis method according to claim 7, characterized in that: The step E includes sub-step E2: judging the title bar corresponding to the content bar by the location of the key point at the same time, obtaining the layout analysis result when performing layout analysis, and outputting the recognition content result in a structured manner.

9. The document layout analysis method according to claim 1, characterized in that: The step C includes a sub-step C3: the width values ​​of the label coordinate frame and the predicted coordinate frame are obtained through coordinate calculation.

Images