Image text line detection method and device

Abstract

The application provides an image text line detection method and device, which comprises the following steps: obtaining a bill image to be detected; inputting the bill image into a trained text line detection model, extracting features through a feature extraction network, inputting the features into a first printing branch and a second printing branch respectively, and outputting classification results of first printing text and background and classification results of second printing text and background respectively; wherein, the first printing branch and the second printing branch both comprise a convolution layer, a feature fusion layer and a deconvolution layer in sequence from the input to the output direction; the feature fusion layers of the first printing branch and the second printing branch are both used for fusing feature maps processed by the convolution layers of the first printing branch and the second printing branch, and inputting the feature maps into the deconvolution layers of the respective branches. The method realizes accurate judgment of first printing and second printing text information through feature fusion and classification of twice printing text lines, and improves the accuracy of key information extraction after identification of twice printing information.

Description

Technical Field

[0001] This invention relates to the field of image recognition, and in particular to a method and apparatus for detecting text lines in an image. Background Technology

[0002] Invoice image recognition is a type of document image recognition, and text line localization is one of its key technologies. The recall and accuracy of localization determine the upper limit of recognition accuracy. Text line localization methods can be divided into two main categories. One category is based on traditional image processing and analysis methods, which extract text blocks through connected components and then cluster and connect them into text lines. The other category consists of deep learning methods that have emerged in recent years, training detection models through sample learning, such as PSENet, DbNet, and CTPN (Detecting Text in Natural Image with Connectionist Text Proposal Network), among other text line detection models and algorithms.

[0003] These algorithms include object detection-based methods that extract features through convolutional networks and then classify candidate regions to determine whether they are text regions. Others are based on image segmentation principles, extracting features through convolution and then using deconvolution to obtain the image segmentation result (i.e., the classification result for each pixel, where each pixel is divided into two categories: part of a text line or not). The former's detection accuracy is lower than that of image segmentation-based methods.

[0004] However, image processing-based methods are easily affected by image background and noise, resulting in unstable detection results and poor performance. Deep learning-based text line localization models significantly improve the accuracy of text line localization under complex conditions.

[0005] In image recognition applications for taxi receipts and common tax invoices, text lines are typically printed in two stages. The first printout is usually done directly during the receipt's production. The second printout is done during or at the end of the service. These two types of text lines are prone to merging and have distinct characteristics.

[0006] For text lines that are not pasted together, it is impossible to accurately determine whether they are reprinted text lines, thus failing to provide a basis for subsequent extraction of key information. Summary of the Invention

[0007] To address the problems existing in the prior art, the present invention provides an image text line detection method and apparatus.

[0008] This invention provides a method for detecting text lines in an image, comprising: acquiring a document image to be detected; inputting the document image to be detected into a trained text line detection model, extracting features through a feature extraction network, inputting the extracted features into a first-print branch and a second-print branch respectively, and outputting the classification results of the first-print text and background, and the classification results of the second-print text and background respectively; wherein, the first-print branch and the second-print branch each include a convolutional layer, a feature fusion layer and a deconvolutional layer in the order from input to output; the feature fusion layer of the first-print branch and the second-print branch are used to fuse the feature maps processed by the convolutional layers of the first-print branch and the second-print branch, and inputting them into the deconvolutional layer of their respective branches; the text line detection model is obtained by training on sample document images labeled with first-print text, second-print text and background.

[0009] According to the image text line detection method provided by the present invention, before acquiring the document image to be detected, the method further includes: training the first printing branch based on the first printing loss function according to the sample document image marked with first printing text, second printing text and background; wherein, the first printing loss function is determined according to a first part loss function, a second part loss function and a third part loss function; the first part loss function is determined according to the classification output of the first printing branch on the first printing text and background of the sample document image, the classification labels of the first printing text and background and the mask weight of the text line region of the first printing text; the second part loss function is the loss function for the first printing text line in the sample document image that does not have a part that is connected with the second printing text; and the third part loss function is the loss function corresponding to the part of the first printing text and the second printing text that are connected in the sample document image.

[0010] According to the image text line detection method provided by the present invention, before acquiring the document image to be detected, the method further includes: training the secondary printing branch based on a secondary printing loss function according to sample document images labeled with first-printed text, second-printed text, and background; wherein the secondary printing loss function is determined according to a fourth part loss function, a fifth part loss function, and a sixth part loss function; the fourth part loss function is determined according to the classification output of the secondary printing branch on the secondary printing text and background of the sample document image, the classification labels of the secondary printing text and background, and the mask weight of the text line region of the secondary printing text; the fifth part loss function is the loss function for the first-printed text line in the sample document image that does not have a part that is connected with the secondary printing text; and the sixth part loss function is the loss function corresponding to the part where the first-printed text and the secondary printing text line are connected in the sample document image.

[0011] According to the image text line detection method provided by the present invention, the first printing loss function and the second printing loss function respectively include:

[0012] L1 = L dice +αL11+βL1 12

[0013] L2 = L dice +λL12+γL1 12

[0014] Where L1 is the initial printing loss function, and L2 is the secondary printing loss function; α is the weight parameter for adjusting the weight of the non-adhesive portion in the initial printing, β is the weight parameter for adjusting the weight of the adhesive portion in the initial printing, λ is the weight parameter for adjusting the weight of the non-adhesive portion in the secondary printing, and γ is the weight parameter for adjusting the weight of the adhesive portion in the secondary printing; in the L1 loss function, L... dice L1 is the first part of the loss function, and L11 is the second part of the loss function. 12 This is the third part of the loss function; in the L2 loss function, L dice L1 is the fourth part of the loss function, and L11 is the fifth part of the loss function. 12 This is the loss function in Part VI.

[0015] According to the image text line detection method provided by the present invention, the first part of the loss function or the fourth part of the loss function includes:

[0016]

[0017] Where M is the weight of the mask matrix with the same size as the output image, and different non-zero values ​​are set according to the text line interior, text rectangle, and text line boundary; for the first part of the loss function, X is the classification result of the first printed text and background output by the first printing branch, and Y is the annotation result of the first printed text and background; for the fourth part of the loss function, X is the classification result of the first printed text and background output by the second printing branch, and Y is the annotation result of the second printed text and background.

[0018] According to the image text line detection method provided by the present invention, the number of convolutional filters in the convolutional layer of the secondary printing branch is greater than that in the first printing branch.

[0019] This invention also provides an image text line detection device, comprising: an acquisition module for acquiring a document image to be detected; and a processing module for inputting the document image to be detected into a trained text line detection model, extracting features through a feature extraction network, inputting the extracted features into a first-print branch and a second-print branch respectively, and outputting classification results of the first-print text and background, and classification results of the second-print text and background respectively; wherein, the first-print branch and the second-print branch each include a convolutional layer, a feature fusion layer, and a deconvolutional layer in the order from input to output; the feature fusion layer of the first-print branch and the second-print branch are both used to fuse the feature maps processed by the convolutional layers of the first-print branch and the second-print branch, and inputting them into the deconvolutional layer of their respective branches; the text line detection model is obtained by training on sample document images labeled with first-print text, second-print text, and background.

[0020] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image text line detection method as described above.

[0021] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image text line detection method as described above.

[0022] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image text line detection method as described above.

[0023] The image text line detection method and apparatus provided by this invention fuses the feature maps processed by the convolutional layers of the first and second printing branches through a feature fusion layer of the first and second printing branches. These feature maps are then input into the deconvolutional layers of their respective branches. This allows both the first and second printings to learn the features of the first and second printings respectively. Finally, by classifying the text lines printed in the two printings, the method accurately judges the text information of the first and second printings, greatly improving the accuracy of key information extraction after the recognition of the two printings. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1This is a flowchart illustrating the image text line detection method provided by the present invention;

[0026] Figure 2 This is a schematic diagram of the model structure of the image text line detection method provided by the present invention;

[0027] Figure 3 This is a schematic diagram of the rectangular box mask weight of the text line provided by the present invention;

[0028] Figure 4 This is a schematic diagram of the IOU provided by the present invention;

[0029] Figure 5 This is a schematic diagram of the image text line detection device provided by the present invention;

[0030] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0032] The following is combined with Figures 1-6 The present invention describes the image text line detection method and apparatus. Figure 1 This is a flowchart illustrating the image text line detection method provided by the present invention, as shown below. Figure 1 As shown, the present invention provides an image text line detection method, including:

[0033] 101. Obtain the image of the ticket to be inspected.

[0034] First, obtain an image of the ticket to be detected, such as a taxi ticket image.

[0035] 102. Input the image of the ticket to be detected into the trained text line detection model. After the feature extraction network extracts features, input the extracted features into the first printing branch and the second printing branch respectively, and output the classification results of the first printing text and background and the classification results of the second printing text and background respectively.

[0036] The first printing branch and the second printing branch each include a convolutional layer, a feature fusion layer, and a deconvolutional layer in the order from input to output. The feature fusion layer of the first printing branch and the second printing branch are used to fuse the feature maps processed by the convolutional layers of the first printing branch and the second printing branch, and input them into the deconvolutional layer of their respective branches. The text line detection model is obtained by training on sample ticket images marked with the first printing text, the second printing text, and the background.

[0037] Figure 2 This is a schematic diagram of the model structure of the image text line detection method provided by the present invention, as shown below. Figure 2 As shown, the detection model proposed in this invention is similar to the deep detection model. The feature extraction part can adopt common convolutional feature extraction structures such as ResNet or FPN, with the aim of extracting image features. These features can be understood as general features of image representation, such as image boundaries and some graphic structure features.

[0038] The first and second printing branches are used for text line detection, which is mainly based on the principle of image segmentation. Therefore, a convolution-deconvolution network structure is adopted. The output of the last deconvolution layer is the same size as the original image, and the result is a binary image. White pixels represent text lines and black pixels represent background areas.

[0039] Based on such Figure 2 The text line detection model is trained by labeling sample ticket images of the first printed text, the second printed text, and the background, resulting in the trained text line detection model in 102, which is used for text line detection on tickets.

[0040] The image text line detection method of the present invention integrates the feature maps processed by the convolutional layers of the first and second printing branches through a feature fusion layer of the first and second printing branches, and inputs them into the deconvolutional layer of their respective branches. This allows both the first and second printings to learn the features of the first and second printings respectively. Finally, by classifying the text lines printed in the two printings, the method can accurately judge the text information of the first and second printings, greatly improving the accuracy of key information extraction after the recognition of the two printings.

[0041] In one embodiment, before acquiring the ticket image to be detected, the method further includes: training the first printing branch based on the first printing loss function according to sample ticket images labeled with first printing text, second printing text, and background; wherein the first printing loss function is determined according to a first part loss function, a second part loss function, and a third part loss function; the first part loss function is determined according to the classification output of the first printing branch on the first printing text and background of the sample ticket image, the classification labels of the first printing text and background, and the mask weight of the text line region of the first printing text; the second part loss function is the loss function for the first printing text line in the sample ticket image that does not have a part that is connected with the second printing text; and the third part loss function is the loss function corresponding to the part of the first printing text and the second printing text that are connected in the sample ticket image.

[0042] Considering that existing deep learning text line detection models often misclassify multiple consecutive text lines as a single line, leading to detection errors, this invention provides a loss function to avoid detection errors when dealing with consecutive text lines. The initial print loss function trained on the initial print branch consists of three parts, with the first and third parts being crucial.

[0043] Figure 3 This is a schematic diagram of the rectangular box mask weight of text lines provided by the present invention, such as... Figure 3 As shown, a mask weight is set for the text line region. The calculation of the first part of the loss function in this invention takes into account the importance of the text line region, mainly reflected in the text line mask weight M. This part of the loss function can use the same form of loss function without distinguishing between the first printing and the second printing.

[0044] The second part, loss function L11, is the loss function for the first printed text line that is not attached to the secondary print. The loss function can be calculated by inputting X. i,j And label Y i,j The average absolute value of the difference between each pixel. The L11 loss function is calculated as follows:

[0045]

[0046] Where X represents the output of the first printed text line that is not attached to the second print, and Y represents the calibration result of the first printed text line that is not attached to the second print. H represents the height of the image, and W represents the width of the image.

[0047] Part Three: Loss Function L1 12The L1 loss corresponding to the overlapping part of the text lines in the first and second printings can be calculated by filtering overlapping rectangles based on the answer annotations of the text lines in the first and second printings. When the IOU between the two is greater than a preset threshold (such as 0.1), they are considered to be overlapping rectangles.

[0048] Figure 4 This is a schematic diagram of the IOU provided by the present invention, such as... Figure 4 As shown. In object detection, assuming box 1 is represented by [x1, y1, x2, y2], and similarly box 2 is represented by [x3, y3, x4, y4], as... Figure 4 As shown, the IOU calculation formula at this time is:

[0049] IOU = Intersection of two detection boxes / Sum of the areas of the intersection of the two detection boxes;

[0050] Then calculate the L1 loss of the corresponding text line detection results.

[0051] The image text line detection method of this invention improves the probability of text line detection through a first loss function, effectively distinguishes the contiguous parts through the second and third loss functions, and combines the above three loss functions to effectively avoid the problem of detecting multiple text lines as one text line when encountering contiguous text lines for the first time, thus causing detection errors.

[0052] In one embodiment, before acquiring the ticket image to be detected, the method further includes: training the secondary printing branch based on a secondary printing loss function using sample ticket images labeled with first-printed text, second-printed text, and background; wherein the secondary printing loss function is determined based on a fourth part loss function, a fifth part loss function, and a sixth part loss function; the fourth part loss function is determined based on the classification output of the secondary printing branch for the secondary printing text and background of the sample ticket image, the classification labels for the secondary printing text and background, and the mask weights of the text line regions of the secondary printing text; the fifth part loss function is the loss function for the first-printed text lines in the sample ticket image that do not have any adhering parts with the secondary printing text; and the sixth part loss function is the loss function corresponding to the adhering parts of the first-printed text and the secondary printing text lines in the sample ticket image.

[0053] The loss function for the second printing branch is formally the same as the loss function for the first printing branch.

[0054] Specifically, the fourth part is the loss function L. dice and the loss function L1 in Part 6 12 The calculation is the same as that in the first printed line detection branch.

[0055] L12 represents the L1 loss for text lines that were printed twice and were not attached to the first print. The calculations are as follows:

[0056]

[0057] Where X represents the output of the text line that was printed twice and is not attached to the first printout, Y represents the calibration result of the text line that was printed twice and is not attached to the first printout, H represents the height of the image, and W represents the width of the image.

[0058] In one embodiment, the first printing loss function and the second printing loss function respectively include:

[0059] L1 = L dice +αL11+βL1 12

[0060] L2 = L dice +λL12+γL1 12

[0061] Where L1 is the initial printing loss function, and L2 is the secondary printing loss function; α is the weight parameter for adjusting the weight of the non-adhesive portion in the initial printing, β is the weight parameter for adjusting the weight of the adhesive portion in the initial printing, λ is the weight parameter for adjusting the weight of the non-adhesive portion in the secondary printing, and γ is the weight parameter for adjusting the weight of the adhesive portion in the secondary printing; in the L1 loss function, L... dice L1 is the first part of the loss function, and L11 is the second part of the loss function. 12 This is the third part of the loss function; in the L2 loss function, L dice L1 is the fourth part of the loss function, and L11 is the fifth part of the loss function. 12 This is the loss function in Part VI.

[0062] Wherein, α and β are empirical parameters, generally between 0 and 1. In this invention, to improve the detection of adhering text lines in the first and second printings of the model pair, β can be set appropriately larger. λ and γ are empirical parameters, generally between 0 and 1. In this invention, to improve the detection of adhering text lines in the first and second printings of the model pair, γ can be set appropriately larger.

[0063] In this embodiment of the invention, the loss of non-adhesive and adhesive parts in the first and second printings can be adjusted by using the weight parameters α, β and λ, γ. Appropriate empirical parameters can be determined based on the actual training results, thereby improving the accuracy of adhesive part detection.

[0064] In one embodiment, the first part of the loss function or the fourth part of the loss function includes:

[0065]

[0066] Where M is the weight of the mask matrix with the same size as the output image, and different non-zero values ​​are set according to the text line interior, text rectangle, and text line boundary; for the first part of the loss function, X is the classification result of the first printed text and background output by the first printing branch, and Y is the annotation result of the first printed text and background; for the fourth part of the loss function, X is the classification result of the first printed text and background output by the second printing branch, and Y is the annotation result of the second printed text and background.

[0067] The Dice coefficient is a set similarity measure function, typically used to calculate the similarity between two sample points (ranging from 0 to 1). It is calculated as follows:

[0068] S=2|X∩Y| / |X|+|Y|

[0069] Here, |X∩Y| is the intersection of X and Y, and |X| and |Y| represent the number of elements in X and Y, respectively. The 2 in the numerator is because the powder contains elements that are common to X and Y and are counted repeatedly.

[0070] This invention improves upon the Dice coefficient by using L... dice The similarity measure between two sets of pixels (output and label) is defined as 1-S, which, combined with the mask matrix M, yields the loss function for the first or fourth part.

[0071] During calculation, the M matrix is ​​initialized to all zeros, and corresponding weights are assigned to the positions of the text-line answer-marked rectangles. For example, a text-line rectangle in M ​​is (x, y, w, h), with coordinates representing the top-left corner and the width and height of the rectangle, respectively. To improve the performance of detecting overlapping rectangles, the weights of the rectangle boundary regions can be increased by increasing the proportion of the corresponding loss in the overall loss. In specific calculations, such as... Figure 3 As shown, the text line mask matrix can be divided into the text line interior, the text rectangle, and the text line boundary. Each part has its own weight, with corresponding pixel weights of τ and 1-τ, respectively. τ can be 0.2 or 0.3, thus giving a larger weight to the text line boundary.

[0072] After calculating the weights of each text line mask, they are filled into the corresponding coordinates of the M matrix. Y represents the calibration result, typically using 1 to represent the inside of a text line and 0 to represent the outside of a text line. X can be a probability map or a binary map, both of which use larger values ​​to represent text lines. Based on experience, this invention recommends outputting the probability map X using softmax.

[0073] The image text line detection method of this invention determines the loss function of the first or fourth part based on the mask matrix, the output of the first or second printing branch and the label result, which can make the recognition of text boxes more accurate.

[0074] In one embodiment, the number of convolutional filters in the convolutional layer of the secondary printing branch is greater than that in the first printing branch.

[0075] Considering that the text features change little in the first printing of a ticket, with relatively small variations in font, color, and keyword length, while the font, scale, and color of the text lines change significantly in the second printing, and the feature variance is large, the second printing branch requires a larger network for feature extraction. Accordingly, this invention sets the number of convolutional filters in the convolutional layers of the second printing branch to be greater than that of the first printing branch. Optionally, the number of convolutional filters in the convolutional layers of the second printing branch is a preset multiple of that of the first printing branch.

[0076] For example, assuming that after general image feature extraction, the number of convolutional layers is set to N convolutional filters, and the features obtained by convolution are... Features after feature fusion are

[0077] After the secondary printing branch and general image feature extraction, the number of convolutional layers is set to M (2N~5N) convolutional filters, and the features obtained from the convolution are... Features after fusion

[0078] The image text line detection method of this invention improves the recognition accuracy of secondary printing by making the number of convolutional filters in the convolutional layer of the secondary printing branch greater than that in the first printing branch, or a preset multiple of that in the first printing branch.

[0079] The image text line detection device provided by the present invention is described below. The image text line detection device described below can be referred to in correspondence with the image text line detection method described above.

[0080] Figure 5 This is a schematic diagram of the image text line detection device provided by the present invention, as shown below. Figure 5As shown, the image text line detection device includes an acquisition module 501 and a processing module 502. The acquisition module 501 acquires the document image to be detected. The processing module 502 inputs the document image to be detected into a trained text line detection model. After feature extraction by a feature extraction network, the extracted features are input into the first printing branch and the second printing branch, respectively, and output the classification results of the first printed text and background, and the classification results of the second printed text and background, respectively. The first printing branch and the second printing branch each include a convolutional layer, a feature fusion layer, and a deconvolutional layer in the order from input to output. The feature fusion layer of both the first printing branch and the second printing branch is used to fuse the feature maps processed by the convolutional layers of the first printing branch and the second printing branch, and inputs them into the deconvolutional layer of their respective branches. The text line detection model is trained based on sample document images labeled with the first printed text, the second printed text, and the background.

[0081] The apparatus embodiments provided in this invention are for implementing the above-described method embodiments. For specific processes and details, please refer to the above-described method embodiments, which will not be repeated here.

[0082] The image text line detection device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned image text line detection method embodiment. For the sake of brevity, any parts not mentioned in the image text line detection device embodiment can be referred to the corresponding content in the aforementioned image text line detection method embodiment.

[0083] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 6As shown, the electronic device may include: a processor 601, a communication interface 602, a memory 603, and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604. The processor 601 can call logical instructions in the memory 603 to execute an image text line detection method. This method includes: acquiring a document image to be detected; inputting the document image to be detected into a trained text line detection model, extracting features through a feature extraction network, inputting the extracted features into a first-print branch and a second-print branch respectively, and outputting the classification results of the first-print text and background, and the classification results of the second-print text and background respectively; wherein, the first-print branch and the second-print branch each include a convolutional layer, a feature fusion layer, and a deconvolutional layer in the order from input to output; the feature fusion layer of the first-print branch and the second-print branch are used to fuse the feature maps processed by the convolutional layers of the first-print branch and the second-print branch, and inputting them into the deconvolutional layer of their respective branches; the text line detection model is obtained after training based on sample document images labeled with first-print text, second-print text, and background.

[0084] Furthermore, the logical instructions in the aforementioned memory 603 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0085] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image text line detection method provided by the above methods. The method includes: acquiring a ticket image to be detected; inputting the ticket image to be detected into a trained text line detection model, extracting features through a feature extraction network, inputting the extracted features into a first-print branch and a second-print branch respectively, and outputting the classification results of the first-print text and background, and the classification results of the second-print text and background respectively; wherein, the first-print branch and the second-print branch each include a convolutional layer, a feature fusion layer and a deconvolutional layer in the order from input to output; the feature fusion layer of the first-print branch and the second-print branch are both used to fuse the feature maps processed by the convolutional layers of the first-print branch and the second-print branch, and inputting them into the deconvolution layer of their respective branches; the text line detection model is obtained by training on sample ticket images labeled with first-print text, second-print text and background.

[0086] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the image text line detection method provided by the above methods. The method includes: acquiring a ticket image to be detected; inputting the ticket image to be detected into a trained text line detection model, extracting features through a feature extraction network, inputting the extracted features into a first-print branch and a second-print branch respectively, and outputting the classification results of the first-print text and background, and the classification results of the second-print text and background respectively; wherein, the first-print branch and the second-print branch each include a convolutional layer, a feature fusion layer, and a deconvolutional layer in the order from input to output; the feature fusion layer of the first-print branch and the second-print branch are both used to fuse the feature maps processed by the convolutional layers of the first-print branch and the second-print branch, and inputting them into the deconvolution layer of their respective branches; the text line detection model is obtained after training based on sample ticket images labeled with first-print text, second-print text, and background.

[0087] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0088] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of image line detection, characterized by, The method comprises the following steps: obtaining a bill image to be detected; inputting the bill image to be detected into a trained text line detection model, extracting features through a feature extraction network, inputting the extracted features into a first printing branch and a second printing branch respectively, and outputting classification results of first printing text and background and classification results of second printing text and background respectively; wherein the first printing branch and the second printing branch both comprise a convolution layer, a feature fusion layer and a deconvolution layer in sequence from the input to the output direction; the feature fusion layers of the first printing branch and the second printing branch are both used for fusing feature maps processed by the convolution layers of the first printing branch and the second printing branch, and inputting the feature maps into the deconvolution layers of the respective branches; the text line detection model is obtained by training based on sample bill images marked with first printing text, second printing text and background; before the step of obtaining the bill image to be detected, the method further comprises the following steps: training the first printing branch based on a first printing loss function according to sample bill images marked with first printing text, second printing text and background; wherein the first printing loss function is determined according to a first partial loss function, a second partial loss function and a third partial loss function; the first partial loss function is determined according to classification outputs of first printing text and background of the first printing branch on the sample bill image, classification labels of the first printing text and the background, and a text line area mask weight of the first printing text; the second partial loss function is a loss function of a part of the first printing text line in the sample bill image that is not adhered to the second printing text; and the third partial loss function is a loss function corresponding to a part of the first printing text and the second printing text that are adhered to each other in the sample bill image; before the step of obtaining the bill image to be detected, the method further comprises the following steps: training the second printing branch based on a second printing loss function according to sample bill images marked with first printing text, second printing text and background; wherein the second printing loss function is determined according to a fourth partial loss function, a fifth partial loss function and a sixth partial loss function; the fourth partial loss function is determined according to classification outputs of second printing text and background of the second printing branch on the sample bill image, classification labels of the second printing text and the background, and a text line area mask weight of the second printing text; the fifth partial loss function is a loss function of a part of the first printing text line in the sample bill image that is not adhered to the second printing text; and the sixth partial loss function is a loss function corresponding to a part of the first printing text and the second printing text that are adhered to each other in the sample bill image.

2. The image line detection method according to claim 1, characterized in that, The first printing loss function and the second printing loss function each comprise: L1 = L dice + αL11 + βL1 12 L2 = L dice + λL12+ γL1 12 Wherein, L1 is the first printing loss function, L2 is the second printing loss function; α is the weight parameter of adjusting the first printing non-adhesion part, β is the weight parameter of adjusting the first printing adhesion part loss, λ is the weight parameter of adjusting the second printing non-adhesion part, γ is the weight parameter of adjusting the second printing adhesion part loss; in L1 loss function, L dice is the first part loss function, L11 is the second part loss function, L1 12 is the third part loss function; in L2 loss function, L dice is the fourth part loss function, L11 is the fifth part loss function, L1 12 is the sixth part loss function.

3. The image line detection method according to claim 2, characterized in that, the first partial loss function or the fourth partial loss function comprises: Wherein, M is the weight of the mask matrix with the same size as the output image, different non-zero values are set according to the internal part of the text line, the text rectangular frame and the boundary part of the text line; for the first part of the loss function, X is the classification result of the first printing text and background output by the first printing branch, Y is the annotation result of the first printing text and background; for the fourth part of the loss function, X is the classification result of the first printing text and background output by the second printing branch, Y is the annotation result of the second printing text and background.

4. The image line detection method of claim 1, wherein The number of convolution layer convolution filters of the second printing branch is greater than that of the first printing branch.

5. An image line detection apparatus characterized by comprising: Comprise: The acquisition module is used for acquiring the bill image to be detected. The processing module is used for inputting the bill image to be detected into the trained text line detection model, extracting features through the feature extraction network, inputting the extracted features into the first printing branch and the second printing branch respectively, and outputting the classification results of the first printing text and background and the classification results of the second printing text and background respectively. Wherein, the order of the first printing branch and the second printing branch from input to output direction includes convolution layer, feature fusion layer and deconvolution layer in turn; the feature fusion layer of the first printing branch and the second printing branch is used for fusing the feature maps processed by the convolution layer of the first printing branch and the convolution layer of the second printing branch, and inputting into the deconvolution layer of the branch respectively; the text line detection model is obtained after being trained according to the sample bill image marked with the first printing text, the second printing text and the background; The image text line detection device further comprises: The training unit is used for training the first printing branch based on the first printing loss function according to the sample bill image marked with the first printing text, the second printing text and the background. Wherein, the first printing loss function is determined according to the first part of the loss function, the second part of the loss function and the third part of the loss function; the first part of the loss function is determined according to the classification output of the first printing branch on the first printing text and background of the sample bill image, the classification label of the first printing text and background, and the mask weight of the text line area of the first printing text; the second part of the loss function is the loss function of the part of the first printing text line in the sample bill image without adhesion with the second printing text; the third part of the loss function is the loss function corresponding to the part of the first printing text and the second printing text adhesion in the sample bill image. The training unit is also used for training the second printing branch based on the second printing loss function according to the sample bill image marked with the first printing text, the second printing text and the background. The secondary printing loss function is determined according to a fourth partial loss function, a fifth partial loss function and a sixth partial loss function; the fourth partial loss function is determined according to the classification output of the secondary printing text and background of the sample bill image by the secondary printing branch, the classification label of the secondary printing text and background and the text line area mask weight of the secondary printing text; the fifth partial loss function is a loss function of a part of the first printing text line in the sample bill image that is not adhered to the secondary printing text; and the sixth partial loss function is a loss function corresponding to a part of the first printing text and the secondary printing text line that are adhered to each other in the sample bill image.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the image text line detection method according to any one of claims 1 to 4 when executing the program.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the image text line detection method according to any one of claims 1 to 4.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the image text line detection method according to any one of claims 1 to 4. The computer program is executed by the processor to implement the image text line detection method according to any one of claims 1 to 4.

Images