End-to-end text recognition method, apparatus, device, and storage medium

Abstract

The present disclosure provides an end-to-end text recognition method, device and equipment and storage medium, which can be applied to the technical field of deep learning and the field of financial technology. The method comprises the following steps: inputting a to-be-tested image into a feature extraction network, and extracting a first feature image by using a resolution attention mechanism, wherein the first feature image comprises a local feature image and a global feature image in the to-be-tested image; correcting the first feature image by using a control point positioning network to obtain a second feature image; and inputting the second feature image into a text recognition network to output text information corresponding to the to-be-tested image.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of deep learning and the technical field of finance, in particular to an end-to-end text recognition method, device, equipment, medium and program product. BACKGROUND

[0002] In the related art, for an end-to-end text detection and recognition method of an arbitrary shape, the following defects exist: due to low text resolution, there are many low-resolution images, which greatly affect text detection; due to the randomness of the text photographer, exposure and shaking may occur, resulting in low detection and recognition accuracy; due to the presence of irregular shapes such as seals in the text, the recognition effect of the subsequent text is crucial, thereby affecting the accuracy of text recognition. SUMMARY

[0003] In view of the above problems, the present disclosure provides an end-to-end text recognition method, device, equipment, medium and program product.

[0004] According to a first aspect of the present disclosure, an end-to-end text recognition method is provided, comprising: inputting a to-be-detected image into a feature extraction network, and extracting a first feature image using a resolution attention mechanism, wherein the first feature image includes a local feature image and a global feature image in the to-be-detected image; correcting the first feature image using a control point positioning network to obtain a second feature image; inputting the second feature image into a text recognition network to output text information corresponding to the to-be-detected image.

[0005] According to an embodiment of the present disclosure, the inputting of the to-be-detected image into the feature extraction network and the extracting of the first feature image using the resolution attention mechanism comprises: inputting the to-be-detected image into the feature extraction network, and extracting the local feature image and the global feature image using the resolution attention mechanism; and splicing the local feature image and the global feature image to obtain the first feature image.

[0006] According to an embodiment of the present disclosure, the resolution attention mechanism includes a first resolution attention mechanism and a second resolution attention mechanism, and the inputting of the to-be-detected image into the feature extraction network and the extracting of the local feature image and the global feature image using the resolution attention mechanism comprises: inputting the to-be-detected image into the feature extraction network, grouping all features of the to-be-detected image according to a preset allocation parameter to obtain a local feature set and a global feature set; processing the local feature set using the first resolution attention mechanism to obtain the local feature image; and processing the global feature set using an average pooling layer and the second resolution attention mechanism to obtain the global feature image.

[0007] According to an embodiment of the present disclosure, the above inputting the to-be-tested image into the feature extraction network, grouping all features of the to-be-tested image according to a preset allocation parameter, to obtain a local feature set and a global feature set, comprises: grouping the number of heads of a multi-head attention mechanism according to the preset allocation parameter based on the multi-head attention mechanism, to obtain the number of features of the first resolution attention mechanism and the number of features of the second resolution attention mechanism; determining the local feature set from all features of the to-be-tested image according to the number of features of the first resolution attention mechanism; and determining the global feature set from all features of the to-be-tested image according to the number of features of the second resolution attention mechanism.

[0008] According to an embodiment of the present disclosure, the above correcting the first feature image by using the control point positioning network to obtain a second feature image comprises: obtaining a set of predicted reference points from the control point positioning network; generating a correction parameter curve and a sampling network according to the set of predicted reference points; sampling the first feature image by using the sampling network to obtain a set of control points; and correcting the set of control points by using the correction parameter curve to obtain the second feature image.

[0009] According to an embodiment of the present disclosure, the above control point set comprises n control points, and the above correcting the set of control points by using the correction parameter curve to obtain the second feature image comprises: for an i-th control point, determining a target offset according to coordinate information of the control point and the correction parameter curve; determining coordinate information of the control point after correction according to the coordinate information of the control point and the target offset; and generating the second feature image according to the coordinate information of all n control points after correction and the n control points, wherein 1≤i≤n, i and n are positive integers.

[0010] According to an embodiment of the present disclosure, the above end-to-end text recognition method further comprises: optimizing the coordinate information of the n control points after correction by using a center point of the correction parameter curve to obtain an optimized control point set; and generating the second feature image according to the optimized control point set.

[0011] According to an embodiment of the present disclosure, the above inputting the second feature image into the text recognition network to output text information corresponding to the to-be-tested image comprises: extracting a time sequence feature vector in the second feature image by using an encoder in the text recognition network; and processing the time sequence feature vector by using a decoder in the text recognition network to obtain the text information corresponding to the to-be-tested image.

[0012] According to an embodiment of the present disclosure, the end-to-end text recognition method further comprises: processing the initial sample image set by using a data segmentation method to obtain a training sample image set and a test sample image set; training the initial end-to-end text recognition model by using the training sample image set and a model loss function to obtain a trained end-to-end text recognition model; verifying the convergence degree of the trained end-to-end text recognition model by using the test sample image set, and obtaining a trained end-to-end text recognition model in a case where the convergence degree meets a preset condition, wherein the trained end-to-end text recognition model comprises the feature extraction network, the control point positioning network and the text recognition network.

[0013] According to an embodiment of the present disclosure, the model loss function comprises a text detection loss function and a text recognition loss function, and the end-to-end text recognition method further comprises: constructing the text detection loss function according to a classification parameter, a center degree parameter of a correction curve and an offset parameter of the control point positioning network; constructing the text recognition loss function according to a decoding order parameter; and constructing the model loss function according to the text detection loss function and the text recognition loss function.

[0014] A second aspect of the present disclosure provides an end-to-end text recognition device, comprising: an extraction module, a correction module and a detection module. The extraction module is configured to input an image to be tested into a feature extraction network, and extract a first feature image by using a resolution attention mechanism, wherein the first feature image comprises a local feature image and a global feature image in the image to be tested. The correction module is configured to correct the first feature image by using a control point positioning network to obtain a second feature image. The detection module is configured to input the second feature image into a text recognition network, and output text information corresponding to the image to be tested.

[0015] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; a memory configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method.

[0016] A fourth aspect of the present disclosure further provides a computer-readable storage medium having stored executable instructions, which, when executed by a processor, cause the processor to perform the method.

[0017] A fifth aspect of the present disclosure further provides a computer program product comprising a computer program, which, when executed by a processor, implements the method.

[0018] The end-to-end text recognition method, device, equipment, medium and program product provided according to the present disclosure can input a to-be-tested image into a feature extraction network, use a resolution attention mechanism, obtain a first feature image containing a high-resolution local feature image and a low-resolution global feature image, solve the problem of low recognition accuracy for a low-resolution image, use a control point positioning network to correct the first feature image, obtain a second feature image after correction, finally input the second feature image after correction into a text recognition network, and output text information corresponding to the to-be-tested image. Since the first feature image is corrected first and then text recognition is performed, the text recognition accuracy can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure taken in conjunction with the accompanying drawings, in which:

[0020] Figure 1 An application scenario diagram of the end-to-end text recognition method, device, equipment, medium and program product according to the embodiments of the present disclosure is schematically shown;

[0021] Figure 2 A flowchart of the end-to-end text recognition method according to the embodiments of the present disclosure is schematically shown;

[0022] Figure 3 A flowchart of extracting the first feature image according to the embodiments of the present disclosure is schematically shown;

[0023] Figure 4 An exemplary system framework of the end-to-end text recognition method according to the embodiments of the present disclosure is schematically shown;

[0024] Figure 5 A framework of the high-low resolution attention mechanism according to the embodiments of the present disclosure is schematically shown;

[0025] Figure 6 A flowchart of obtaining the second feature image according to the embodiments of the present disclosure is schematically shown;

[0026] Figure 7 An exemplary architecture of correcting the first feature image using the control point positioning network according to the embodiments of the present disclosure is schematically shown;

[0027] Figure 8 A control point positioning network module according to the embodiments of the present disclosure is schematically shown;

[0028] Figure 9 A flowchart of outputting text information corresponding to the to-be-tested image according to the embodiments of the present disclosure is schematically shown;

[0029] Figure 10A framework of an encoder according to an embodiment of the present disclosure is schematically shown;

[0030] Figure 11 A text recognition network module according to an embodiment of the present disclosure is schematically shown;

[0031] Figure 12 A flowchart of obtaining a trained end-to-end text recognition model according to an embodiment of the present disclosure is schematically shown;

[0032] Figure 13 A structural block diagram of an end-to-end text recognition apparatus according to an embodiment of the present disclosure is schematically shown; and

[0033] Figure 14 A block diagram of an electronic device suitable for implementing an end-to-end text recognition method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0034] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It is to be understood, however, that the description is merely exemplary of the present disclosure, and is not intended to limit the scope of the present disclosure. In the following detailed description of the embodiments of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it would be apparent to one skilled in the art that the present disclosure can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring aspects of the present disclosure.

[0035] The terms used herein are merely used to describe specific embodiments, and are not intended to limit the present disclosure. The terms "include" and "have" and the like used herein indicate the presence of the described features, steps, operations, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

[0036] All terms used herein, including technical and scientific terms, have the same meanings as those generally understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the present description, and should not be interpreted in an idealized or excessively formal manner.

[0037] In the case of using expressions similar to "at least one of A, B, and C, etc.", it is generally to be interpreted as including one or more of the same in the same manner as the meaning of the expression "at least one of A, B, and C" is understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include a system having A alone, a system having B alone, a system having C alone, a system having A and B together, a system having A and C together, a system having B and C together, and / or a system having A, B, and C together, etc.).

[0038] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of data (such as, but not limited to, user personal information) comply with relevant laws and regulations, necessary security measures are taken, and the public order and good customs are not violated.

[0039] It is found in the implementation of the present disclosure that in the end-to-end text detection and recognition method for any shape, due to low text resolution, exposure, shaking and the like may occur during shooting, and the text contains irregular shapes such as seals, and in the related art, there is a problem of low accuracy of text recognition.

[0040] Therefore, an embodiment of the present disclosure provides an end-to-end text recognition method, comprising: inputting a to-be-detected image into a feature extraction network, and extracting a first feature image by using a resolution attention mechanism, wherein the first feature image comprises a local feature image and a global feature image in the to-be-detected image; correcting the first feature image by using a control point positioning network to obtain a second feature image; inputting the second feature image into a text recognition network to output text information corresponding to the to-be-detected image.

[0041] Figure 1 An application scenario diagram of the end-to-end text recognition according to an embodiment of the present disclosure is schematically shown.

[0042] As shown in Figure 1 The application scenario 100 according to the embodiment can include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0043] A user can use at least one of the first terminal device 101, the second terminal device 102 and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102 and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0044] The first terminal device 101, the second terminal device 102 and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc.

[0045] For example, the image to be tested can be acquired by a client of at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0046] The server 105 can be a server that provides various services, such as a background management server that provides support for a website browsed by a user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (only as an example). The background management server can perform analysis and the like on received user requests and the like, and feed back the processing results (such as a web page, information, or data acquired or generated according to a user request, and the like) to the terminal device.

[0047] For example, the image to be tested can be input into a feature extraction network by the server 105, a first feature image can be extracted using a resolution attention mechanism, the first feature image can be corrected using a control point positioning network to obtain a second feature image, and finally the second feature image can be input into a text recognition network to output text information corresponding to the image to be tested.

[0048] It should be noted that the end-to-end text recognition method provided by the embodiments of the present disclosure can generally be executed by the server 105. Correspondingly, the end-to-end text recognition apparatus provided by the embodiments of the present disclosure can generally be arranged in the server 105. The end-to-end text recognition method provided by the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Correspondingly, the end-to-end text recognition apparatus provided by the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0049] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the above-described scenario is only illustrative. Any number of terminal devices, networks, and servers can be provided according to implementation needs.

[0050] The end-to-end text recognition method according to the embodiments of the present disclosure will be described in detail below based on the scenario described above. Figure 1 The end-to-end text recognition method according to the embodiments of the present disclosure will be described in detail below based on the scenario described above. Figures 2-12 The end-to-end text recognition method according to the embodiments of the present disclosure will be described in detail below based on the scenario described above.

[0051] Figure 2 A flowchart of an end-to-end text recognition method according to an embodiment of the present disclosure is schematically shown.

[0052] As shown in Figure 2 The end-to-end text recognition method 200 of this embodiment includes operations S210 to S230.

[0053] In operation S210, a to-be-tested image is input into a feature extraction network, and a first feature image is extracted by using a resolution attention mechanism, where the first feature image can include a local feature image and a global feature image in the to-be-tested image.

[0054] According to an embodiment of the present disclosure, the feature extraction network can be a High and Low Attention Transformer (HLA Transformer) based on a high-low resolution attention mechanism, and can extract features of text of an arbitrary shape. The high resolution attention in the high-low resolution attention mechanism can capture local features, and the low resolution attention can capture global features.

[0055] According to an embodiment of the present disclosure, the to-be-tested image can include high-resolution text information and low-resolution background information. For example, the text information can include a curved “Hello word”, and the background information can include a blurred tree and house.

[0056] According to an embodiment of the present disclosure, the high-resolution text information in the to-be-tested image, the curved “Hello word”, is extracted by using the high resolution attention in the resolution attention mechanism, and the low-resolution background information in the to-be-tested image, the blurred tree and house, is extracted by using the low resolution attention in the resolution attention mechanism.

[0057] According to an embodiment of the present disclosure, the first feature image can include the curved “Hello word” and the blurred tree and house.

[0058] In operation S220, the first feature image is corrected by using a control point positioning network to obtain a second feature image.

[0059] According to an embodiment of the present disclosure, the control point positioning network can be used to detect and predict control points of text of an arbitrary shape.

[0060] According to an embodiment of the present disclosure, the control point positioning network is used to predict the control points of the text “Hello word”. The predicted text control points are corrected to correct the curved “Hello word” to obtain an aligned “Hello word”, thereby obtaining the second feature image.

[0061] In operation S230, the second feature image is input into a text recognition network, and text information corresponding to the to-be-tested image is output.

[0062] According to an embodiment of the present disclosure, the text recognition network can be used to decode text content from the second feature image.

[0063] According to an embodiment of the present disclosure, the second feature image including the aligned "Hello word" is input into the text recognition network, so that text information Hello word corresponding to the curved "Hello word" in the to-be-tested image is output.

[0064] According to an embodiment of the present disclosure, the to-be-tested image is input into the feature extraction network, and the resolution attention mechanism is used to obtain the first feature image containing high-resolution local feature images and low-resolution global feature images, so that the problem of low recognition accuracy for low-resolution images can be solved. The first feature image is corrected by using the control point positioning network, so that the second feature image after correction is obtained. Finally, the second feature image after correction is input into the text recognition network, and text information corresponding to the to-be-tested image is output. Since the first feature image is corrected first and then text recognition is performed, the text recognition accuracy can be improved.

[0065] Figure 3 A flowchart for extracting a first feature image according to an embodiment of the present disclosure is schematically shown.

[0066] As shown in Figure 3 The method 300 for extracting a first feature image according to the embodiment includes operations S310-S320.

[0067] In operation S310, the to-be-tested image is input into the feature extraction network, and the resolution attention mechanism is used to extract local feature images and global feature images respectively.

[0068] According to an embodiment of the present disclosure, the high-resolution attention in the resolution attention mechanism is used to extract high-resolution text information, i.e., the curved "Hello word", in the to-be-tested image, so that the local feature image is obtained. The low-resolution attention in the resolution attention mechanism is used to extract low-resolution background information, i.e., the blurred tree and house, in the to-be-tested image, so that the global feature image is obtained. In operation S320, the local feature image and the global feature image are spliced to obtain the first feature image.

[0069] According to an embodiment of the present disclosure, the local feature image including the curved "Hello word" and the global feature image including the blurred tree and house are spliced to obtain the first feature image including the curved "Hello word" and the blurred tree and house.

[0070] According to an embodiment of the present disclosure, the to-be-tested image is input into the feature extraction network, and the resolution attention mechanism is used to extract high-resolution local feature images and low-resolution global feature images respectively, and then the local feature images and the global feature images are spliced to obtain the first feature image for recognition, so that the problem of low recognition accuracy for low-resolution images in the conventional method can be solved.

[0071] Figure 4 An exemplary system framework of an end-to-end text recognition method according to an embodiment of the present disclosure is schematically shown.

[0072] As shown in Figure 4 The image "Hello word" can represent text information in the image to be tested. The image to be tested is input into a feature extraction network HLA Transformer, features in the image to be tested are extracted, and the features are spliced and fused to obtain a first feature image. The control point positioning network is used to predict the control points of the text based on the first feature map, and the control points are corrected to align the text features. Finally, the aligned features are input into a text recognition network to obtain the text information corresponding to the image to be tested.

[0073] According to an embodiment of the present disclosure, the resolution attention mechanism includes a first resolution attention mechanism and a second resolution attention mechanism. The image to be tested is input into a feature extraction network, and the resolution attention mechanism is used to extract a local feature image and a global feature image, including:

[0074] The image to be tested is input into a feature extraction network, and all features of the image to be tested are grouped according to a preset allocation parameter to obtain a local feature set and a global feature set;

[0075] The local feature set is processed using the first resolution attention mechanism to obtain a local feature image;

[0076] The global feature set is processed using the average pooling layer and the second resolution attention mechanism to obtain a global feature image.

[0077] According to an embodiment of the present disclosure, the first resolution attention mechanism can represent a high-resolution attention mechanism. The second resolution attention mechanism can represent a low-resolution attention mechanism.

[0078] According to an embodiment of the present disclosure, the features in the image to be tested are grouped according to a preset allocation parameter, the local feature set is processed using the first resolution attention mechanism, and the global feature set is processed using the average pooling layer and the second resolution attention mechanism, which can improve the processing efficiency.

[0079] According to an embodiment of the present disclosure, the image to be tested is input into a feature extraction network, and all features of the image to be tested are grouped according to a preset allocation parameter to obtain a local feature set and a global feature set, including:

[0080] Based on the multi-head attention mechanism, the number of heads of the multi-head attention mechanism is grouped according to a preset allocation parameter to obtain the number of features of the first resolution attention mechanism and the number of features of the second resolution attention mechanism;

[0081] determine the local feature set from all features of the image to be tested according to the number of features of the first resolution attention mechanism;

[0082] determine the global feature set from all features of the image to be tested according to the number of features of the second resolution attention mechanism.

[0083] According to an embodiment of the present disclosure, the number of heads of the multi-head attention mechanism can represent the sum of the number of features of the first resolution attention mechanism and the number of features of the second resolution attention mechanism.

[0084] According to an embodiment of the present disclosure, based on the multi-head attention mechanism, the number of heads of the multi-head attention mechanism is grouped according to the preset allocation parameter, so that the local feature set and the global feature set are respectively determined according to the number of features of the first resolution attention mechanism and the number of features of the second resolution attention mechanism, which can improve the processing efficiency.

[0085] Figure 5 The framework of the high-low resolution attention mechanism according to an embodiment of the present disclosure is schematically shown.

[0086] According to an embodiment of the present disclosure, the high-low resolution attention mechanism captures the local feature image by the high resolution attention, and captures the global feature image by the low resolution attention.

[0087] According to an embodiment of the present disclosure, (1-α)N h heads under the multi-head self-attention mechanism are allocated to the high resolution attention mechanism to capture fine-grained high resolution features. The obtained high resolution features are concatenated to obtain a high resolution attention mechanism map.

[0088] According to an embodiment of the present disclosure, αN h heads under the multi-head self-attention mechanism are allocated to the low resolution attention mechanism. In the low resolution attention mechanism, an average pooling layer is applied to obtain low resolution features. The positions of Query (Q, which can represent the texture feature information extracted from the low resolution, used for texture search) in the feature map and Key (K, which can represent the texture information of the image consistent with the low resolution image obtained by first downsampling and then upsampling from the high resolution image, used for texture search) and Value (V, which can represent the texture information of the image to be tested, used for texture migration) in the low resolution are modeled. The obtained low resolution features are concatenated to obtain a low resolution attention mechanism map.

[0089] According to an embodiment of the present disclosure, the high-resolution attention mechanism map and the low-resolution attention mechanism map are connected into the following layer to obtain a first feature image.

[0090] According to an embodiment of the present disclosure, for the allocation of the number of heads, the number of heads (1-α)N allocated by the high-resolution attention mechanism h and the number of heads αN allocated by the low-resolution attention mechanism h are the same as the number of heads N in the multi-head self-attention mechanism. h

[0091] According to an embodiment of the present disclosure, the feature extraction network divides the same number of heads into two groups in an allocation ratio α, where α can represent a human hyperparameter. The expression of the allocation ratio α can be represented as the following formula (1).

[0092] HLA(X) = [H-F(X); L-F(X)] (1)

[0093] Where HLA(X) can represent the number of heads in the multi-head self-attention mechanism, H-F(X) can represent the number of heads allocated by the high-resolution attention mechanism, and L-F(X) can represent the number of heads allocated by the low-resolution attention mechanism.

[0094] Figure 6 A flowchart for obtaining a second feature image according to an embodiment of the present disclosure is schematically shown.

[0095] As shown in Figure 6 , the method 600 for obtaining a second feature image of this embodiment includes operation S610 to operation S640.

[0096] In operation S610, a set of predicted reference points is obtained from a control point positioning network.

[0097] According to an embodiment of the present disclosure, the control point positioning network can be used to detect text of any shape. The control point positioning network uses a single-stage anchor-free framework as a detection branch.

[0098] In operation S620, a correction parameter curve and a sampling network are generated according to the set of predicted reference points. In operation S630, the first feature image is sampled using the sampling network to obtain a set of control points.

[0099] In operation S640, the set of control points is corrected using the correction parameter curve to obtain a second feature image.

[0100] According to an embodiment of the present disclosure, according to the correction parameter curve, the control points of all text lines are optimized using regression.

[0101] ​According to an embodiment of the present disclosure, a control point positioning network is used to predict a set of reference points, and a first feature image is processed by a correction parameter curve. Since the text with irregular shape is corrected before recognition, the recognition accuracy can be improved.

[0102] Figure 7 An exemplary architecture for correcting a first feature image by a control point positioning network according to an embodiment of the present disclosure is schematically shown.

[0103] According to an embodiment of the present disclosure, a positioning network determines the parameters of the transformation according to the predicted points on the feature map, and performs geometric transformation on the feature map processed by HLA and input into a sampling network (TPS Transformer). After multiple iterations, a calibrated feature map is obtained.

[0104] According to an embodiment of the present disclosure, a control point positioning network is used to predict a set of reference points. In a grid generator, the transformation parameters are calculated according to the predicted points, and a sampling network is generated. The task of the sampling network is to obtain a grid and an input image, and generate a direction-corrected image.

[0105] According to an embodiment of the present disclosure, the loss of recognition can be propagated back to the detection branch, so that the detection and recognition jointly optimize the detection branch, and the problem of inconsistency between text detection and recognition is largely alleviated.

[0106] According to an embodiment of the present disclosure, the control point set includes n control points, and the control point set is corrected by a correction parameter curve to obtain a second feature image, including:

[0107] For the i-th control point, the target offset is determined according to the coordinate information of the control point and the correction parameter curve;

[0108] According to the coordinate information of the control point and the target offset, the coordinate information of the control point after correction is determined;

[0109] According to the coordinate information of the control point after correction and the n control points, a second feature image is generated, wherein 1≤i≤n, i and n are positive integers.

[0110] According to an embodiment of the present disclosure, n text alignment text features are output according to n control points, which are mainly realized based on a differentiable spatial transformation network (STN, Spatial Transformer Network).

[0111] According to an embodiment of the present disclosure, the control point set is corrected by a correction parameter curve to obtain a second feature image in which the text information is corrected. Since the text with irregular shape is corrected before recognition, the recognition accuracy can be improved.

[0112] According to an embodiment of the present disclosure, the end-to-end text recognition method further includes:

[0113] The n control points are optimized by using the center point of the correction parameter curve, and a set of optimized control points is obtained.

[0114] A second feature image is generated according to the set of optimized control points.

[0115] According to an embodiment of the present disclosure, the center point of the correction parameter curve is used to optimize the corrected control points, loss back propagation is realized, text detection and recognition are jointly optimized, and the recognition accuracy is improved.

[0116] Figure 8 The control point positioning network module according to an embodiment of the present disclosure is schematically shown.

[0117] According to an embodiment of the present disclosure, for each line of text, the center region is sampled as a positive pixel, and the control points of the text line are sampled.

[0118] According to an embodiment of the present disclosure, a cubic bezier curve is first fitted, the control points are recalculated, and the expression of uniformly sampling n MCP (Main Control Point) points can be represented as formula (2) as follows.

[0119]

[0120] wherein P k may represent the kth sampling point, C i may represent the ith bezier control point, n mcp may represent the number of control points on each side.

[0121] According to an embodiment of the present disclosure, B i,n may represent a Bernstein basis polynomial, and its expression can be represented as formula (3) as follows.

[0122]

[0123] wherein, may represent a binomial coefficient, n may represent the number of control points, i may represent the ith control point (i = 0, 1, 2, … n), and t may represent a point on the bezier curve, and the value range is 0-1.

[0124] According to an embodiment of the present disclosure, the n MCP points of the sampling are defined as the whole correction control points of the text instance, and are used to generate the training target. Specifically, for one positive pixel of (x, y) position, the expression of the offset target can be represented as formula (4) as follows.

[0125] Δ k,x =P k,x -x, Δ k,y =P k,y -y (4)

[0126] Wherein, P k,x and P k,y represent the coordinates of the kth control point, Δ k,x and Δ k,y represent the offset of the target to the kth control point.

[0127] According to an embodiment of the present disclosure, a single-stage anchor-free (no pre-selected frame) frame is used as a detection branch to densely regress the control points of all text lines. The size of the regression result can be represented as (H / S, W / S, 4x n mcp ), wherein n mcp represents the number of control points per edge, S represents the down-sampling scale of the input image, and H and W represent the height and width of the feature map respectively.

[0128] Figure 9 A flowchart of outputting text information corresponding to a to-be-detected image according to an embodiment of the present disclosure is schematically shown.

[0129] As Figure 9 shown, the end-to-end text recognition method 900 of this embodiment includes operation S910 to operation S920.

[0130] In operation S910, the time sequence feature vector in the second feature image is extracted by using the encoder in the text recognition network.

[0131] According to an embodiment of the present disclosure, the encoder part in the text recognition network is down-sampled for 4 times in total, which is reduced to 16 times of the original image size, and a high-low resolution attention mechanism is added. The patch X p (network patch X p ) is mapped to the latent D-dimensional embedding space using a trainable linear projection.

[0132] According to an embodiment of the present disclosure, in order to encode the slice space information, a specific position embedding is learned, which is added to the patch embedding vector to preserve the position information. Finally, the expression of the N one-dimensional vectors can be represented as formula (5) as follows.

[0133]

[0134] where E is the embedded projection of the patch, E pos represents the position embedding. The input embedding vector x is re-encoded into a two-dimensional vector where the patch size is P x P, and the number of control points is

[0135] At operation S920, the decoder in the text recognition network is used to process the time sequence feature vector to obtain text information corresponding to the to-be-tested image.

[0136] According to an embodiment of the present disclosure, the decoder part in the text recognition network makes 4 upsampling in total, and finally restores the one-dimensional vector to the original image. The encoding part and the decoding part also make three jump connections.

[0137] According to an embodiment of the present disclosure, the time sequence feature vector in the second feature image is extracted by using the encoder in the text recognition network, and the text information corresponding to the to-be-tested image can be obtained by processing the time sequence feature vector by using the decoder in the text recognition network.

[0138] Figure 10 The framework of the encoder according to an embodiment of the present disclosure is schematically shown.

[0139] According to an embodiment of the present disclosure, the to-be-tested image is processed by the CNN convolutional neural network, is sampled, and is gradually reduced to 1 / 2 of the original image, 1 / 4 of the original image, and finally 1 / 8 of the original image, and the number of channels is gradually increased to obtain the features of each different sampling layer. After being processed by the hidden layer, a sequence is obtained by linear mapping. The sequence enters the N = 12 HLA Transformers, the HLA processes the sequence to obtain a feature extraction image, and then enters the hidden layer feature. By using the reshape function, half of (512, H / 16, W / 16) and half of the original image which is reduced to 1 / 8 of the original image are spliced to obtain (256, H / 8, W / 8).

[0140] According to an embodiment of the present disclosure, the encoder is composed of L layers of high / low resolution attention mechanism (HLA) and multilayer perceptron block (MLP, Multilayer Perceptron). Therefore, the output of the i-th layer can be represented as the following formulas (6) and (7).

[0141]

[0142]

[0143] ​where LN(·) can represent a layer normalization operation, may represent an encoded image.

[0144] Figure 11 The text recognition network module according to an embodiment of the present disclosure is schematically shown.

[0145] According to an embodiment of the present disclosure, the text recognition network module is a model based on an attention-based encoder and decoder, which extracts sequence feature information from an input image feature sequence. The corrected image is input to the text recognition network, which first encodes the image through the encoder to output a feature sequence vector, and then inputs the feature sequence vector into a convolutional layer to obtain the corresponding recognized characters through existing CNN convolutional neural network calculation.

[0146] According to an embodiment of the present disclosure, the text recognition network module directly predicts the character sequence of the corrected image through the text recognition network. And according to the sequence model, sequence mapping is performed to obtain the sequence recognition network.

[0147] According to an embodiment of the present disclosure, the input of the sequence recognition network is an ideal horizontal correction image. It is based on an attention-based encoder-decoder model. The encoder extracts sequence feature representations from the input image IP (Image Processing). Then, the decoder uses LSTM (Long short-term memory) to conditionally reflect the continuous representation according to the appropriate content seen at each step, and continuously outputs a sequence.

[0148] According to an embodiment of the present disclosure, the network combines convolutional layers with recurrent networks, replacing the traditional sliding window-based CNN feature extraction method. In order to expand the receptive field of the features extracted by ConvNet (CNN), Bi-LSTM (Bidirectional Long Short-Term Memory) is used to analyze the features forward and backward to obtain long-term dependencies. The new feature sequence can be represented as formula (8) as follows.

[0149] L = [l1, l2, … l m ] (8)

[0150] where m = W, l i may represent the i-th feature sequence (i = 1, 2, … w).

[0151] According to an embodiment of the present disclosure, the decoder is a sequence model based on attention. At time step t, the decoder predicts one character or sequence according to the encoder L input. The calculation of attention weight is performed by an attention mechanism, which can be represented as the following formulas (9) and (10).

[0152] d t,j =u T tanh(Us t-1 +Wlj+bias) (9)

[0153]

[0154] where d t,j may represent the attention weight at each time, l j may represent the linear vector of the encoder output at t, a t,j may represent the normalized attention weight, S t may represent the encoding hidden layer state, and u, Wmay represent trainable weights.

[0155] According to an embodiment of the present disclosure, the encoding hidden layer state can be represented as the following formula (11).

[0156] s t =f(s t-1 , y t-1 , a t ) (11)

[0157] where f(.) can represent a nonlinear activation function, y t-1 may represent the character prediction output at t-1, and the decoder will predict and recognize the characters from left to right in sequence.

[0158] According to an embodiment of the present disclosure, the character prediction output at t can be represented as the following formula (12).

[0159] y t =f(y t-1 , s t-1 , a t ) (12)

[0160] The decoder based on the attention mechanism can capture the dependency between characters and recognize the characters.

[0161] Figure 12 A flowchart for obtaining a trained end-to-end text recognition model according to an embodiment of the present disclosure is schematically shown.

[0162] As Figure 12As shown, the method 1200 for obtaining the trained end-to-end text recognition model of the embodiment includes operation S1210 to operation S1230.

[0163] In operation S1210, the initial sample image set is processed by using the data segmentation method to obtain a training sample image set and a test sample image set.

[0164] According to an embodiment of the present disclosure, 20% of the training data is used as a validation set to validate the pictures, so as to observe whether the model converges. When the model converges, it is stopped and can be used for subsequent evaluation.

[0165] According to an embodiment of the present disclosure, the original sample is data-augmented by using a CV algorithm before model training. Image preprocessing is performed by using image flipping, rotation, brightness change, elastic deformation, adding salt and pepper noise, motion blur, enhancing contrast, creating a complex background, and the like.

[0166] In operation S1220, the initial end-to-end text recognition model is trained by using the training sample image set and the model loss function to obtain a trained end-to-end text recognition model.

[0167] According to an embodiment of the present disclosure, in addition to data division, there is model hyperparameter setting. The network has one input of 8 pictures each time during training. The network is optimized by using an Adam optimizer (adaptive moment estimation optimizer) with β1=0.9, β2=0.999, ∈=10 -8 The initial learning rate is 10 -4 A total of 500 rounds of training are performed by using a Pytorch deep learning framework.

[0168] In operation S1230, the trained end-to-end text recognition model is verified by using the test sample image set to verify the convergence degree of the trained end-to-end text recognition model. When the convergence degree meets a preset condition, a trained end-to-end text recognition model is obtained, wherein the trained end-to-end text recognition model includes a feature extraction network, a control point positioning network, and a text recognition network.

[0169] According to an embodiment of the present disclosure, the obtained trained end-to-end text recognition model can be used for recognizing end-to-end text of any shape.

[0170] According to an embodiment of the present disclosure, the model loss function includes a text detection loss function and a text recognition loss function. The above-mentioned end-to-end text recognition method further includes:

[0171] According to the classification parameter, the center degree parameter of the correction curve, and the offset parameter of the control point positioning network, a text detection loss function is constructed;

[0172] According to the decoding order parameter, a text recognition loss function is constructed.

[0173] According to the text detection loss function and the text recognition loss function, a model loss function is constructed.

[0174] According to an embodiment of the present disclosure, the model loss function is composed of a detection loss and a recognition loss .

[0175] According to an embodiment of the present disclosure, the definitions of the detection loss and the recognition loss can be represented as the following formula (13).

[0176]

[0177] where λ det and λ rec may represent hyperparameters.

[0178] The detection loss L det may be represented as the following formula (14).

[0179]

[0180] where the detection loss function L may represent a multi-task loss function. and are used for classification and center prediction, respectively. may represent a loss function of the MCP head, which is realized by Sm, and the loss function of the MCP head can be represented as the following formula (15).

[0181]

[0182] where Δ pred and Δ target are the predicted offset and the target offset of the control point positioning network, respectively. Here λ mcp is used to balance the importance and is set to 0.2 by default in the experiment. The recognition loss function L is used to optimize the recognition branch.

[0183] The recognition loss function L may be represented as the following formula (16).

[0184]

[0185] where y1, … y t , …, y t are the ground truth (real data) texts represented by a character sequence. The target is to average the loss on the left-to-right and right-to-left decoders. Their prediction distributions are represented by pltr and p rtl indicates.

[0186] According to an embodiment of the present disclosure, a text detection loss function is constructed according to a classification parameter, a center degree parameter of a correction curve and an offset parameter of a control point positioning network, a text recognition loss function is constructed according to a decoding order parameter, and finally a model loss function is constructed according to the text detection loss function and the text recognition loss function. The end-to-end text recognition model can be optimized according to the constructed model loss function to obtain a trained end-to-end text recognition model.

[0187] Based on the above end-to-end text recognition method, the present disclosure further provides an end-to-end text recognition device. The following will be described in detail Figure 13 the device.

[0188] Figure 13 The structure block diagram of the end-to-end text recognition device according to an embodiment of the present disclosure is schematically shown.

[0189] As Figure 13 shown, the end-to-end text recognition device 1300 of this embodiment includes an extraction module 1310, a correction module 1320 and a detection module 1330.

[0190] The extraction module 1310 is configured to input the to-be-tested image into a feature extraction network, and extract a first feature image by using a resolution attention mechanism, wherein the first feature image includes a local feature image and a global feature image in the to-be-tested image. In an embodiment, the extraction module 1310 can be configured to perform the operation S210 described above, and details are not repeated here.

[0191] The correction module 1320 is configured to perform correction processing on the first feature image by using a control point positioning network to obtain a second feature image. In an embodiment, the correction module 1320 can be configured to perform the operation S220 described above, and details are not repeated here.

[0192] The detection module 1330 is configured to input the second feature image into a text recognition network, and output text information corresponding to the to-be-tested image. In an embodiment, the detection module 1330 can be configured to perform the operation S230 described above, and details are not repeated here.

[0193] According to an embodiment of the present disclosure, the extraction module 1310 includes a first extraction submodule and a first obtaining submodule. The first extraction submodule is configured to input the to-be-tested image into a feature extraction network, and extract a local feature image and a global feature image by using a resolution attention mechanism respectively. The first obtaining submodule is configured to splice the local feature image and the global feature image to obtain a first feature image.

[0194] According to an embodiment of the present disclosure, the resolution attention mechanism includes a first resolution attention mechanism and a second resolution attention mechanism, and the first extraction submodule includes a first obtaining unit, a second obtaining unit and a third obtaining unit. The first obtaining unit is configured to input the to-be-tested image into a feature extraction network, group all features of the to-be-tested image according to preset allocation parameters, and obtain a local feature set and a global feature set. The second obtaining unit is configured to process the local feature set by using the first resolution attention mechanism to obtain a local feature image. The third obtaining unit is configured to process the global feature set by using an average pooling layer and the second resolution attention mechanism to obtain a global feature image.

[0195] According to an embodiment of the present disclosure, the first obtaining unit includes a first obtaining subunit, a first determining subunit and a second determining subunit. The first obtaining subunit is configured to group the number of heads of the multi-head attention mechanism according to preset allocation parameters based on the multi-head attention mechanism to obtain the number of features of the first resolution attention mechanism and the number of features of the second resolution attention mechanism. The first determining subunit is configured to determine the local feature set from all features of the to-be-tested image according to the number of features of the first resolution attention mechanism. The second determining subunit is configured to determine the global feature set from all features of the to-be-tested image according to the number of features of the second resolution attention mechanism.

[0196] According to an embodiment of the present disclosure, the correction module 1320 includes a first obtaining submodule, a first generating submodule, a second obtaining submodule and a third obtaining submodule. The first obtaining submodule is configured to obtain a set of predicted reference points from a control point positioning network. The first generating submodule is configured to generate a correction parameter curve and a sampling network according to the set of predicted reference points. The second obtaining submodule is configured to sample the first feature image by using the sampling network to obtain a set of control points. The third obtaining submodule is configured to correct the set of control points by using the correction parameter curve to obtain a second feature image.

[0197] According to an embodiment of the present disclosure, the set of control points includes n control points, and the third obtaining submodule includes a first determining unit, a second determining unit and a first generating unit. The first determining unit is configured to determine a target offset for the i th control point according to the coordinate information of the control point and the correction parameter curve. The second determining unit is configured to determine the coordinate information of the control point after correction according to the coordinate information of the control point and the target offset. The first generating unit is configured to generate the second feature image according to the coordinate information of all n control points after correction and the n control points, where 1≤i≤n, i and n are positive integers.

[0198] According to an embodiment of the present disclosure, the third obtaining sub-module further includes a fourth obtaining unit and a second generating unit. The fourth obtaining unit is configured to optimize the coordinate information of the n control points after correction by using the center point of the correction parameter curve to obtain an optimized control point set. The second generating unit is configured to generate the second feature image according to the optimized control point set.

[0199] According to an embodiment of the present disclosure, the detection module 1330 includes a second extracting sub-module and a fourth obtaining sub-module. The second extracting sub-module is configured to extract a time sequence feature vector in the second feature image by using an encoder in the text recognition network. The fourth obtaining sub-module is configured to process the time sequence feature vector by using a decoder in the text recognition network to obtain text information corresponding to the image to be detected.

[0200] According to an embodiment of the present disclosure, the end-to-end text recognition device 1300 further includes a first obtaining module, a second obtaining module and a third obtaining module. The first obtaining module is configured to process the initial sample image set by using a data segmentation method to obtain a training sample image set and a test sample image set. The second obtaining module is configured to train the initial end-to-end text recognition model by using the training sample image set and a model loss function to obtain a trained end-to-end text recognition model. The third obtaining module is configured to verify the convergence degree of the trained end-to-end text recognition model by using the test sample image set, and obtain a trained end-to-end text recognition model in the case where the convergence degree meets a preset condition, wherein the trained end-to-end text recognition model includes a feature extraction network, a control point positioning network and a text recognition network.

[0201] According to an embodiment of the present disclosure, the model loss function includes a text detection loss function and a text recognition loss function, and the end-to-end text recognition device 1300 further includes a first constructing module, a second constructing module and a third constructing module. The first constructing module is configured to construct the text detection loss function according to a classification parameter, a center degree parameter of the correction curve and an offset parameter of the control point positioning network. The second constructing module is configured to construct the text recognition loss function according to a decoding order parameter. The third constructing module is configured to construct the model loss function according to the text detection loss function and the text recognition loss function.

[0202] According to embodiments of the present disclosure, any of the extraction module 1310, the correction module 1320, and the detection module 1330 can be implemented in one module, or any of them can be split into multiple modules. Alternatively, at least part of the function of one or more of these modules can be combined with at least part of the function of the other modules, and implemented in one module. According to embodiments of the present disclosure, at least one of the extraction module 1310, the correction module 1320, and the detection module 1330 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable manner that can be integrated or packaged by a circuit, etc. or implemented by hardware or firmware, or implemented in any one of software, hardware, and firmware or in a proper combination of any of them. Alternatively, at least one of the extraction module 1310, the correction module 1320, and the detection module 1330 can be at least partially implemented as a computer program module that can perform corresponding functions when the computer program module is run.

[0203] Figure 14 A block diagram of an electronic device suitable for implementing an end-to-end text recognition method according to embodiments of the present disclosure is schematically shown.

[0204] As shown in Figure 14 The electronic device 1400 according to embodiments of the present disclosure includes a processor 1401 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1402 or loaded from a storage portion 1408 into a random access memory (RAM) 1403. The processor 1401 can include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a special-purpose microprocessor (such as an application specific integrated circuit (ASIC)), and the like. The processor 1401 can also include an on-board memory for cache use. The processor 1401 can include a single processing unit or a plurality of processing units for performing different actions of the method processes according to embodiments of the present disclosure.

[0205] In the RAM 1403, various programs and data required for the operation of the electronic device 1400 are stored. The processor 1401, the ROM 1402, and the RAM 1403 are connected to each other via the bus 1404. The processor 1401 performs various operations of the method flow according to the embodiments of the present disclosure by executing the programs in the ROM 1402 and / or the RAM 1403. It should be noted that the programs can also be stored in one or more memories other than the ROM 1402 and the RAM 1403. The processor 1401 can also perform various operations of the method flow according to the embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0206] According to an embodiment of the present disclosure, the electronic device 1400 can further include an input / output (I / O) interface 1405, which is also connected to the bus 1404. The electronic device 1400 can further include one or more of the following components connected to the I / O interface 1405: an input part 1406 including a keyboard, a mouse, etc.; an output part 1407 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 1408 including a hard disk, etc.; and a communication part 1409 including a network interface card such as a LAN card, a modem, etc. The communication part 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the I / O interface 1405 as necessary. A removable recording medium 1411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1410 as necessary, so that a computer program read out therefrom is installed in the storage part 1408 as necessary.

[0207] The present disclosure also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, when the one or more programs are executed, the method according to the embodiments of the present disclosure is implemented.

[0208] According to an embodiment of the present disclosure, the computer readable storage medium can be a nonvolatile computer readable storage medium, for example, can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, the computer readable storage medium can include one or more memories, such as the ROM 1402 and / or the RAM 1403 described above, and / or one or more memories other than the ROM 1402 and the RAM 1403.

[0209] Embodiments of the present disclosure also include a computer program product, which includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the item recommendation method provided by the embodiments of the present disclosure.

[0210] The above-described functions defined in the system / device of the embodiments of the present disclosure are performed when the computer program is executed by the processor 1401. According to an embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by computer program modules.

[0211] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal on a network medium, and be downloaded and installed through the communication part 1409, and / or be installed from the detachable medium 1411. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to wireless, wired, etc., or any appropriate combination thereof.

[0212] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1409, and / or be installed from the detachable medium 1411. When the computer program is executed by the processor 901, the above-described functions defined in the system of the embodiments of the present disclosure are performed. According to an embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0213] According to embodiments of the present disclosure, program code of the computer program for performing the methods provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages, and can be implemented in a computer program product. Specifically, the computer program can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. The programming language includes, but is not limited to, Java, C++, python, “C” language, or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, and partly on a remote computing device, or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider (ISP).

[0214] The computer program product of the present disclosure can be a computer program product, which is a machine-readable medium (or computer readable medium) having stored therein a sequence of instructions readable by a machine (e.g., a computer). The instructions in the machine-readable medium focus on the function of the computer program (e.g., in the form of a program unit) and on the cooperation behavior between such program units. The program can be distributed to clients on a data storage medium such as, but not limited to, a floppy disk, a DVD, a CD-ROM, or a USB flash drive, or it can be distributed to clients electronically, such as over the Internet, over other computer networks, or over a wireless data connection.

[0215] Those skilled in the art can understand that the features described in various embodiments of the present disclosure and / or claims can be combined or / and integrated, even if such combinations or integrations are not explicitly described in the present disclosure. In particular, the features described in various embodiments of the present disclosure and / or claims can be combined and / or integrated in various combinations, without departing from the spirit and teachings of the present disclosure. All these combinations and / or integrations fall within the scope of the present disclosure.

[0216] The above describes embodiments of the present disclosure. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present disclosure, and these substitutions and modifications should all fall within the scope of the present disclosure.

Claims

1. An end-to-end text recognition method, comprising: The image to be tested is input into a feature extraction network, and a first feature image is extracted using a resolution attention mechanism. The first feature image includes local feature images and global feature images of the image to be tested. The first feature image is corrected using a control point localization network to obtain the second feature image; The second feature image is input into a text recognition network, which outputs text information corresponding to the image under test. Wherein: the resolution attention mechanism includes a first resolution attention mechanism and a second resolution attention mechanism; the step of inputting the image to be tested into the feature extraction network and extracting the local feature image and the global feature image using the resolution attention mechanism respectively includes: The image to be tested is input into the feature extraction network, and all features of the image to be tested are grouped according to preset allocation parameters to obtain a local feature set and a global feature set; The local feature set is processed using the first resolution attention mechanism to obtain the local feature image; The global feature set is processed using an average pooling layer and a second resolution attention mechanism to obtain the global feature image; The first resolution attention mechanism represents high-resolution attention; the second resolution attention mechanism represents low-resolution attention; the high-resolution attention is used to extract high-resolution text information from the image under test to obtain the local feature image; the low-resolution attention is used to extract low-resolution background information from the image under test to obtain the global feature image.

2. The method according to claim 1, wherein, The step of inputting the image to be tested into the feature extraction network and extracting the first feature image using a resolution attention mechanism further includes: The local feature image and the global feature image are concatenated to obtain the first feature image.

3. The method according to claim 1, wherein, The process involves inputting the image to be tested into a feature extraction network, and grouping all features of the image to be tested according to preset allocation parameters to obtain a local feature set and a global feature set, including: Based on the multi-head attention mechanism, the number of heads of the multi-head attention mechanism is grouped according to the preset allocation parameters to obtain the number of features of the first resolution attention mechanism and the number of features of the second resolution attention mechanism. The local feature set is determined from all features of the image to be tested according to the number of features of the first resolution attention mechanism. The global feature set is determined from all features of the image under test according to the number of features of the second resolution attention mechanism.

4. The method according to claim 1, wherein, The step of using a control point localization network to correct the first feature image to obtain a second feature image includes: Obtain the set of predicted reference points from the control point positioning network; Generate correction parameter curves and a sampling network based on the predicted reference point set; The first feature image is sampled using the sampling network to obtain a set of control points; The control point set is corrected using the correction parameter curve to obtain the second feature image.

5. The method according to claim 4, wherein, The control point set includes n control points. The step of correcting the control point set using the correction parameter curve to obtain the second feature image includes: For the i-th control point, the target offset is determined based on the coordinate information of the control point and the correction parameter curve; Based on the coordinate information of the control point and the target offset, determine the coordinate information of the control point after correction; The second feature image is generated based on the coordinate information of all n control points after correction and the n control points, where 1≤i≤n and i and n are both positive integers.

6. The method according to claim 5, further comprising: Using the center point of the correction parameter curve, the coordinate information of the n control points after correction is optimized to obtain an optimized set of control points; The second feature image is generated based on the optimized set of control points.

7. The method according to claim 1, wherein, The step of inputting the second feature image into the text recognition network and outputting text information corresponding to the image to be tested includes: The temporal feature vector in the second feature image is extracted using the encoder in the text recognition network; The temporal feature vector is processed using the decoder in the text recognition network to obtain the text information corresponding to the image under test.

8. The method according to claim 1, further comprising: The initial sample image set is processed using a data segmentation method to obtain the training sample image set and the test sample image set; The initial end-to-end text recognition model is trained using the training sample image set and the model loss function to obtain the trained end-to-end text recognition model. The convergence of the trained end-to-end text recognition model is verified using the test sample image set. If the convergence meets the preset conditions, a trained end-to-end text recognition model is obtained. The trained end-to-end text recognition model includes the feature extraction network, the control point localization network, and the text recognition network.

9. The method according to claim 8, wherein, The model loss function includes a text detection loss function and a text recognition loss function, and the method further includes: The text detection loss function is constructed based on the classification parameters, the centrality parameter of the calibration curve, and the offset parameter of the control point localization network. Construct a text recognition loss function based on the decoding order parameters; The model loss function is constructed based on the text detection loss function and the text recognition loss function.

10. An end-to-end text recognition device, comprising: The extraction module is used to input the image to be tested into the feature extraction network and extract a first feature image using a resolution attention mechanism, wherein the first feature image includes local feature images and global feature images in the image to be tested; A correction module is used to perform correction processing on the first feature image using a control point localization network to obtain a second feature image; and The detection module is used to input the second feature image into the text recognition network and output text information corresponding to the image to be tested; The resolution attention mechanism includes a first resolution attention mechanism and a second resolution attention mechanism, and the extraction module includes: The first acquisition unit is used to input the image to be tested into the feature extraction network, and group all the features of the image to be tested according to the preset allocation parameters to obtain a local feature set and a global feature set. The second obtaining unit is used to process the local feature set using the first resolution attention mechanism to obtain the local feature image; The third obtaining unit is used to process the global feature set using an average pooling layer and the second resolution attention mechanism to obtain the global feature image; The first resolution attention mechanism represents high-resolution attention; the second resolution attention mechanism represents low-resolution attention; the high-resolution attention is used to extract high-resolution text information from the image under test to obtain the local feature image; the low-resolution attention is used to extract low-resolution background information from the image under test to obtain the global feature image.

11. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images