Text recognition method, electronic equipment and storage medium

Abstract

The invention discloses a text recognition method, electronic equipment and a storage medium. The text recognition method comprises the following steps: acquiring at least one initial image feature by utilizing a text recognition model, wherein each initial image feature is obtained by performing feature extraction on a to-be-recognized image; performing position coding on each initial image feature to obtain a position query vector corresponding to each initial image feature; and based on each initial image feature and each position query vector, determining a text recognition result of the to-be-recognized image, the text recognition result including at least one of the following: a type of each region in the to-be-recognized image, coordinate information of each region in the to-be-recognized image, and character information of each region in the to-be-recognized image. According to the scheme, the accuracy of the text recognition result obtained by text recognition can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a text recognition method, an electronic device, and a storage medium. Background Art

[0002] Scene text recognition mainly includes two tasks: detection and recognition of text in images. Currently, the commonly used recognition technology is based on deep learning methods, which can be a multi-stage method according to the processing process. The multi-stage method usually uses a detection algorithm to output the bounding rectangle of all text in the image, crops each text box area, corrects and enhances it, and then uses a character recognition algorithm to recognize each character in the text. Among them, the multi-stage method improves the accuracy of regular text recognition by correcting a single text image, but for curved text, a large amount of background noise will be introduced in the cropping stage, affecting the accuracy of subsequent character recognition.

[0003] In view of the existing technical defects, how to provide an effective text recognition solution is a technical problem that needs to be solved urgently by technical personnel in this field. Summary of the invention

[0004] The present application at least provides a text recognition method, an electronic device, and a storage medium.

[0005] The present application provides a text recognition method, including: obtaining at least one initial image feature using a text recognition model, each initial image feature is obtained by extracting features from an image to be recognized; performing position encoding on each initial image feature to obtain a position query vector corresponding to each initial image feature; determining a text recognition result of the image to be recognized based on each initial image feature and each position query vector, wherein the text recognition result includes at least one of the following: the type of each area in the image to be recognized, the coordinate information of each area in the image to be recognized, and the character information of each area in the image to be recognized.

[0006] The present application provides a text recognition device, including: an acquisition module, an encoding module and a determination module; the acquisition module is used to acquire at least one initial image feature using a text recognition model, and each initial image feature is obtained by extracting features from an image to be recognized; the encoding module is used to perform position encoding on each initial image feature to obtain a position query vector corresponding to each initial image feature; the determination module is used to determine a text recognition result of the image to be recognized based on each initial image feature and each position query vector, and the text recognition result includes at least one of the following: the type of each area in the image to be recognized, the coordinate information of each area in the image to be recognized, and the character information of each area in the image to be recognized.

[0007] The present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned text recognition method.

[0008] The present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the above-mentioned text recognition method is implemented.

[0009] The above scheme uses a text recognition model to obtain at least one initial image feature, and then performs position encoding based on each initial image feature to obtain a position query vector corresponding to each initial image feature, wherein each position query vector and each initial image feature can combine feature information of different scales and different positions, thereby improving the accuracy of determining the text recognition result of the image to be recognized.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0012] Figure 1 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 1 ; Figure 2 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 2 ; Figure 3 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 3 ; Figure 4 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 4 ; Figure 5 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 5 ; Figure 6 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 6 ; Figure 7 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 7 ; Figure 8a is a schematic diagram of a preset decoding layer in an embodiment of the text recognition method of the present application; Figure 8b is a schematic diagram of a sample image in an embodiment of a text recognition method of the present application; Figure 8c is a schematic diagram of a training module in an embodiment of a text recognition method of the present application; Figure 8d is a schematic diagram of a visual query vector construction module in an embodiment of a text recognition method of the present application; Figure 8e Schematic diagram of a text-guided query vector construction module in an embodiment of the text recognition method of the present application; Figure 8f is a schematic diagram of a character-guided query vector construction module in an embodiment of the text recognition method of the present application; Fig. 9 It is a structural diagram of an embodiment of a text recognition device of the present application; Fig.10 It is a structural schematic diagram of an embodiment of the electronic device of the present application; Fig.11 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0013] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0014] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0015] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0016] The present application provides some text recognition methods and text recognition devices. The application scenarios of the text recognition method include text recognition of images with text in natural scenes. The execution subject of the text recognition method can be any device with text recognition function, for example, a text recognition device. For example, the text recognition device can be set in a terminal device or a server or other processing device, wherein the terminal device can be a device for text recognition, a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, etc. In some possible implementations, the text recognition method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0017] See also Figure 1 , Figure 1 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 1 Specifically, the text recognition method may include the following steps: Step S11: using a text recognition model to obtain at least one initial image feature.

[0018] Each initial image feature is obtained by extracting features from the image to be identified. Each initial image feature can be obtained by extracting features from the image to be identified in a natural scene. Among them, the image to be identified can be an image containing text to be identified. Specifically, the image to be identified can be a natural scene picture, a document scan, a handwritten font picture, a screenshot, an emoticon package with text information, a design work, and a medical imaging material. Among them, the natural scene picture can be a picture with text information such as a signboard, billboard, store sign, road sign, menu, product packaging, etc. on the street. Among them, the document scan can be a digital scan of a document such as a contract, file, report, invoice, business card, etc., and the text information therein needs to be extracted. Among them, the handwritten font picture can be a handwritten text picture of different styles and qualities including handwritten notes, letters, envelopes, business cards, calligraphy works, etc. Among them, the screenshot can be a screenshot picture containing text information such as an application interface, a webpage screenshot, a mobile phone page screenshot, etc. Among them, the artwork or design work can be text information in artworks or design works such as posters, advertising design drafts, book covers, paintings, etc. The medical imaging data may be pictures in the medical field such as medical reports, imaging diagnosis reports, annotation information in medical images, etc. This is only an example and does not limit the type of image to be identified.

[0019] The text recognition model may be a model with text recognition function deployed on the text recognition device. The input of the text recognition model may be an image to be recognized, and the output of the text recognition model may be a text recognition result of the image to be recognized. It is understood that each image to be recognized may correspond to at least one initial image feature. Each image to be recognized corresponds to a text recognition result.

[0020] In some application scenarios, the way to extract features from the image to be identified may be to input the image to be identified into the backbone network. The choice of the backbone network is relatively flexible, and it can be any backbone network with strong feature extraction capabilities, such as the ResNet series, the Swin Transformer series, etc. In other application scenarios, the way to extract features from the image to be identified may be to input the image to be identified into the backbone network and the feature encoder in sequence, so as to learn the global information of the image to be identified, adjust the attention of the features at different positions in the image to be identified, strengthen the key features, weaken the irrelevant features, and obtain at least one initial image feature. In some application scenarios, the initial image features may include image features obtained by extracting features from the image to be identified. The at least one initial image feature may be an initial image feature corresponding to the image to be identified, or it may be multiple initial image features corresponding to the image to be identified. It is understandable that at least one refers to one or more.

[0021] Step S12: performing position encoding on each initial image feature to obtain a position query vector corresponding to each initial image feature.

[0022] Each initial image feature may correspond to a feature vector. Each position query vector may be a feature vector having richer feature information than the corresponding initial image feature and having image position information.

[0023] Position coding may be to encode each initial image feature according to a preset coding method. In some application scenarios, the preset coding method may be to directly encode each initial image feature through a preset position, and use the initial image feature encoded by the preset position as the position query vector corresponding to the initial image feature. Among them, the preset position coding may be cosine position coding or sine position coding. In other application scenarios, the preset coding method may also be to extract the preset feature from the initial image feature encoded by the preset position to obtain the position query vector corresponding to the initial image feature. Among them, the preset feature extraction may be to input the initial image feature encoded by the preset position into a preset feature extraction module. The preset feature extraction module is used to extract features from the input features. Among them, a preset feature extraction network may be provided on the preset feature extraction module. Specifically, the preset feature extraction network may be a feature extraction network such as a convolutional neural network, a recurrent neural network, a long short-term memory network, a gated recurrent unit, a BiGRU neural network, an attention mechanism, etc. Exemplarily, the preset feature extraction network in each prediction module may be a long short-term memory network (LSTM, Long Short-Term Memory) in a recurrent neural network. In other application scenarios, the preset feature extraction network may be a preset perceptron network. The preset perceptron network may be obtained by at least one perceptron module. Each perceptron module in the at least one perceptron module may be a multi-layer perceptron (MLP). Each initial image feature may be grouped into an initial image feature set. The above step S12 may be to use each initial image feature or initial image feature set as an input of the preset perceptron network to obtain a position query vector of each initial image feature output by the preset perceptron network.

[0024] Step S13: Determine the text recognition result of the image to be recognized based on each initial image feature and each position query vector.

[0025] The text recognition result includes at least one of the following: the type of each region in the image to be recognized, the coordinate information of each region in the image to be recognized, and the character information of each region in the image to be recognized.

[0026] Each region in the image to be identified may be an image region corresponding to at least one image block in the image to be identified. The type of each region in the image to be identified includes a foreground type or a background type. The foreground region may represent an image region where a text object is located. The background region may represent an image region where a non-text object is located. The coordinate information of each region in the image to be identified may represent the position information of the region relative to the image to be identified. The character information of each region in the image to be identified may represent the prediction result of the text object corresponding to the region. It can be understood that, in the case where the type of any region in the image to be identified is a foreground type, the character information of the region in the image to be identified may represent the prediction result of the text object corresponding to the region. In the case where the type of any region in the image to be identified is a background type, the character information of the region in the image to be identified may represent the preset text of the non-text object corresponding to the region. Among them, the preset text may be set to a value of 0.

[0027] Optionally, a text recognition model may be pre-trained, and each initial image feature and each position query vector may be used as input of the text recognition model, and the output of the text recognition model may be the text recognition result of the image to be recognized. Alternatively, the output of the text recognition model may be post-processed to obtain the text recognition result of the image to be recognized. The post-processing may be to select the text recognition result with the highest confidence from the text recognition results corresponding to each initial image feature as the text recognition result of the image to be recognized.

[0028] The above scheme uses a text recognition model to obtain at least one initial image feature, and then performs position encoding based on each initial image feature to obtain a position query vector corresponding to each initial image feature, wherein each position query vector and each initial image feature can combine feature information of different scales and different positions, thereby improving the accuracy of determining the text recognition result of the image to be recognized.

[0029] In some embodiments, the above step S11 may include the following steps: first, using a text recognition model to obtain an image to be recognized. Then, multi-scale feature extraction is performed on the image to be recognized to obtain a multi-scale feature set corresponding to the image to be recognized. The multi-scale feature set contains at least two scale features and the scales of each scale feature are different. Next, feature fusion is performed on each scale feature in the multi-scale feature set to obtain at least one initial image feature.

[0030] The method of acquiring the image to be identified may be to acquire the image to be identified in real time through an image acquisition device, or to use the image to be identified stored in a database, or to capture a frame of image in a video file.

[0031] After obtaining the image to be identified, the image to be identified is used as the input of the backbone network, and multi-scale feature extraction of the image to be identified is implemented in the backbone network to obtain a multi-scale feature set corresponding to the image to be identified output by the backbone network. The multi-scale feature set contains at least two scale features and the scales of each scale feature are different.

[0032] In some application scenarios, the method of fusing the scale features in the multi-scale feature set to obtain at least one initial image feature may be to directly combine the scale features in the scale feature set arbitrarily to obtain at least one initial image feature with the same number of scale features. In other application scenarios, the method of fusing the scale features in the multi-scale feature set to obtain at least one initial image feature may be to splice the scale features in the scale feature set to obtain at least one initial image feature with the same number of scale features. In other application scenarios, the method of fusing the scale features in the multi-scale feature set to obtain at least one initial image feature may be to set a weight for each scale feature according to the distance from the current scale feature, and then for each current scale feature, based on the weight of each scale feature, fuse all scale features to obtain the initial image feature corresponding to the current scale feature. In other application scenarios, the method of fusing the scale features in the multi-scale feature set to obtain at least one initial image feature may be to use each scale feature in the scale feature set as an input of a preset feature encoder to obtain the initial image feature corresponding to each scale feature output by the preset feature encoder.

[0033] Exemplarily, after the backbone network is used to extract multi-scale features from the image to be identified to obtain multiple scale features, the above-mentioned method of fusing the scale features in the multi-scale feature set to obtain at least one initial image feature may be: performing a convolution operation on the scale features in the scale feature set to unify the number of channels of the multiple scale features to obtain each processed scale feature, and flattening and splicing the processed scale features to obtain a spliced ​​feature. The spliced ​​feature can be expressed as In the spliced ​​features, B can represent the batch size, S represents the sum of the width and height products of each scale feature, and C represents the number of channels of the spliced ​​features. The spliced ​​features are used as the input of the preset feature encoder to obtain the initial image features corresponding to each scale feature output by the preset feature encoder. Each initial image feature output by the preset feature encoder can be represented as . Each initial image feature In , B may represent a batch size, S represents the sum of the width and height products of each scale feature, and C represents the number of channels of each initial image feature. The preset feature encoder may be a Transformer feature encoder.

[0034] It can be understood that by sending the spliced ​​features F into the Transformer feature encoder, the global information of the image to be recognized can be learned, the attention of features at different positions in the spliced ​​features F can be adjusted, key features can be strengthened, irrelevant features can be weakened, and each initial image feature can be obtained. In addition, each initial image feature has global context information, so that the text recognition results obtained by using each initial image feature in subsequent processing are more comprehensive and accurate.

[0035] See also Figure 2 , Figure 2 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 2 .

[0036] In some embodiments, the above step S12 may include the following steps: for each initial image feature, perform the following Figure 2 The following steps are shown: Step S21: Input the initial image feature into the first perceptron module to obtain the sampling point coordinate information of the initial image feature output by the first perceptron module. Step S22: Use the sampling point coordinate information processed by the preset encoding as the input of the second perceptron module to obtain the position query vector corresponding to the initial image feature output by the second perceptron module.

[0037] The first perception module and the second perception module may be a multi-layer perceptron (MLP), wherein the first perception module and the second perception module may have the same structure but different parameters.

[0038] The sampling point coordinate information of each initial image feature may be the position information of the sampling point coordinates of a preset number of Bezier curves related to the image to be identified in the image to be identified. The preset coding process may be the preset position coding described above. Specifically, the preset position coding may be a cosine position coding or a sine position coding.

[0039] In some application scenarios, after step S21, the sampling point coordinate information of the initial image feature is subjected to a preset encoding process to obtain the sampling point coordinate information subjected to the preset encoding process. In other application scenarios, before the above step S21, the initial image feature is input into a preset linear layer to obtain the confidence corresponding to the initial image feature. Based on the confidence ranking of the initial image feature, at least one initial image feature that meets the preset ranking is selected from each initial image feature as the target initial image feature. The preset ranking may be the top k in the confidence ranking of the initial image feature. Among them, the confidence ranking of the initial image feature is obtained by the confidence corresponding to each initial image feature. Then, the above step S21 may be that for each target initial image feature, the target initial image feature is input into the first perceptron module to obtain the sampling point coordinate information of the initial image feature corresponding to the target initial image feature output by the first perceptron module.

[0040] In some other application scenarios, the above step S22 may be to use the sampling point coordinate information processed by the preset encoding as the input of the second perceptron module to obtain the initial position query vector corresponding to the initial image feature output by the second perceptron module. Obtain the content query vector. Among them, the content query vector may be randomly initialized or a preset query vector. It is understandable that the scale of the content query vector and the scale of the initial position query vector may be the same scale, or the scale of the initial position query vector may be higher than the scale of the content query vector. The initial position query vector corresponding to the above initial image feature is merged with the content query vector to obtain the position query vector corresponding to the initial image feature.

[0041] Exemplarily, constructing the position query vector corresponding to each initial image feature may be performed by the initial position query vector and content query vector Among them, each initial image feature The probability score of each initial image feature is output through the classification linear layer Linear, and the index of the top k features ranked from high to low is obtained. Input the Bezier coordinate converter composed of the first perceptron module MLP to obtain the L sampling point coordinates of the central Bezier curve corresponding to each text instance, where L refers to the uniform text length. According to the first k indexes, the L sampling point coordinates of the central Bezier curve corresponding to the top k features with the highest scores are obtained. , and the sampling point coordinates The sampling point coordinate information is used as the initial image feature. The sampling point coordinate information is cosine position encoded and input into the position encoder composed of the second perceptron module to obtain the initial position query vector The position query vector corresponding to each initial image feature in step S22 is obtained by adding the position query vector and the content query vector. . Content Query Vector Obtained by random initialization.

[0042] In some application scenarios, the above step S13 may be to fuse the initial image features with the position query vectors to obtain target fusion features corresponding to the initial image features. Then, based on the target fusion features, the text recognition result of the image to be recognized is determined.

[0043] See also Figure 3 , Figure 3 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 3 .

[0044] In some embodiments, the above step S13 may include the following steps: Step S31: Input each initial image feature and each position query vector into a multi-layer decoder for feature fusion to obtain a target fusion feature corresponding to each initial image feature. Step S32: Determine the text recognition result of the image to be recognized based on each target fusion feature.

[0045] The multi-layer decoder may be a decoder having at least two preset feature extraction networks. For example, the multi-layer decoder may be a decoder having at least two attention mechanisms. The target fusion feature corresponding to each initial image feature includes feature information combined between the initial image feature and the position query vector corresponding to the initial image feature.

[0046] The above step S13 may be to input each initial image feature and each position query vector into a multi-layer decoder to obtain a target fusion feature corresponding to each initial image feature output by the multi-layer decoder.

[0047] See also Figure 4 , Figure 4 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 4 .

[0048] In some embodiments, the multi-layer decoder includes several preset decoding layers arranged in cascade, each preset decoding layer has the same structure but different parameters, and the features output by the last preset decoding layer are used as the features output by the multi-layer decoder. The features output by the multi-layer decoder are the target fusion features corresponding to the features of each initial image. Each preset decoding layer includes an intra-group attention module. The above step S31 may include the following steps: Step S41: Using the first query key-value pair as the input of the intra-group attention module in the current preset decoding layer, the first feature output by the intra-group attention module in the current preset decoding layer is obtained. The query data, key data and value data in the first query key-value pair are respectively obtained from the query vectors of each position or from the features output by the previous preset decoding layer. The current preset decoding layer is one of several preset decoding layers. Step S42: The first feature output by the intra-group attention module in the current preset decoding layer is fused with the input of the intra-group attention module in the current preset decoding layer to obtain the first fused feature. Step S43: Advanced fusion processing is performed on the first fused feature and each initial image feature to obtain the feature output by the current preset decoding layer.

[0049] The multi-layer decoder may include several preset decoding layers arranged in cascade, each preset decoding layer having the same structure but different parameters, and the features output by the last preset decoding layer are used as the features output by the multi-layer decoder. In some application scenarios, each preset decoding layer may include a feature fusion module that can directly fuse the features of each initial image feature and each position query vector. In other application scenarios, each preset decoding layer includes a cross-attention module. Among them, the cross-attention module included in each preset decoding layer can fuse the features of each initial image feature and each position query vector to obtain the target fusion feature corresponding to the initial image feature.

[0050] In other application scenarios, each preset decoding layer may include a single cross-attention module, or may include other attention modules in addition to the cross-attention module. In the case where the preset decoding layer includes the cross-attention module and other attention modules, the cross-attention module and other attention modules may be arranged in series or in parallel. The arrangement order of the cross-attention module and other attention modules in each preset decoding layer is not limited here.

[0051] It can be understood that the first feature output by the in-group attention module in the current preset decoding layer includes the first feature corresponding to each initial image feature. The first fused feature includes the first fused feature corresponding to each initial image feature obtained by fusing the first feature corresponding to each initial image feature with each feature input by the in-group attention module in the current preset decoding layer.

[0052] In some application scenarios, each position query vector or the features output by the previous preset decoding layer are directly used as the query data, key data and value data in the first query key-value pair. In other application scenarios, each position query vector or the features output by the previous preset decoding layer are mapped to obtain each mapped position query vector or the features output by the previous preset decoding layer after mapping. And each mapped position query vector or the features output by the previous preset decoding layer after mapping are used as the query data, key data and value data in the first query key-value pair in the above-mentioned current preset decoding layer. Specifically, the mapping process can be achieved through different linear transformations. Thereby ensuring that the query data, key data and value data in the first query key-value pair are in the same vector space.

[0053] In some application scenarios, when the current preset decoding layer is the first preset decoding layer in a multi-layer decoder, the input of the intra-group attention module in the current preset decoding layer is the first query key-value pair. The query data, key data and value data in the first query key-value pair are obtained from each position query vector respectively. The first feature output by the intra-group attention module in the current preset decoding layer is fused with the input of the intra-group attention module in the current preset decoding layer, that is, the first feature output by the intra-group attention module in the current preset decoding layer is fused with each position query vector to obtain the first fused feature. The specific fusion processing method may be to perform cross-layer addition processing, and use the first feature output by the intra-group attention module in the current preset decoding layer and each position query vector as the input of the first preset interaction module to obtain the first fused feature output by the first preset interaction module. Among them, the first preset interaction module may be a network module or an algorithm module provided with a residual connection or a jump connection.

[0054] In other application scenarios, when the current preset decoding layer is not the first preset decoding layer in the multi-layer decoder, the input of the intra-group attention module in the current preset decoding layer is the first query key-value pair. The query data, key data and value data in the first query key-value pair are respectively obtained from the features output by the previous preset decoding layer. The first feature output by the intra-group attention module in the current preset decoding layer is fused with the input of the intra-group attention module in the current preset decoding layer, that is, the first feature output by the intra-group attention module in the current preset decoding layer is fused with the feature output by the previous preset decoding layer to obtain the first fused feature. The specific fusion processing method can be to perform cross-layer addition processing, and use the first feature output by the intra-group attention module in the current preset decoding layer and the feature output by the previous preset decoding layer as the input of the first preset interaction module to obtain the first fused feature output by the first preset interaction module.

[0055] The advanced fusion processing may be cross-fusion processing of at least two features. Specifically, the advanced fusion processing may be inputting the first fusion feature and each initial image feature into a cross-attention module in each preset decoding layer to obtain a feature output by the cross-attention module.

[0056] In some application scenarios, the above step S43 can be to input the first fusion feature and each initial image feature into the cross-attention module in the current preset decoding layer for cross-attention processing to obtain the output feature of the cross-attention module, and use the output feature of the cross-attention module as the output feature of the current preset decoding layer.

[0057] It can be understood that by first processing the position query vector through the intra-group attention module, the first fusion feature obtained subsequently can be advancedly fused with each initial image feature, so that the feature output by the multi-layer decoder is a feature with richer feature information, thereby improving the accuracy of subsequent text recognition results.

[0058] See also Figure 5 , Figure 5 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 5 .

[0059] In some embodiments, each preset decoding layer includes a cross-attention module, and the above-mentioned step S43 may include the following steps: Step S51: Use the second query key-value pair as the input of the cross-attention module in the current preset decoding layer to obtain the second feature output by the cross-attention module in the current preset decoding layer. The query data in the second query key-value pair is obtained by the first fused feature, and the key data and value data in the second query key-value pair are respectively obtained by each initial image feature. Step S52: Fuse the second feature output by the cross-attention module in the current preset decoding layer with the first fused feature to obtain a second fused feature. Step S53: Based on the second fused feature, obtain the feature output by the current preset decoding layer.

[0060] In some application scenarios, the first fused feature is directly used as the query data in the second query key-value pair, and all the initial image features are directly used as the key data and value data in the second query key-value pair. In other application scenarios, the first fused feature is mapped to obtain the mapped first fused feature. And all the initial image features are mapped to obtain the mapped initial image features. And each mapped initial image feature is used as the key data and value data in the second query key-value pair, and the mapped first fused feature is used as the query data in the second query key-value pair. Specifically, the mapping process can be achieved through different linear transformations. Thereby ensuring that the query data, key data and value data in the second query key-value pair are in the same vector space.

[0061] The second feature output by the cross-attention module in the current preset decoding layer is fused with the first fused feature to obtain the second fused feature, which can be obtained by cross-layer addition of the feature output by the cross-attention module and the input of the cross-attention module to obtain the feature output by the processed cross-attention module. It can be understood that the feature output by the cross-attention module can be the second feature, the input of the cross-attention module can be the first fused feature, and the fusion of the second feature and the first fused feature can be a cross-layer addition process. Specifically, the second feature and the first fused feature are input into the second preset interaction module for fusion processing to obtain the second fused feature. Among them, the second preset interaction module can be a network module or an algorithm module that is provided with a residual connection or a jump connection.

[0062] In some application scenarios, the above step S53 may be to directly use the second fused feature as the feature output by the current preset decoding layer. In other application scenarios, the above step S53 may be to input the second fused feature into a feedforward neural network for linear processing to obtain the feature output by the feedforward neural network. The feature output by the feedforward neural network and the second fused feature are input into the target preset interaction module for cross-layer addition processing to obtain the feature output by the target preset interaction module, and the feature output by the target preset interaction module is used as the feature output by the current preset decoding layer. Among them, the target preset interaction module may be a network module or algorithm module that is capable of performing residual connection or jump connection.

[0063] In some application scenarios, the above step S32 may be to input each target fusion feature into at least one linear layer to obtain a text recognition result corresponding to each target fusion feature. A text recognition result that satisfies a preset condition is selected from the text recognition results corresponding to each target fusion feature as the text recognition result of the image to be recognized. The preset condition may be to use the character information with the highest confidence in the text recognition result corresponding to the target fusion feature as the text recognition result of the image to be recognized.

[0064] See also Figure 6 , Figure 6 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 6 .

[0065] In some embodiments, the above step S32 may include the following steps: Step S61: Based on each target fusion feature, determine the confidence ranking corresponding to each target fusion feature, and use the target fusion feature with the largest confidence ranking as the target feature. Step S62: Input the target feature into the first linear layer, the second linear layer, and the third linear layer for mapping processing, respectively, to obtain the type of each area in the image to be recognized output by the first linear layer, the coordinate information of each area in the image to be recognized output by the second linear layer, and the character information of each area in the image to be recognized output by the third linear layer.

[0066] The first linear layer, the second linear layer, and the third linear layer have the same structure but different parameters. The first linear layer is used to classify the scene of each input target fusion feature to obtain the type of each area in the image to be identified. The type of each area in the identified image includes a foreground type or a background type. The second linear layer is used to determine the coordinate information of each input target fusion feature to obtain the coordinate information of each area in the image to be identified. The third linear layer is used to determine the character information of each input target fusion feature to obtain the coordinate information of each area in the image to be identified.

[0067] In other application scenarios, the above step S32 may be inputting the target fusion feature into a preset linear layer to obtain the confidence corresponding to the target fusion feature. Based on the confidence ranking of the target fusion feature, the target fusion feature that meets the preset ranking is selected from each target fusion feature as the target feature. The preset ranking may be the top m in the confidence ranking of the target fusion feature. Among them, the confidence ranking of the target fusion feature is obtained by the confidence corresponding to each target fusion feature. The top m may be 1. Perform at least one preset post-processing on the target feature to obtain the text recognition result after each preset post-processing. Among them, when the text recognition results are at least one of the type of each area in the image to be recognized, the coordinate information of each area in the image to be recognized, and the character information of each area in the image to be recognized, the various preset post-processings match the type judgment of the area in the text recognition result, the determination of the coordinate information of the area, and the determination of the character information of the area.

[0068] See also Figure 7 , Figure 7 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 7 .

[0069] In some embodiments, the above-mentioned text recognition method may further include a training step for the text recognition model, and the training step includes: Step S71: Obtain a sample image and a sample recognition result corresponding to the sample image. The sample recognition result is obtained through pre-annotation, and the sample recognition result includes real character information and a text-guided query vector related to the real character information. Step S72: Based on the sample image and the text-guided query vector, obtain the character-guided query vector corresponding to the sample image. Step S73: Use the sample image, the text-guided query vector and the character-guided query vector as inputs to the text recognition model to obtain a predicted recognition result corresponding to the sample image. Step S74: Based on the difference between the sample recognition result and the predicted recognition result, obtain the target loss. Step S75: Use the target loss to adjust the parameters in the text recognition model to obtain a trained text recognition model.

[0070] The sample image is an image of the same type as the above-mentioned image to be identified, or an image of a different type. The sample image contains real character information corresponding to the text object. The specific definition of the sample image can refer to the above-mentioned image to be identified, which will not be repeated here. The above-mentioned step S71 may include constructing features for the sample image to obtain a text-guided query vector for the sample image. The text-guided query vector of each sample image can be, for each sample image, segmenting the real character information of the sample image to obtain an index sequence of the sample image. A masking operation is performed on the index sequence of the sample image to obtain a masked index sequence corresponding to the sample image. A mapping operation is performed on the masked index sequence to obtain a text-guided query vector for the sample image.

[0071] Exemplarily, the text-guided query vector construction module mainly includes a word segmenter and a masker. The process of constructing a text-guided query vector may include: segmenting all the texts of each sample image through the subword segmenter of the commonly used language model. Specifically, for each sample image, first obtain the word segmentation index sequence corresponding to each text in the sample image, and use the [SEP] index to connect different texts so that each image corresponds to an index sequence. When training the text recognition model, padding is performed according to the longest sequence length T in the same batch of images to ensure that the sequence length of a batch is the same, and the [PAD] index is used as the padding flag. A random masking operation is performed on the index sequence corresponding to each text in the sequence, specifically randomly masking each text index sequence. The word index is replaced by the [MASK] index to ensure the semantic information of the text. The value of should not be too large. The complete sequence after masking is mapped to the corresponding text embedding feature according to the index in it to obtain the text guided query vector .

[0072] In the application process of the text recognition model, the text recognition model includes an acquisition module, a preset perceptron module, a multi-layer decoder and at least one linear layer. The acquisition module is used to obtain the input of the text recognition model. The preset perceptron module includes a first perceptron module and a second perceptron module. Among them, the input of the preset perceptron module can be each initial image feature, and the output of the preset perceptron module can be a position query vector corresponding to each initial image feature. The input of the multi-layer decoder can be each initial image feature and a position query vector corresponding to each initial image feature, and the output of the multi-layer decoder can be a target fusion feature. The input of at least one linear layer can be a target fusion feature or a sample target fusion feature, and the output of at least one linear layer can be a text recognition result of the image to be recognized or a predicted recognition result of a sample image.

[0073] See also Figure 8a , Figure 8aIt is a schematic diagram of a preset decoding layer in an embodiment of the text recognition method of the present application.

[0074] In some application scenarios, the multi-layer decoder in the text recognition model also includes a target group attention module, an auxiliary group attention module and an inter-group attention module. In the application process of the text recognition model, that is, in the above step S31, only the target group attention module, the auxiliary group attention module and the target group attention module in the inter-group attention module are used as the above-mentioned group attention module, and only the target group attention module participates in the processing calculation in the multi-layer decoder. In the training process of the text recognition model, the target group attention module, the auxiliary group attention module and the target group attention module in the inter-group attention module all participate in the processing calculation in the multi-layer decoder. The input of the target group attention module is the candidate feature. When the inter-group attention module participates in the processing calculation of the multi-layer decoder, the input of the inter-group attention module is the output of the target group attention module. When the inter-group attention module does not participate in the processing calculation of the multi-layer decoder, the output of the target group attention module is directly used as the input of the first preset interaction module. The input of the auxiliary group attention module is the text-guided query vector. In addition, the preset interaction modules in any preset decoding layer in the multi-layer decoder can all be Add&Norm modules.

[0075] In the training process of the text recognition model, the above step S72 may be inputting the sample image and the text-guided query vector into the text recognition model to obtain the features output by the multi-layer decoder in the text recognition model. It is understandable that this is the first time that the sample image and the text-guided query vector are input into the text recognition model. And the features output by the multi-layer decoder when the text recognition model is first input are used as the character-guided query vector corresponding to the sample image. In the case of the first input into the text recognition model, the sample image and the text-guided query vector are sequentially passed through the above acquisition module, the preset perceptron module, and the multi-layer decoder, and the features output by the multi-layer decoder are used as the character-guided query vector corresponding to the sample image. The above step S73 may be passing the sample initial image features of the sample image through the preset perceptron module to obtain the sample position query vector. And the sample position query vector is used as a candidate feature. It is understandable that in the case of the first input into the text recognition model, the input of the attention module in the target group is the candidate feature, and the candidate feature is only the sample position query vector. In the case of the first input into the text recognition model, the attention module in the group does not participate in the processing and calculation of the multi-layer decoder. After the character-guided query vector is determined, the state is determined to be the non-first input into the text recognition model. In the case of a non-first input text recognition model, the intra-group attention module does not participate in the processing calculations of the multi-layer decoder. In the case of a non-first input text recognition model, the target intra-group attention module, the auxiliary intra-group attention module, and the target intra-group attention module in the inter-group attention module all participate in the processing calculations in the multi-layer decoder. In the case of a non-first input text recognition model, the output of the multi-layer decoder is used as the sample target fusion feature. Based on each sample target fusion feature, the predicted recognition result of the sample image is determined. Specifically, the predicted recognition result of the sample image can be determined with reference to the process of steps S61 to S62 above, which will not be repeated here. The predicted recognition result of the sample image includes at least one of the following: the sample type of each area in the sample image, the sample coordinate information of each area in the sample image, and the sample character information of each area in the sample image.

[0076] The above step S74 may be to determine the target loss function according to the difference between the real character information in the sample recognition result and the sample character information in each area in the predicted recognition result of the sample image. It can be considered that, when the sample image is sufficient data, the parameters in the text recognition model are trained using the determined target loss until the scene detection model converges to obtain the trained text recognition model. The above steps S11 to S13 are executed using the trained text recognition model.

[0077] Exemplarily, a loss function is constructed, and a scene text recognition model is trained using a data set. When training the text recognition model, only the sample position query vector follows the strategy of assigning labels using the Hungarian algorithm in DETR, and the character-guided query vector calculates the loss of the corresponding task based on the pre-assigned labels. The loss functions calculated for the outputs of the sample position query vector and the character-guided query vector include: the channel dimension C of the sample type output by the classification head in the first linear layer uses the FocalLoss loss function, the sampling point coordinates P in the sample coordinate information output by the regression head in the second linear layer uses the L1 Loss, and the sample character information T output by the text head in the third linear layer uses the standard cross entropy loss function for supervision. The output TC of the text-guided query vector is supervised using the cross entropy loss function, and the corresponding label is an index sequence without masking operation. Exemplarily, the training of the text recognition model also includes constructing a character-guided query vector. In some application scenarios, when training the text recognition model, the features of the sample position query vector first output by the multi-modal multi-layer decoder are used as the character-guided query vector. In other application scenarios, after the features of the sample position query vector first output by the multi-layer decoder are post-processed and selected by the confidence threshold, p candidate results remain from the k candidate results, and the sampling points of the Bezier curves of the upper and lower edges of the predicted text are used to form a polygon surrounding the text. The IOU of the predicted result and the label polygon is calculated, and the p results are assigned to the corresponding labels. The text content of the p predicted results is compared with the text content of the corresponding label, and the predicted results are corrected by the minimum edit distance. The characters that have changed before and after the correction are referred to by the [UNK] symbol, and then the processed text results are input into the character segmenter to obtain the embedding vector corresponding to each character to obtain the character query vector. .

[0078] In some application scenarios, the training of the text recognition model also includes constructing a multi-modal multi-layer decoder, which consists of multiple preset decoding layers with the same structure and three task heads consisting of linear layers. , sample location query vector , text-guided query vector and character-guided query vector After inputting the output of the last decoding layer into the multimodal decoder and then inputting it into the respective task heads, the predicted text result of the sample image can be obtained. The specific steps can be referred to as follows: The structure of each preset decoding layer is as follows Figure 8aAs shown, the cross attention module can be a structure in Transformer, not a specific structure. The first preset interaction module, the second preset interaction module and the target preset interaction module can all be Add&Norm modules, which can perform addition operations and layer normalization operations on features. The feedforward neural network can be represented as FFN. The character query vector is obtained by transforming the model output prediction results, so the attention mask map needs to be used in the calculation of the inter-group attention module, that is, The first input uses the middle part of the attention mask to learn sample position query The relationship between different query vectors in the sample location query Second and character guided query vector When input together, the calculation of the first input can be omitted, and the upper half of the attention mask map is used to make the characters guide the query vector Can learn sample position query vectors related to the character itself This operation ensures that the sample position query vector follows the causal relationship and is not affected by the output result. The black squares in the mask map represent masks and do not calculate attention.

[0079] Text-guided query vectors The semantic relationship within each text in the image is calculated through the intra-group attention module. Different texts are separated by [SEP]. At least part of the attention mask map is used in the intra-group attention module to control each text to only learn the semantic relationship between its own internal word vectors and ignore other text information. The target intra-group attention module shares parameters with the auxiliary intra-group attention module, so that the auxiliary intra-group attention module simultaneously learns the feature information and semantic information of the sample position query vector in the same text. Text-guided query vector Help the network understand the text semantics by predicting the correct word segmentation index.

[0080] After the sample position query vector, the text-guided query vector, and the character-guided query vector pass through the target group intra-attention module, the auxiliary attention module, and the inter-group attention module, the output of the first preset interaction module is compared with the sample initial image feature. Input the cross attention module to calculate the cross attention, so that the sample position query vector and character-guided query vector In the sample initial image features Perceive information related to the position of the text and each of its constituent characters, and let the text guide the query vector In the initial image features of the sample The text semantic information is semantically associated with the corresponding visual features in the interaction, which improves the ability of the multimodal decoder to learn the text context semantic information and improves the accuracy of text recognition. The output of the three through the cross attention module is passed through the feedforward neural network FFN and the target preset interaction module Add&Norm, and then output to the last preset decoding layer to obtain the sample target fusion features corresponding to the sample position query vector. , sample target fusion features corresponding to the text-guided query vector , sample target fusion features corresponding to the character-guided query vector . The output of the last preset decoding layer , Input to the classification head, regression head, and text head composed of linear layers to obtain the corresponding category confidence of each query vector after decoding , sampling point coordinates of the upper and lower edge Bezier curves , text content , where m is the total number of character categories in the character library plus 2. The output of the decoding layer Input to the word classification head composed of linear layers to predict the category index of each word vector , N represents the total number of word categories in the vocabulary.

[0081] See also Figure 8b , Figure 8b It is a schematic diagram of a sample image in an embodiment of the text recognition method of the present application.

[0082] The sample image is any scene text image data in the dataset. The sample image contains the text position and character annotations. Figure 8b As shown, the sample images in the data set are scene images containing text, and the label of each sample image is converted into the format of the coordinates of the four control points of the Bezier curve of the upper and lower edges of the text in the image and the character annotation. The character annotation means that the above sample recognition results include real character information. Specifically, each sample image includes at least character annotation and Bezier curve. In other application scenarios, each sample image also contains its own pattern elements. Among them, Figure 8b Points P1 to P4 in represent Bezier curve control points, and a smooth edge curve can be generated through the coordinates of the control points. To facilitate the training of the text recognition model, the text annotation is padded to a uniform length L, and the padded value is the total number of character categories in the character library plus 1.

[0083] See also Figure 8c , Figure 8c It is a schematic diagram of a training module in an embodiment of the text recognition method of the present application.

[0084] like Figure 8cThe training module shown is used to train the text recognition model of the present application. The training module is used to execute the above steps S71 to S75 to obtain a trained text recognition model. The text recognition model includes a multi-scale feature extraction module, a preset feature encoder, a visual query vector construction module, a multi-layer decoder, a text-guided query vector construction module, and a character-guided query vector construction module. The sample image is sequentially passed through the multi-scale feature extraction module and the preset feature encoder to obtain each initial image feature in the initial image feature set. Each initial image feature is passed through the visual query vector construction module to obtain a sample position query vector Q of the sample image. V The real character information is passed through the text-guided query vector construction module to obtain the text-guided query vector Q related to the real character information. T . The sample position query vector Q of each initial image feature and sample image V And the text-guided query vector Q related to the real character information T The output obtained by inputting the multi-layer decoder is used as the input of the character-guided query vector construction module to obtain the character-guided query vector Q C . The sample position query vector Q of each initial image feature and sample image V And the text-guided query vector Q related to the real character information T and the character-guided query vector Q C The output obtained by inputting the multi-layer decoder is used as the sample target fusion feature, and the sample target fusion feature is processed through the at least one linear layer to obtain the prediction recognition result of the sample image.

[0085] See also Figure 8d , Figure 8d It is a schematic diagram of a visual query vector construction module in an embodiment of the text recognition method of the present application.

[0086] Exemplarily, the sample initial image feature set includes a plurality of sample initial image features. Constructing the sample position query vector corresponding to each sample initial image feature may be performed by the sample initial position query vector and sample content query vector Among them, the initial image features of each sample The probability score of each sample's initial image feature is output through the classification linear layer Linear, and the index of the top k features ranked from high to low is obtained. Input the Bezier coordinate converter composed of the first perceptron module MLP to obtain the L sampling point coordinates of the central Bezier curve corresponding to each text instance, where L refers to the uniform text length. According to the first k indexes, the L sampling point coordinates of the central Bezier curve corresponding to the top k features with the highest scores are obtained. , and the sampling point coordinates The sampling point coordinate information is used as the initial image feature. The sampling point coordinate information is cosine position encoded and input into the position encoder composed of the second perceptron module to obtain the sample initial position query vector The sample position query vector corresponding to the initial image feature of each sample is obtained by adding the sample position query vector and the sample content query vector. Sample content query vector Obtained by random initialization.

[0087] See also Figure 8e , Figure 8e It is a schematic diagram of a text-guided query vector construction module in an embodiment of the text recognition method of the present application.

[0088] Exemplarily, the text-guided query vector construction module mainly includes a word segmenter and a masker. The process of constructing a text-guided query vector may include: segmenting all the texts of each sample image through the subword segmenter of the commonly used language model. Specifically, for each sample image, first obtain the word segmentation index sequence corresponding to each text in the sample image, and use the [SEP] index to connect different texts so that each image corresponds to an index sequence. When training the text recognition model, padding is performed according to the longest sequence length T in the same batch of images to ensure that the sequence length of a batch is the same, and the [PAD] index is used as the padding flag. A random masking operation is performed on the index sequence corresponding to each text in the sequence, specifically randomly masking each text index sequence. The word index of is replaced by the [MASK] index. To ensure the semantic information of the text, the value of a should not be too large. The complete sequence after masking is mapped to the corresponding text embedding feature according to the index, and the text guided query vector is obtained. .

[0089] Exemplarily, the preset text can represent the real character information in the sample image. The preset text passes through the word segmenter and the masker in sequence to obtain a text-guided query vector. The output obtained by the preset text passing through the word segmenter is a padded word segmentation index sequence QT'. The result obtained by performing a random masking operation on the padded word segmentation index sequence is used as the text-guided query vector.

[0090] See also Figure 8f , Figure 8f It is a schematic diagram of a character-guided query vector construction module in an embodiment of the text recognition method of the present application.

[0091] like Figure 8fAs shown, the starting sequence includes a first sequence and a second sequence, wherein the starting sequence can be a text prediction result obtained by passing the features representing the sample position query vector and the text guide query vector through the target linear layer for the first output of the multi-layer decoder. The starting sequence passes through a masker to obtain an intermediate sequence. The intermediate sequence includes a padded first sequence and a new second sequence. The starting sequence passes through a masker to padded the first sequence to obtain a padded first sequence. The padded first sequence includes the first sequence and a preset padding. The preset padding can be the [PAD] index corresponding to the above padding flag. The starting sequence passes through a masker to perform a random masking operation on the first sequence to obtain a new second sequence. The intermediate sequence is mapped through a word segmenter to obtain a final sequence, and the final sequence is used as the above character guide query vector. The final sequence includes the processed first sequence and the processed second sequence. Exemplarily, the first sequence can represent "MALIBU". The second sequence can represent "27 MILFS OF SCENIC PEAUTY". The padded first sequence can be represented as "MALIBU+preset padding", that is, "MALIBU[PAD]". The new second sequence may be represented as "27 MIL[UNK]SOF SCENIC [UNK]EAUTY".

[0092] The above scheme uses a text recognition model to obtain at least one initial image feature, and then performs position encoding based on each initial image feature to obtain a position query vector corresponding to each initial image feature, wherein each position query vector and each initial image feature can combine feature information of different scales and different positions, thereby improving the accuracy of determining the text recognition result of the image to be recognized.

[0093] See also Fig. 9 , Fig. 9 It is a structural diagram of an embodiment of the text recognition device of the present application. The text recognition device 90 includes an acquisition module 91, an encoding module 92 and a determination module 93; the acquisition module 91 is used to acquire at least one initial image feature using a text recognition model, and each initial image feature is obtained by extracting features from the image to be recognized; the encoding module 92 is used to perform position encoding on each initial image feature to obtain a position query vector corresponding to each initial image feature; the determination module 93 is used to determine the text recognition result of the image to be recognized based on each initial image feature and each position query vector, and the text recognition result includes at least one of the following: the type of each area in the image to be recognized, the coordinate information of each area in the image to be recognized, and the character information of each area in the image to be recognized.

[0094] The above scheme uses a text recognition model to obtain at least one initial image feature, and then performs position encoding based on each initial image feature to obtain a position query vector corresponding to each initial image feature, wherein each position query vector and each initial image feature can combine feature information of different scales and different positions, thereby improving the accuracy of determining the text recognition result of the image to be recognized.

[0095] Please refer to the text recognition method for the functions performed by each module, which will not be repeated here.

[0096] See also Fig.10 , Fig.10 1 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 100 includes a memory 101 and a processor 102, and the processor 102 is used to execute program instructions stored in the memory 101 to implement the steps in the above text recognition method embodiment. In a specific implementation scenario, the electronic device 100 may include but is not limited to: a multi-camera device, a microcomputer, and a server. In addition, the electronic device 100 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0097] Specifically, the processor 102 is used to control itself and the memory 101 to implement the steps in the above-mentioned text recognition method embodiment. The processor 102 can also be called a CPU (Central Processing Unit). The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 102 can be implemented by an integrated circuit chip.

[0098] The above scheme uses a text recognition model to obtain at least one initial image feature, and then performs position encoding based on each initial image feature to obtain a position query vector corresponding to each initial image feature, wherein each position query vector and each initial image feature can combine feature information of different scales and different positions, thereby improving the accuracy of determining the text recognition result of the image to be recognized.

[0099] See also Fig.11 , Fig.11The computer-readable storage medium 110 stores program instructions 1101 thereon, and when the program instructions 1101 are executed by a processor, the steps in any of the above-mentioned text recognition method embodiments are implemented.

[0100] The above scheme uses a text recognition model to obtain at least one initial image feature, and then performs position encoding based on each initial image feature to obtain a position query vector corresponding to each initial image feature, wherein each position query vector and each initial image feature can combine feature information of different scales and different positions, thereby improving the accuracy of determining the text recognition result of the image to be recognized.

[0101] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0102] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0103] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0104] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0105] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A text recognition method, characterized in that: The method comprises: Acquire at least one initial image feature using a text recognition model, wherein each of the initial image features is obtained by extracting features from an image to be recognized; Performing position encoding on each of the initial image features to obtain a position query vector corresponding to each of the initial image features; Based on each of the initial image features and each of the position query vectors, a text recognition result of the image to be recognized is determined, and the text recognition result includes at least one of the following: the type of each area in the image to be recognized, the coordinate information of each area in the image to be recognized, and the character information of each area in the image to be recognized.

2. The method according to claim 1, characterized in that The determining of the text recognition result of the image to be recognized based on each of the initial image features and each of the position query vectors includes: Inputting each of the initial image features and each of the position query vectors into a multi-layer decoder for feature fusion to obtain a target fusion feature corresponding to each of the initial image features; Based on each of the target fusion features, a text recognition result of the image to be recognized is determined.

3. The method according to claim 2, characterized in that The multi-layer decoder includes a plurality of preset decoding layers arranged in cascade, each of the preset decoding layers has the same structure but different parameters, the features output by the last preset decoding layer are used as the features output by the multi-layer decoder, the features output by the multi-layer decoder are the target fusion features corresponding to each of the initial image features, and each of the preset decoding layers includes an intra-group attention module, Inputting each of the initial image features and each of the position query vectors into a multi-layer decoder for feature fusion to obtain target fusion features corresponding to each of the initial image features includes: Using the first query key-value pair as an input of an intra-group attention module in a current preset decoding layer, obtaining a first feature output by the intra-group attention module in the current preset decoding layer, wherein the query data, key data, and value data in the first query key-value pair are respectively obtained from each of the position query vectors or from features output by a previous preset decoding layer, and the current preset decoding layer is one of the plurality of preset decoding layers; Fusing the first feature output by the intra-group attention module in the current preset decoding layer with the input of the intra-group attention module in the current preset decoding layer to obtain a first fused feature; The first fusion feature is subjected to advanced fusion processing with each of the initial image features to obtain the feature output by the current preset decoding layer.

4. The method according to claim 3, characterized in that Each of the preset decoding layers includes a cross attention module, and the first fusion feature and each of the initial image features are advancedly fused to obtain the features output by the current preset decoding layer, including: Using a second query key-value pair as an input of a cross-attention module in the current preset decoding layer, obtaining a second feature output by the cross-attention module in the current preset decoding layer, wherein query data in the second query key-value pair is obtained by the first fused feature, and key data and value data in the second query key-value pair are respectively obtained by each of the initial image features; Fusing the second feature output by the cross attention module in the current preset decoding layer with the first fused feature to obtain a second fused feature; Based on the second fusion feature, a feature output by the current preset decoding layer is obtained.

5. The method according to claim 2, characterized in that: The determining of the text recognition result of the image to be recognized based on each of the target fusion features includes: Based on each of the target fusion features, determine the confidence ranking corresponding to each of the target fusion features, and use the target fusion feature with the largest confidence ranking as the target feature; The target features are respectively input into the first linear layer, the second linear layer and the third linear layer for mapping processing to obtain the types of each area in the image to be recognized output by the first linear layer, the coordinate information of each area in the image to be recognized output by the second linear layer, and the character information of each area in the image to be recognized output by the third linear layer.

6. The method according to any one of claims 1 to 5, characterized in that The performing position encoding on each of the initial image features to obtain a position query vector corresponding to each of the initial image features includes: For each of the initial image features, perform the following steps: Inputting the initial image feature into a first perceptron module to obtain sampling point coordinate information of the initial image feature output by the first perceptron module; The sampling point coordinate information processed by the preset encoding is used as the input of the second perceptron module to obtain the position query vector corresponding to the initial image feature output by the second perceptron module.

7. The method according to any one of claims 1 to 5, characterized in that The step of obtaining at least one initial image feature by using a text recognition model includes: Acquire the image to be recognized by using the text recognition model; Performing multi-scale feature extraction on the image to be identified to obtain a multi-scale feature set corresponding to the image to be identified, wherein the multi-scale feature set includes at least two scale features and the scales of the scale features are different; The scale features in the multi-scale feature set are subjected to feature fusion to obtain the at least one initial image feature.

8. The method according to any one of claims 1 to 5, characterized in that The method further includes a step of training the text recognition model, wherein the training step includes: Acquire a sample image and a sample recognition result corresponding to the sample image, wherein the sample recognition result is obtained through pre-annotation, and the sample recognition result includes real character information and a text-guided query vector related to the real character information; Based on the sample image and the text-guided query vector, obtaining a character-guided query vector corresponding to the sample image; Using the sample image, the text-guided query vector, and the character-guided query vector as inputs of the text recognition model to obtain a predicted recognition result corresponding to the sample image; Obtaining a target loss based on a difference between the sample recognition result and the predicted recognition result; The target loss is used to adjust the parameters in the text recognition model to obtain a trained text recognition model.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 8.

Images