Text processing method, apparatus, medium, and device

Abstract

The present disclosure relates to a text processing method, device, medium and equipment, the method comprising: receiving a text image to be processed; inputting the text image into a text processing model to obtain a detection result and a recognition result corresponding to the text image; wherein the text processing model comprises a feature extraction layer, an encoding layer, a decoding layer and a prediction layer, the feature extraction layer is used to perform feature extraction on the text image to obtain image features corresponding to the text image; the encoding layer is used to encode the image features to obtain detection features and recognition features corresponding to the text image, and the detection features and the recognition features are spliced to obtain image splicing features; the decoding layer is used to perform decoding processing on the image splicing features to obtain processing features corresponding to the text image; and the prediction layer is used to perform prediction based on the processing features to obtain the detection result and the recognition result corresponding to the text image.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a text processing method, apparatus, medium, and device. Background Technology

[0002] Currently, text recognition tasks are typically divided into two stages. The first stage uses an image segmentation-based detection model to identify the text-containing parts of the image. The second stage uses an image recognition model to identify the text content, such as extracting the text-containing parts of the image for text content recognition. This process is not only computationally intensive, but also suffers from insufficient interaction between the detection and recognition models. The multi-stage recognition approach makes the results of each stage overly dependent on the results of the previous stage, leading to insufficient stability of the final result. Summary of the Invention

[0003] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, this disclosure provides a text processing method, the method comprising:

[0005] Receive the text image to be processed;

[0006] The text image is input into a text processing model to obtain the detection and recognition results corresponding to the text image;

[0007] The text processing model includes a feature extraction layer, an encoding layer, a decoding layer, and a prediction layer. The feature extraction layer extracts features from the text image to obtain corresponding image features. The encoding layer encodes these image features to obtain detection and recognition features, which are then concatenated to obtain image stitching features. The decoding layer decodes these image stitching features to obtain processing features corresponding to the text image. Finally, the prediction layer performs predictions based on these processing features to obtain detection and recognition results corresponding to the text image.

[0008] Secondly, this disclosure provides a text processing apparatus, the apparatus comprising:

[0009] The receiving module is used to receive the text image to be processed;

[0010] The processing module is used to input the text image into the text processing model to obtain the detection result and recognition result corresponding to the text image;

[0011] The text processing model includes a feature extraction layer, an encoding layer, a decoding layer, and a prediction layer. The processing module includes: a first extraction submodule, used to extract features from the text image through the feature extraction layer to obtain image features corresponding to the text image; a first encoding submodule, used to encode the image features through the encoding layer to obtain detection features and recognition features corresponding to the text image, and to concatenate the detection features and recognition features to obtain image concatenation features; a decoding submodule, used to decode the image concatenation features through the decoding layer to obtain processing features corresponding to the text image; and a first processing submodule, used to predict based on the processing features through the prediction layer to obtain detection results and recognition results corresponding to the text image.

[0012] Thirdly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect.

[0013] Fourthly, this disclosure provides an electronic device, comprising:

[0014] A storage device on which computer programs are stored;

[0015] A processing device for executing the computer program in the storage device to implement the steps of the method described in the first aspect.

[0016] The above technical solution utilizes an end-to-end text processing model to process text images, simultaneously obtaining detection and recognition results, thereby improving text processing efficiency to some extent. Furthermore, in this solution, a shared decoder is used for processing detection and recognition features within the text processing model. This enables multimodal interaction between detection and recognition features during decoding, facilitating the simultaneous combination of both features for text processing. This not only improves processing efficiency but also enhances the accuracy of the detection and recognition results for the text images.

[0017] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0018] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings:

[0019] Figure 1 This is a flowchart of a text processing method provided according to one embodiment of the present disclosure;

[0020] Figure 2 This is a schematic diagram of the structure of the decoding layer in a text processing model provided according to one embodiment of the present disclosure;

[0021] Figure 3 This is a block diagram of a text processing apparatus provided according to one embodiment of the present disclosure;

[0022] Figure 4 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0023] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0024] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0025] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0026] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0027] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0028] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0029] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0030] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0031] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0032] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0033] Meanwhile, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0034] Figure 1 The diagram shown is a flowchart of a text processing method provided according to an embodiment of this disclosure. Figure 1 As shown, the method includes:

[0035] In step 11, the text image to be processed is received. The text image to be processed can be an image taken by the user to identify the text content, or an image exported or downloaded from a webpage; there is no limitation on this.

[0036] In step 12, the text image is input into the text processing model to obtain the detection and recognition results corresponding to the text image. The text processing model includes a feature extraction layer, an encoding layer, a decoding layer, and a prediction layer.

[0037] The text image is subjected to feature extraction by the feature extraction layer to obtain image features corresponding to the text image; the image features are encoded by the encoding layer to obtain detection features and recognition features corresponding to the text image, and the detection features and recognition features are concatenated to obtain image concatenation features; the image concatenation features are decoded by the decoding layer to obtain processing features corresponding to the text image; and the detection result and recognition result corresponding to the text image are obtained by prediction based on the processing features by the prediction layer.

[0038] Therefore, the above technical solution can process text images through an end-to-end text processing model to simultaneously obtain the detection and recognition results corresponding to the text image, thereby improving the efficiency of text processing to a certain extent. Furthermore, in the above technical solution, the processing of detection and recognition features in the text processing model can use a shared decoder. This enables multimodal interaction of detection and recognition features during the decoding process, facilitating the simultaneous combination of detection and recognition features for text processing. This not only improves the efficiency of text processing but also enhances the accuracy of the detection and recognition results corresponding to the text image.

[0039] In one possible embodiment, the step of extracting features from the text image through the feature extraction layer to obtain the image features corresponding to the text image may include:

[0040] The text image is used to extract features based on the multi-scale feature extraction module in the feature extraction layer to obtain multi-scale features.

[0041] As an example, text images can be preprocessed to resize them to a fixed aspect ratio, ensuring compatibility and adaptation for input images of various sizes and improving the applicability of text processing methods. Features can then be extracted based on the resized text image.

[0042] For example, this multi-scale feature extraction module can be implemented using a ResNet50 network. When performing multi-scale feature extraction, the common extraction methods of ResNet50 networks in this field can be adopted, which will not be elaborated further here. Through multi-scale feature extraction, the obtained multi-scale features can represent the text image at multiple scales, improving the expressive power of multi-scale features and providing a more comprehensive feature representation for subsequent text processing.

[0043] Positional encoding and hierarchical encoding are added to the multi-scale features, and the features after adding the positional encoding and hierarchical encoding are transformed into a one-dimensional feature representation to obtain the image features.

[0044] The positional encoding (Position Embedding) can be implemented based on Positional Encoding in the transformer architecture. Its construction can be based on any positional encoding method within the transformer architecture, such as integer value labeling, binary vector labeling, or periodic functions; this disclosure does not limit this approach. Furthermore, in this disclosure, feature extraction of text images yields multi-scale features. Feature points in different feature layers may have the same (w, h) coordinates. If positional encoding alone is insufficient to accurately represent the positional information of elements on the multi-scale feature map, then in this embodiment, a scale-level embedding can be further added. This scale-level embedding distinguishes different feature layers. All feature points in the same feature layer correspond to the same scale-level embedding, while feature points in different feature layers correspond to different scale-level embeddings. For example, this scale-level embedding can be randomly initialized and trained along with the network, making it learnable.

[0045] Furthermore, since the feature encoding at the encoding layer is based on one-dimensional features, the features after adding positional encoding and hierarchical encoding can be further transformed into a one-dimensional feature representation. This can be achieved by decomposing the two-dimensional feature map in the transformer architecture into a transformation method that the transformer can accept for one-dimensional sequence features. For example, the two-dimensional feature map corresponding to the text image can be transformed into a one-dimensional feature representation through patch embedding to obtain the image features for subsequent encoding.

[0046] Therefore, in the above technical solution, by performing multi-scale feature extraction on the text image, more comprehensive features corresponding to the text image can be obtained. Furthermore, by adding position encoding to the extracted features, the position sensitivity of the text processing is guaranteed in the subsequent text processing, ensuring the accuracy of the subsequent detection results. Moreover, by adding hierarchical encoding to distinguish different feature layers under multi-scale features, the comprehensiveness and accuracy of the image features are further improved, providing reliable data support for subsequent text processing.

[0047] In one possible embodiment, encoding the image features through the encoding layer to obtain the detection and recognition features corresponding to the text image may include:

[0048] Based on the image features, feature encoding is performed using a transformer encoding layer to obtain the encoded features corresponding to the text image.

[0049] The encoding layer can be implemented using a transformer encoder, which can contain a 6-layer structure, with each layer connected in series. Each layer can consist of a deformable attention module and a feed-forward network (FFN). During the computation of deformable attention, each feature in the image features can be treated as a query to determine its correlation with every other feature in the image. The top N features with the highest correlation, ranked from largest to smallest, are then used as the relevant features for that query, thus modeling the relationships between features.

[0050] Next, the weights corresponding to the relevant features are determined based on their relevance. For example, the weights of the relevant features can be determined according to their respective relevance ratios, with the sum of the weights for each relevant feature being 1. This allows for the allocation of attention weights based on the relevant features. During attention calculation, attention can be performed by modeling the corresponding weights, ensuring that each feature prioritizes features that are relatively relevant to it during feature encoding. This reduces the amount of data computation while effectively ensuring the accuracy and effectiveness of the encoded features. Furthermore, the obtained feature map is output to the next layer through an FFN, and the features output from the last layer are used as the encoded features.

[0051] Based on the fully connected coding layer, each feature in the coding features is predicted to obtain the predicted position information corresponding to each feature in the coding features, and the target position information is determined from the predicted position information.

[0052] For example, this fully connected encoding layer can be implemented based on a general fully connected layer structure, which will not be elaborated here. Further, for each feature in the encoded features, the predicted location information corresponding to that feature and the confidence level corresponding to that predicted location information can be predicted. For example, a predicted bounding box and its corresponding confidence level can be generated for each feature. Then, the first m predicted location information can be selected as the target location information in descending order of confidence level. Here, m can be set according to the actual application scenario, such as m being preset to 100. The predicted location information can be represented by an 8*25 rectangle.

[0053] For each target location information, the identification feature corresponding to the target location information is determined according to the location indicated by the target location information, and the feature used to predict the target location information in the encoded features is determined as the detection feature.

[0054] For each target location, the recognition features can be determined based on the location indicated by the target location information, such as the 8*25 rectangle mentioned above, and the features within the rectangle. Specifically, the features corresponding to the center line of the rectangle can be used as the recognition features corresponding to the target location information; that is, the 1*25 feature in the middle is selected as the recognition feature corresponding to the target location information. As mentioned above, the target location information is predicted based on a certain feature in the encoded features. Therefore, the feature used to predict the target location information in the encoded features can be determined as the detection feature, and the detection feature and recognition feature corresponding to the target location information are concatenated. For example, for each target location information, a 1*26 feature can be obtained.

[0055] Therefore, through the above technical solution, accurate encoded features can be obtained by encoding image features, and preliminary predictions of detection and recognition features can be made to ensure their accuracy. In this technical solution, for each target location information, the corresponding detection and recognition features are determined in the encoded features. This improves the accuracy of detection and recognition features and provides reliable data support for subsequent multimodal interactive decoding based on detection and recognition features.

[0056] In one possible embodiment, such as Figure 2 As shown, the decoding layer is as follows Figure 2 As shown in A, the process of decoding the image stitching features through the decoding layer to obtain the processing features corresponding to the text image may include:

[0057] Multimodal attention processing is performed based on the image stitching features to obtain the first attention feature.

[0058] like Figure 2 S can represent the image stitching feature, where D represents the detection feature and R represents the recognition feature. As an example, the multimodal attention processing can be calculated using the multi-modal attention common in the art to obtain the first attention feature.

[0059] As another example, the step of performing multimodal attention processing based on the image stitching features to obtain the first attention features may include:

[0060] The recognition features are subjected to text classification processing, and the features of the last feature layer in the text classification process are used as the text features of the recognition features.

[0061] For example, for recognition features, the recognition features can be classified and normalized to various character classes to obtain the character probability matrix corresponding to the recognition features. This matrix serves as the text feature. Based on the text feature, the initial recognition result corresponding to the recognition feature can be obtained, which can be determined in the following way:

[0062] P = softmax(W1R)

[0063] Where P represents the text feature. The weights for training are R, which represents the recognition feature, C, which represents the feature dimension, and U, which represents the character class number.

[0064] The detection features and the text features are concatenated to obtain the text concatenation features.

[0065] For example, the text features mapped to language features can be concatenated with their corresponding detection features in the following way:

[0066] L = cat(D, W2P)

[0067] Where L represents the text concatenation feature, D represents the detection feature, and cat represents the concatenation operation to join the two features into a single vector. These are the weights used for training.

[0068] Furthermore, multimodal attention processing can be performed using image stitching features and text stitching features. In general attention processing mechanisms, the query vector, key vector, and value vector are determined based on the same input features. However, in this embodiment, multimodal attention processing can be performed based on the image stitching features and the text stitching features to obtain the first attention feature. For example, the query vector can be determined based on the image stitching features, and the key vector and value vector can be generated based on the text stitching features. Multimodal attention processing can then be performed based on the query vector, the key vector, and the value vector to obtain the first attention feature.

[0069] As an example, the query vector Q, key vector K, and value vector V can be determined by transformation using methods commonly used in this field, and determined based on a general attention calculation method to obtain the first attention feature, which will not be elaborated further here.

[0070] As another example, an attention mask matrix can be added when determining the multimodal attention weight matrix during multimodal attention processing. For instance, the first attention feature F can be determined in the following way:

[0071]

[0072] Where M represents the attention mask matrix, PE() is used to represent the positional encoding in the DETR model, S is used to represent the image stitching feature, and the attention mask matrix can be used to query the vector that is overly focused on itself when performing multimodal attention weight calculation, thereby improving the accuracy of the first attention feature. At the same time, the attention mask matrix method can enable the process to decode in parallel, further improving the decoding efficiency.

[0073] Therefore, through the above technical solution, in the multimodal attention processing process, not only can the detection features and recognition features interact simultaneously for attention processing, but the accuracy of the first attention feature can also be ensured by combining the detection features and recognition features. Furthermore, the recognition features in the input image stitching features are image features. In this embodiment, by converting them into language features, effective and reliable data support can be further provided for subsequent text recognition. Feature processing can be performed from both image and language perspectives, improving the accuracy of feature processing.

[0074] Then, the first attention feature is subjected to factorized attention processing to obtain the second attention feature.

[0075] In this embodiment, the first attention feature can be mapped using an FFN network, and factorized attention processing can be performed based on the mapped features. The factorized attention processing can employ a factorized self-attention algorithm, which can perform self-attention calculations within the same feature (modeling the relationships within text lines corresponding to the same feature) and self-attention calculations between different features (modeling the relationships between different features, i.e., different texts). This further improves the representation of the text content's association features in the second attention feature.

[0076] The second attention feature is subjected to cross-attention processing to obtain the processed feature.

[0077] The cross-attention processing can be calculated based on deformable cross attention in this field, and its specific calculation method will not be elaborated here. Therefore, feature extraction can be further performed on the image features of the text image, and the extracted features can be mapped and output through FFN as the processing features, such as... Figure 2 As shown in C.

[0078] Therefore, through the above technical solution, detection features and recognition features can be synchronously decoded by sharing the same decoding layer. In this process, multimodal attention processing effectively increases the interaction between detection features and recognition features, thereby improving the accuracy of the processing features in representing detection and recognition information to a certain extent. Furthermore, through factorized attention processing and cross-attention processing, the correlation between features and features can be further modeled, while improving the comprehensiveness and richness of features in the processing features, thus providing effective data support for the accurate identification of detection and recognition results.

[0079] In one possible embodiment, the step of obtaining the detection and recognition results corresponding to the text image by predicting based on the processed features through the prediction layer may include:

[0080] The processed features are classified based on the first fully connected layer in the prediction layer to determine the target classification corresponding to the processed features;

[0081] Based on the second fully connected layer in the prediction layer, position regression is performed on the processed features to determine the position information corresponding to the processed features;

[0082] Based on the third fully connected layer in the prediction layer, text recognition is performed on the processed features to determine the text information corresponding to the processed features;

[0083] The detection result and recognition result are determined based on the target classification corresponding to the processing feature, the location information, and the text information.

[0084] The first fully connected layer, the second fully connected layer, and the third fully connected layer in the prediction layer can all be implemented based on the structure of a fully connected layer commonly used in this field. During the training of the text processing model, their corresponding parameters are adjusted and optimized so that in the trained text processing model, the output of the first fully connected layer is directly used as the target classification, the output of the second fully connected layer is used as the location information, and the output of the third fully connected layer is used as the text information.

[0085] As an example, determining the detection result and recognition result based on the target classification corresponding to the processing features, the location information, and the text information may include:

[0086] For each processing feature, if the target classification corresponding to the processing feature is a foreground classification, then the location information corresponding to the processing feature is used as the detection result, and the text information corresponding to the processing feature is determined as the recognition result.

[0087] As described above, in the process of processing text images based on the text processing model, when determining the detection features, the first m predicted position information can be selected as the target position information, that is, m detection results and recognition results can be determined in this process. Further, in this embodiment, the processed features can be classified using a first fully connected layer. The classification categories can include foreground classification and background classification, where foreground classification indicates that the processed feature corresponds to a real text feature, and background classification indicates that the processed feature corresponds to a noise feature. Accordingly, if the classification corresponding to the processed feature is determined to be foreground classification, that is, if the detection result and recognition result corresponding to the processed feature are the corresponding real text results, then the position information corresponding to the processed feature can be used as the detection result, and the text information corresponding to the processed feature can be determined as the recognition result, so as to simultaneously achieve the detection and recognition of the text image.

[0088] As an example, the text boundaries in a text image may be irregular. In this embodiment, the positional information can be represented by multiple location points, such as eight points. The area corresponding to the positional information of these eight points can be used as the text area in the text image to further improve the precision and accuracy of the text processing method.

[0089] This disclosure also provides a method for training a text processing model, which may include:

[0090] Obtain a training sample set, wherein each sample in the training sample set contains a training text image and the corresponding annotation detection information and annotation recognition information of the training text image;

[0091] The training text image is used to extract features from the training text image by using the feature extraction layer in the preset model to obtain the training image features corresponding to the training text image.

[0092] The training image features are encoded through the encoding layer in the preset model to obtain the training detection features and training recognition features corresponding to the training text image, and the training detection features and training recognition features are concatenated to obtain the training image concatenated features.

[0093] The training image splicing features are decoded by the decoding layer in the preset model to obtain the training processed features corresponding to the training text image.

[0094] The training detection result and training recognition result corresponding to the training text image are obtained by predicting based on the training processing features through the prediction layer in the preset model.

[0095] Based on the annotation detection information and annotation recognition information corresponding to the training text image, as well as the training detection results and training recognition results corresponding to the training text image, the target loss of the preset model is determined, and the preset model is trained based on the target loss. The trained preset model is then determined as the text processing model.

[0096] Therefore, during the training process of the text processing model, training text images can be input into the text processing model. Based on a process similar to that described above, training detection results and training recognition results can be determined. Then, based on the labeled detection information and labeled recognition information corresponding to the training text images, as well as the training detection results and training recognition results corresponding to the training text images, the target loss of the preset model can be determined, which may include:

[0097] For example, the order of the model output training detection results may not be the same as the sequence of ground truth. In this case, binary matching can be performed based on the bounding boxes in the annotation detection information corresponding to the training text image and the predicted boxes in the training detection results to determine the bounding boxes corresponding to each predicted box. This can be achieved based on bipartite matching.

[0098] After determining the labeled bounding box corresponding to each predicted bounding box, the detection loss and recognition loss can be calculated based on the matched predicted bounding boxes and labeled bounding boxes. For example, the detection loss can include GIoU loss, L1 loss, and classification loss, and the recognition loss can include classification loss. The above losses can be calculated using common loss calculation methods in the art. Furthermore, the detection loss in the detection direction and the recognition loss in the recognition direction can be weighted and summed to determine the target loss of the preset model. The weights corresponding to each weight can be set according to the actual application scenario, and this disclosure does not limit this.

[0099] Once the target loss is determined, the parameters in the feature extraction layer, encoding layer, decoding layer, and prediction layer of the preset model can be optimized and updated based on the target loss. For example, gradient descent can be used to optimize the parameters until the target loss of the preset model is less than a preset threshold, or the preset model has been trained a preset number of times, so as to complete the training of the preset model.

[0100] Therefore, the above technical solution can train an end-to-end text processing model, enabling the simultaneous acquisition of detection and recognition results for text images during processing, thus improving text processing efficiency. Furthermore, by using a shared decoder for processing detection and recognition features within the text processing model, multimodal interaction between detection and recognition features can be achieved during decoding. This allows for the simultaneous combination of detection and recognition features for text processing, improving the training efficiency of the text processing model and reducing the computational data volume, thereby enhancing the efficiency and accuracy of text processing based on this model.

[0101] This disclosure also provides a text processing apparatus, such as Figure 3 As shown, the device 30 includes:

[0102] Receiver module 31 is used to receive the text image to be processed;

[0103] Processing module 32 is used to input the text image into a text processing model to obtain the detection result and recognition result corresponding to the text image;

[0104] The text processing model includes a feature extraction layer, an encoding layer, a decoding layer, and a prediction layer. The processing module includes: a first extraction submodule, used to extract features from the text image through the feature extraction layer to obtain image features corresponding to the text image; a first encoding submodule, used to encode the image features through the encoding layer to obtain detection features and recognition features corresponding to the text image, and to concatenate the detection features and recognition features to obtain image concatenation features; a decoding submodule, used to decode the image concatenation features through the decoding layer to obtain processing features corresponding to the text image; and a first processing submodule, used to predict based on the processing features through the prediction layer to obtain detection results and recognition results corresponding to the text image.

[0105] Optionally, the decoding submodule includes:

[0106] The second processing submodule is used to perform multimodal attention processing based on the image stitching features to obtain the first attention features;

[0107] The third processing submodule is used to perform factorized attention processing on the first attention feature to obtain the second attention feature;

[0108] The fourth processing submodule is used to perform cross-attention processing on the second attention feature to obtain the processed feature.

[0109] Optionally, the second processing submodule includes:

[0110] The first determining submodule is used to perform text classification processing on the recognition features, and to take the features of the last feature layer in the text classification process as the text features of the recognition features;

[0111] The splicing submodule is used to splice the detected features and the text features to obtain text splicing features;

[0112] The fifth processing submodule is used to determine a query vector based on the image splicing features, generate a key vector and a value vector based on the text splicing features, and perform the multimodal attention processing based on the query vector, the key vector and the value vector to obtain the first attention feature.

[0113] Optionally, the first extraction submodule includes:

[0114] The second extraction submodule is used to extract features from the text image based on the multi-scale feature extraction module in the feature extraction layer to obtain multi-scale features.

[0115] A submodule is added to add positional and hierarchical encoding to the multi-scale features, and the features after adding the positional and hierarchical encoding are converted to a one-dimensional feature representation to obtain the image features.

[0116] Optionally, the first encoding submodule includes:

[0117] The second encoding submodule is used to perform feature encoding based on the image features using a transformer encoding layer to obtain the encoded features corresponding to the text image;

[0118] The first prediction submodule is used to predict each feature in the encoded features based on the fully connected coding layer, obtain the prediction position information corresponding to each feature in the encoded features, and determine the target position information from the prediction position information;

[0119] The second determining submodule is used to determine the identification features corresponding to each target location information according to the location indicated by the target location information, and to determine the features used to predict the target location information in the encoded features as the detection features.

[0120] Optionally, the first processing submodule includes:

[0121] The classification submodule is used to classify the processed features based on the first fully connected layer in the prediction layer, and determine the target classification corresponding to the processed features;

[0122] The regression submodule is used to perform position regression on the processed features based on the second fully connected layer in the prediction layer to determine the position information corresponding to the processed features.

[0123] The recognition submodule is used to perform text recognition on the processed features based on the third fully connected layer in the prediction layer, and determine the text information corresponding to the processed features;

[0124] The third determining submodule is used to determine the detection result and the recognition result based on the target classification corresponding to the processing feature, the location information, and the text information.

[0125] Optionally, the third determining submodule is further configured to:

[0126] For each processing feature, if the target classification corresponding to the processing feature is a foreground classification, then the location information corresponding to the processing feature is used as the detection result, and the text information corresponding to the processing feature is determined as the recognition result.

[0127] This disclosure also provides a training apparatus for a text processing model, the apparatus comprising:

[0128] The acquisition module is used to acquire a training sample set, wherein each sample in the training sample set contains a training text image and the corresponding annotation detection information and annotation recognition information of the training text image;

[0129] The extraction module is used to extract features from the training text image through the feature extraction layer in the preset model to obtain the training image features corresponding to the training text image;

[0130] The encoding module is used to encode the training image features through the encoding layer in the preset model to obtain the training detection features and training recognition features corresponding to the training text image, and to concatenate the training detection features and training recognition features to obtain the training image concatenation features.

[0131] The decoding module is used to decode the splicing features of the training image through the decoding layer in the preset model to obtain the training processed features corresponding to the training text image;

[0132] The prediction module is used to make predictions based on the training processing features through the prediction layer in the preset model, and obtain the training detection result and training recognition result corresponding to the training text image;

[0133] The training module is used to determine the target loss of the preset model based on the annotation detection information and annotation recognition information corresponding to the training text image, as well as the training detection result and training recognition result corresponding to the training text image, and to train the preset model based on the target loss, and to determine the trained preset model as the text processing model.

[0134] The following is for reference. Figure 4 This diagram illustrates a structural schematic of an electronic device 600 suitable for implementing embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0135] like Figure 4 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0136] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0137] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0138] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0139] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0140] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0141] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: receive a text image to be processed; input the text image into a text processing model to obtain detection and recognition results corresponding to the text image; wherein the text processing model includes a feature extraction layer, an encoding layer, a decoding layer, and a prediction layer; the feature extraction layer extracts features from the text image to obtain image features corresponding to the text image; the encoding layer encodes the image features to obtain detection and recognition features corresponding to the text image, and concatenates the detection and recognition features to obtain image stitching features; the decoding layer decodes the image stitching features to obtain processing features corresponding to the text image; and the prediction layer makes predictions based on the processing features to obtain detection and recognition results corresponding to the text image.

[0142] Alternatively, the aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire a training sample set, wherein each sample in the training sample set contains a training text image and corresponding annotation detection information and annotation recognition information; extract features from the training text image through a feature extraction layer in a preset model to obtain training image features corresponding to the training text image; encode the training image features through an encoding layer in the preset model to obtain training detection features and training recognition features corresponding to the training text image, and further process the training detection features and training recognition features... Line concatenation is performed to obtain training image concatenation features; the training image concatenation features are decoded by the decoding layer in the preset model to obtain training processing features corresponding to the training text image; the prediction layer in the preset model makes predictions based on the training processing features to obtain training detection results and training recognition results corresponding to the training text image; based on the annotation detection information and annotation recognition information corresponding to the training text image, as well as the training detection results and training recognition results corresponding to the training text image, the target loss of the preset model is determined, and the preset model is trained based on the target loss, and the trained preset model is determined as the text processing model.

[0143] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0144] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0145] The modules described in the embodiments of this disclosure can be implemented in software or in hardware. The names of the modules are not necessarily limiting in certain circumstances; for example, a receiving module can also be described as "a module for receiving text images to be processed".

[0146] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0147] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0148] According to one or more embodiments of this disclosure, Example 1 provides a text processing method, wherein the method includes:

[0149] Receive the text image to be processed;

[0150] The text image is input into a text processing model to obtain the detection and recognition results corresponding to the text image;

[0151] The text processing model includes a feature extraction layer, an encoding layer, a decoding layer, and a prediction layer. The feature extraction layer extracts features from the text image to obtain corresponding image features. The encoding layer encodes these image features to obtain detection and recognition features, which are then concatenated to obtain image stitching features. The decoding layer decodes these image stitching features to obtain processing features corresponding to the text image. Finally, the prediction layer performs predictions based on these processing features to obtain detection and recognition results corresponding to the text image.

[0152] According to one or more embodiments of this disclosure, Example 2 provides the method of Example 1, wherein the step of decoding the image stitching features through the decoding layer to obtain the processed features corresponding to the text image includes:

[0153] Multimodal attention processing is performed based on the image stitching features to obtain the first attention feature;

[0154] The first attention feature is subjected to factorized attention processing to obtain the second attention feature;

[0155] The second attention feature is subjected to cross-attention processing to obtain the processed feature.

[0156] According to one or more embodiments of this disclosure, Example 3 provides the method of Example 2, wherein the step of performing multimodal attention processing based on the image stitching features to obtain first attention features includes:

[0157] The recognition features are subjected to text classification processing, and the features of the last feature layer in the text classification process are used as the text features of the recognition features.

[0158] The detection features and the text features are concatenated to obtain the text concatenation features;

[0159] The query vector is determined based on the image splicing features, and the key vector and value vector are generated based on the text splicing features. The multimodal attention processing is then performed based on the query vector, the key vector, and the value vector to obtain the first attention feature.

[0160] According to one or more embodiments of this disclosure, Example 4 provides the method of Example 1, wherein the step of extracting features from the text image through the feature extraction layer to obtain image features corresponding to the text image includes:

[0161] Based on the multi-scale feature extraction module in the feature extraction layer, feature extraction is performed on the text image to obtain multi-scale features;

[0162] Positional encoding and hierarchical encoding are added to the multi-scale features, and the features after adding the positional encoding and hierarchical encoding are transformed into a one-dimensional feature representation to obtain the image features.

[0163] According to one or more embodiments of this disclosure, Example 5 provides the method of Example 1, wherein encoding the image features through the encoding layer to obtain detection features and recognition features corresponding to the text image includes:

[0164] Based on the image features, feature encoding is performed using a transformer coding layer to obtain the encoded features corresponding to the text image;

[0165] Based on the fully connected coding layer, each feature in the coding features is predicted to obtain the predicted position information corresponding to each feature in the coding features, and the target position information is determined from the predicted position information;

[0166] For each target location information, the identification feature corresponding to the target location information is determined according to the location indicated by the target location information, and the feature used to predict the target location information in the encoded features is determined as the detection feature.

[0167] According to one or more embodiments of this disclosure, Example 6 provides the method of Example 1, wherein the step of obtaining the detection result and recognition result corresponding to the text image by predicting based on the processing features through the prediction layer includes:

[0168] The processed features are classified based on the first fully connected layer in the prediction layer to determine the target classification corresponding to the processed features;

[0169] Based on the second fully connected layer in the prediction layer, position regression is performed on the processed features to determine the position information corresponding to the processed features;

[0170] Based on the third fully connected layer in the prediction layer, text recognition is performed on the processed features to determine the text information corresponding to the processed features;

[0171] The detection result and recognition result are determined based on the target classification corresponding to the processing feature, the location information, and the text information.

[0172] According to one or more embodiments of this disclosure, Example 7 provides the method of Example 6, wherein determining the detection result and the recognition result based on the target classification corresponding to the processing feature, the location information, and the text information includes:

[0173] For each processing feature, if the target classification corresponding to the processing feature is a foreground classification, then the location information corresponding to the processing feature is used as the detection result, and the text information corresponding to the processing feature is determined as the recognition result.

[0174] According to one or more embodiments of this disclosure, Example 8 provides a text processing apparatus, the apparatus comprising:

[0175] The receiving module is used to receive the text image to be processed;

[0176] The processing module is used to input the text image into the text processing model to obtain the detection result and recognition result corresponding to the text image;

[0177] The text processing model includes a feature extraction layer, an encoding layer, a decoding layer, and a prediction layer. The processing module includes: a first extraction submodule, used to extract features from the text image through the feature extraction layer to obtain image features corresponding to the text image; a first encoding submodule, used to encode the image features through the encoding layer to obtain detection features and recognition features corresponding to the text image, and to concatenate the detection features and recognition features to obtain image concatenation features; a decoding submodule, used to decode the image concatenation features through the decoding layer to obtain processing features corresponding to the text image; and a first processing submodule, used to predict based on the processing features through the prediction layer to obtain detection results and recognition results corresponding to the text image.

[0178] According to one or more embodiments of the present disclosure, Example 9 provides a computer-readable medium having a computer program stored thereon that, when executed by a processing device, implements the steps of the method described in any one of Examples 1-7.

[0179] According to one or more embodiments of this disclosure, Example 10 provides an electronic device, including:

[0180] A storage device on which computer programs are stored;

[0181] A processing device for executing the computer program in the storage device to implement the steps of any one of the methods in Examples 1-7.

[0182] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0183] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0184] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. A text processing method, characterized in that, The method includes: Receive the text image to be processed; The text image is input into a text processing model to obtain the detection and recognition results corresponding to the text image; The text processing model includes a feature extraction layer, an encoding layer, a decoding layer, and a prediction layer. The feature extraction layer extracts features from the text image to obtain image features corresponding to the text image. The encoding layer encodes the image features to obtain detection and recognition features corresponding to the text image, and then concatenates the detection and recognition features to obtain image stitching features. The decoding layer decodes the image stitching features to obtain processed features corresponding to the text image. The prediction layer performs prediction based on the processed features to obtain the detection and recognition results corresponding to the text image. The step of decoding the image stitching features through the decoding layer to obtain the processing features corresponding to the text image includes: performing multimodal attention processing based on the image stitching features to obtain a first attention feature; performing factorized attention processing on the first attention feature to obtain a second attention feature; and performing cross-attention processing on the second attention feature to obtain the processing features. The step of performing multimodal attention processing based on the image stitching features to obtain a first attention feature includes: performing text classification processing on the recognition features, using the features of the last feature layer in the text classification process as the text features of the recognition features; stitching the detection features and the text features together to obtain text stitching features; determining a query vector based on the image stitching features, generating a key vector and a value vector based on the text stitching features, and performing the multimodal attention processing based on the query vector, the key vector, and the value vector to obtain the first attention feature.

2. The method according to claim 1, characterized in that, The step of extracting features from the text image through the feature extraction layer to obtain the image features corresponding to the text image includes: Based on the multi-scale feature extraction module in the feature extraction layer, feature extraction is performed on the text image to obtain multi-scale features; Positional encoding and hierarchical encoding are added to the multi-scale features, and the features after adding the positional encoding and hierarchical encoding are transformed into a one-dimensional feature representation to obtain the image features.

3. The method according to claim 1, characterized in that, The process of encoding the image features through the encoding layer to obtain the detection features and recognition features corresponding to the text image includes: Based on the image features, feature encoding is performed using a transformer coding layer to obtain the encoded features corresponding to the text image; Based on the fully connected coding layer, each feature in the coding features is predicted to obtain the predicted position information corresponding to each feature in the coding features, and the target position information is determined from the predicted position information; For each target location information, the identification feature corresponding to the target location information is determined according to the location indicated by the target location information, and the feature used to predict the target location information in the encoded features is determined as the detection feature.

4. The method according to claim 1, characterized in that, The step of obtaining the detection and recognition results corresponding to the text image by predicting based on the processed features through the prediction layer includes: The processed features are classified based on the first fully connected layer in the prediction layer to determine the target classification corresponding to the processed features; Based on the second fully connected layer in the prediction layer, position regression is performed on the processed features to determine the position information corresponding to the processed features; Based on the third fully connected layer in the prediction layer, text recognition is performed on the processed features to determine the text information corresponding to the processed features; The detection result and recognition result are determined based on the target classification corresponding to the processing feature, the location information, and the text information.

5. The method according to claim 4, characterized in that, The step of determining the detection result and recognition result based on the target classification corresponding to the processed features, the location information, and the text information includes: For each processing feature, if the target classification corresponding to the processing feature is a foreground classification, then the location information corresponding to the processing feature is used as the detection result, and the text information corresponding to the processing feature is determined as the recognition result.

6. A text processing device, characterized in that, The device includes: The receiving module is used to receive the text image to be processed; The processing module is used to input the text image into the text processing model to obtain the detection result and recognition result corresponding to the text image; The text processing model includes a feature extraction layer, an encoding layer, a decoding layer, and a prediction layer. The processing module includes: a first extraction submodule, used to extract features from the text image through the feature extraction layer to obtain image features corresponding to the text image; a first encoding submodule, used to encode the image features through the encoding layer to obtain detection features and recognition features corresponding to the text image, and to concatenate the detection features and recognition features to obtain image stitching features; a decoding submodule, used to decode the image stitching features through the decoding layer to obtain processing features corresponding to the text image; and a first processing submodule, used to predict based on the processing features through the prediction layer to obtain detection results and recognition results corresponding to the text image. The decoding submodule includes: The second processing submodule is used to perform multimodal attention processing based on the image stitching features to obtain a first attention feature; the third processing submodule is used to perform factorized attention processing on the first attention feature to obtain a second attention feature; and the fourth processing submodule is used to perform cross-attention processing on the second attention feature to obtain the processed feature. The second processing submodule includes: The first determining submodule is used to perform text classification processing on the recognition features, and take the features of the last feature layer in the text classification process as the text features of the recognition features; the concatenation submodule is used to concatenate the detection features and the text features to obtain text concatenation features; the fifth processing submodule is used to determine a query vector based on the image concatenation features, generate a key vector and a value vector based on the text concatenation features, and perform the multimodal attention processing based on the query vector, the key vector and the value vector to obtain the first attention features.

7. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processing device, it implements the steps of the method described in any one of claims 1-5.

8. An electronic device, characterized in that, include: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1-5.

Images