Text recognition method, server, storage medium and program product

Abstract

The invention provides a text recognition method, a server, a storage medium and a program product. According to the method, a to-be-recognized text image is input into a text recognition model for text recognition, and text marks in a target text sequence contained in the text image are sequentially output. Wherein the process of outputting the next text mark in the target text sequence through the text recognition model comprises the following steps of: extracting visual features of a text image; mapping the generated text mark into a context embedding vector; and attention calculation is carried out on the visual features, the initialized query features and the context embedding vectors to obtain visual attention features and query attention features, the visual information of the image and the language information of the generated text can be effectively integrated in the process, cross-modal feature representation can be better realized, the query attention features with higher quality can be obtained, and the user experience can be improved. And the next text mark is output according to the query attention feature, so that the text recognition accuracy in complex scenes such as shielding and blurring can be improved.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a text recognition method, server, storage medium, and program product. Background Technology

[0002] Text recognition is a task that combines vision and language, encompassing the visual texture of stroke patterns and the semantic context between character sequences. In recent years, recognizing text content from natural scenes has attracted significant attention and is a crucial prerequisite for many computer vision tasks, including scene understanding, autonomous driving, and large-scale document-based language models. Scene Text Recognition (STR), as a fundamental component of scene text reading, aims to decode text images in natural scenes into a series of characters, and it plays a key role in practical applications.

[0003] Traditional text recognition methods fall into two categories: one uses visual modality features to recognize text content, but its accuracy is low in complex scenarios such as occlusion and blurriness; the other considers the importance of linguistic context, using a separate language model to correct the visual feature-based recognition results, but this in turn ignores hidden visual information, leading to semantic illusions. In general, traditional text recognition methods have low accuracy in complex scenarios (e.g., occlusion, blurriness). Summary of the Invention

[0004] This application provides a text recognition method, server, storage medium, and program product to solve the problem of low recognition accuracy of traditional text recognition methods in complex scenarios (such as occlusion, blurring, etc.).

[0005] In a first aspect, this application provides a text recognition method, comprising: inputting a text image to be recognized into a text recognition model for text recognition, and sequentially outputting text markers in a target text sequence contained in the text image; wherein, the process of outputting the next text marker in the target text sequence through the text recognition model includes: extracting visual features of the text image; mapping the generated text markers to context embedding vectors; performing attention calculations on the visual features, initialized query features, and the context embedding vectors to obtain visual attention features and query attention features; and outputting the next text marker based on the query attention features.

[0006] Secondly, this application provides a text recognition method, comprising: in response to a call request to the text recognition model, acquiring a text image to be recognized; inputting the text image and the starting marker of the text sequence to be predicted into the text recognition model, performing text recognition, and sequentially generating text markers in the target text sequence contained in the text image; returning the target text sequence; wherein, the process of outputting the next text marker in the target text sequence through the text recognition model includes: extracting visual features of the text image; mapping the generated text markers to context embedding vectors; performing attention calculation on the visual features, initialized query features, and the context embedding vectors to obtain visual attention features and query attention features; and outputting the next text marker according to the query attention features.

[0007] Thirdly, this application provides a server, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the server to perform the methods provided in any of the foregoing aspects.

[0008] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the method provided in any of the foregoing aspects.

[0009] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the methods provided in any of the foregoing aspects.

[0010] The text recognition method, server, storage medium, and program products provided in this application, during text recognition, input the text image to be recognized into a text recognition model for text recognition, and sequentially output text markers in the target text sequence contained in the text image. The process of outputting the next text marker in the target text sequence through the text recognition model includes: extracting visual features of the text image; mapping the generated text markers to context embedding vectors; performing attention calculations on the visual features, initialized query features, and context embedding vectors to obtain visual attention features and query attention features. This process can effectively integrate the visual information of the image and the linguistic information of the generated text, better achieving cross-modal feature representation. The obtained query attention features are higher-quality multimodal features. Outputting the next text marker based on the query attention features can improve the accuracy of text recognition in complex scenes such as occlusion and blur. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0012] Figure 1 This is a schematic diagram of an example system architecture to which this application applies;

[0013] Figure 2 A flowchart for generating the next text tag is provided as an exemplary embodiment of this application;

[0014] Figure 3 Example architecture diagram of the text recognition model provided in the embodiments of this application;

[0015] Figure 4 A schematic diagram of the structure of a masked visual language decoding layer provided in an exemplary embodiment of this application;

[0016] Figure 5 Flowchart of a text recognition model training method provided for an exemplary embodiment of this application;

[0017] Figure 6 Example architecture diagram of a visual-language joint modeling model provided as an exemplary embodiment of this application;

[0018] Figure 7 A flowchart of visual-language joint modeling training provided for an exemplary embodiment of this application;

[0019] Figure 8 A flowchart of the masked visual language decoding layer provided as an exemplary embodiment of this application;

[0020] Figure 9 Example diagram of an attention mask provided for an exemplary embodiment of this application;

[0021] Figure 10 A training framework diagram for the first stage provided in an exemplary embodiment of this application;

[0022] Figure 11 A training flowchart for an autoregressive text generation task provided as an exemplary embodiment of this application;

[0023] Figure 12 A training framework diagram for the second stage provided in an exemplary embodiment of this application;

[0024] Figure 13 A flowchart of a text recognition method provided as another exemplary embodiment of this application;

[0025] Figure 14 This is a schematic diagram of the structure of a server provided in an embodiment of this application.

[0026] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0028] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0029] First, let me explain the terms used in this application:

[0030] Natural scene text recognition: refers to the technology of recognizing text content from images taken in natural scenes.

[0031] Masked Auto Encoder (MAE) is a pre-training technique for visual backbone networks.

[0032] Visual information: In this embodiment, it refers to the visual features of text in an image, such as strokes and textures.

[0033] Language information: In this embodiment, it refers to the context of each letter in the text in the image.

[0034] Text Recognition Benchmarks (TRB) test set: refers to a benchmark test set used to evaluate the accuracy of text recognition algorithms in a scene.

[0035] Text image: refers to an image containing text. If the original image contains images, headers, footers, blank areas, etc., in addition to text areas, the text area is extracted from the original image to form the text image.

[0036] A token, also known as a discrete token or discrete representation, refers to a discrete representation of speech or text. To distinguish between the discrete representations of text and speech, the discrete representation of speech is called a speech token, and the discrete representation of text is called a text token. The vocabulary used by the model contains both speech and text tokens that may appear in the prediction results. Each speech token in the vocabulary corresponds to a speech unit, such as a phoneme, syllable, or speech segment. Each text token in the vocabulary corresponds to a text unit, such as a character or word. This embodiment uses text tokens.

[0037] Transformer is a deep learning architecture widely used in natural language processing tasks, which can effectively solve the problem of modeling sequential data. Its core is the attention mechanism.

[0038] Attention is a deep learning algorithm that allows the model to focus on the task-relevant parts of the input data while ignoring irrelevant parts. The attention mechanism maps the input sequence to attention vectors, which include three input vectors: query vector q, key vector k, and value vector v. These three input vectors are also known as the query input, key input, and value input.

[0039] In this application, considering that text images possess both text texture in stroke patterns and semantics in words or lines, the importance of the interaction between vision and semantics becomes particularly prominent when there are occluded characters, blurred backgrounds, or messy handwritten text in the text image. Therefore, it is crucial to comprehensively consider both visual and semantic information in text images when performing text recognition.

[0040] However, traditional text recognition methods fall into two categories: one uses features from a single visual modality to recognize text content, resulting in low accuracy in complex scenarios such as occlusion and blurring; the other, while acknowledging the importance of linguistic context, uses an independent language model to correct visual feature-based recognition results, which in turn ignores hidden visual information and can lead to semantic illusions in the model.

[0041] In general, traditional text recognition methods have low accuracy in complex scenarios (such as occlusion and blurring), and there is considerable room for improvement.

[0042] To address the aforementioned technical issues, this application provides a text recognition method. The method inputs a text image to be recognized into a text recognition model for text recognition, and sequentially outputs text markers from the target text sequence contained within the text image. The process of outputting the next text marker in the target text sequence through the text recognition model includes: extracting visual features from the text image; mapping the generated text markers to context embedding vectors; and performing attention calculations on the visual features, initialized query features, and context embedding vectors to obtain visual attention features and query attention features. This process effectively integrates the visual information of the image and the linguistic information of the generated text, better achieving cross-modal feature representation. The obtained query attention features are higher-quality multimodal features. Outputting the next text marker based on the query attention features can improve the accuracy of text recognition in complex scenarios such as occlusion and blurring.

[0043] This application also provides a text recognition model training method. By acquiring a visual-language joint modeling model as the base model for the text recognition model, and obtaining a training set including image samples and corresponding text sequences, the visual-language joint modeling model is trained in two stages: In the first stage, the visual-language joint modeling model is trained with the visual-language joint modeling task as the training objective to obtain a first-stage model; in the second stage, the first-stage model is trained with the autoregressive text generation task as the training objective to obtain the aforementioned text recognition model. In this method, during model training, training is first performed with the visual-language joint modeling task as the training objective, followed by training with the autoregressive text generation task as the training objective. The visual-language joint reconstruction task serves as a bridge, simultaneously modeling and aligning features from both visual and semantic modalities. This effectively integrates visual and semantic information, enabling the model to learn powerful cross-modal representation capabilities. This helps the model achieve better performance in complex scenes with occlusion and blur, thereby improving the text recognition accuracy of the model in complex scenes with occlusion and blur.

[0044] Figure 1 This is a schematic diagram of an example system architecture to which this application applies. Figure 1 As shown, the system architecture includes a server and endpoint devices. The server and endpoint devices have a communication link, enabling communication between them.

[0045] The endpoint device can be an electronic device running downstream applications. Specifically, it can be a hardware device with network communication, computing, and information display functions, including but not limited to smartphones, tablets, desktop computers, local servers, and cloud servers. The downstream applications running on the endpoint device can be systems that include text recognition capabilities, such as text recognition systems, information audit management systems, network security governance systems, report recognition systems, and form recognition systems. During the operation of downstream applications, when text recognition is required, the endpoint device acquires the text image to be recognized and provides it to the server.

[0046] A server is a computing device deployed in the cloud or locally, such as a cloud cluster. Servers store text recognition models and provide text recognition services. The server is responsible for using the text recognition model to identify the text content in a given text image, obtaining the target text sequence, i.e., the text recognition result.

[0047] Furthermore, the server can return the text recognition results to the edge device. The edge device receives the text recognition results returned by the server and executes subsequent processing logic based on the text recognition results.

[0048] In addition, the server can also be responsible for model training to obtain a text recognition model. Specifically, the server acquires a visual-language joint modeling model and a training set. The sample data in the training set includes image samples and the corresponding text sequences. Using the training set, the visual-language joint modeling model is trained in the first stage to obtain a first-stage model, with the visual-language joint modeling task as the training objective. Using the training set, the first-stage model is trained in the second stage to obtain a text recognition model, with the autoregressive text generation task as the training objective.

[0049] It should be noted that, Figure 1 This example illustrates how a single server can simultaneously train and run a text recognition model, providing text recognition services to external users. In other example architectures, the training and running of the text recognition model (using text recognition services as an example) can also be implemented by different servers. For instance, a first server might train the model to obtain the text recognition model. The trained text recognition model is then deployed to a second server. The second server runs the text recognition model, providing text recognition services to external users.

[0050] The method in this embodiment can be applied to various text recognition scenarios, including but not limited to: unstructured text recognition in various industry scenarios, image text recognition in network scenarios, full text recognition, table recognition, handwriting recognition, and health code recognition for epidemic prevention.

[0051] For example, unstructured text recognition in various industry scenarios is a general text recognition scenario. It is applicable to unstructured text recognition in various industry scenarios and supports returning location coordinate information. In this scenario, the image to be recognized can be an unstructured document image. Text detection is performed on the unstructured document image to determine the text regions within the unstructured document. The text regions are then extracted from the structured document to obtain a text image. This text image is then input into a text recognition model for text recognition to obtain the text content contained within the text image.

[0052] For example, image text recognition in online scenarios is used to quickly and accurately identify characters in images from online environments such as e-commerce product promotional images, online community images, and UGC (User-generated Content) images to obtain the text content contained in the images. It has significant application value in scenarios such as identifying illegal advertisements, information review and management, and network security governance.

[0053] For example, full-text recognition refers to the ability to identify key fields within the entirety of an unstructured document. For instance, identifying key fields in a detection report includes, but is not limited to: name, ID number, detection structure, and detection result.

[0054] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0055] This application provides a text recognition method, in which the server in the aforementioned system architecture is the executing entity. The server inputs the text image to be recognized into the text recognition model for text recognition, and sequentially outputs the text tags in the target text sequence contained in the text image.

[0056] In this embodiment, the text image to be identified refers to the image of the text portion. If the original image contains images, headers, footers, blank areas, etc., in addition to the text area, then the text portion needs to be extracted from the original image to form the text image.

[0057] In one example scenario, when a device needs to perform text recognition, it can extract a text image from various original images and send the text image to the server. The device sends a text recognition request to the server, which includes the text image to be recognized. Upon receiving the text recognition request, the server retrieves the text image to be recognized.

[0058] In one example scenario, when a device needs to perform text recognition, it sends a text recognition request to the server. This text recognition request includes an input image. The input image can be an original image containing all parts except for the text. Upon receiving the text recognition request, the server acquires the input image; performs text detection on the input image to determine the regions containing text; and segments the regions containing text from the input image to obtain the text image.

[0059] The server performs text detection on the input image to determine the regions in the input image that contain text. This can be achieved using existing algorithms / models, and no specific limitations are made here.

[0060] Figure 2 A flowchart for generating the next text tag is provided for an exemplary embodiment of this application. (e.g.) Figure 2 As shown, the process of outputting the next text tag in the target text sequence through the text recognition model is as follows:

[0061] Step S201: Using the generated text tags as context information, perform cross-modal feature extraction and fusion on the text image and context information to obtain fused features.

[0062] In this step, the generated text tags are used as context information. The text recognition model performs cross-modal feature extraction and fusion on the text image and context information to obtain fused features. Then, the next text tag is output based on the fused features.

[0063] Specifically, a text recognition model extracts visual features from text images and maps the generated text tags to context embedding vectors. The text recognition model includes a visual encoding module and a language embedding module. When generating the next text tag, the text image is input into the visual encoding module of the text recognition model for visual encoding to extract the visual features of the text image. The generated text tags are then input into the language embedding module of the text recognition model to map the generated text tags to context embedding vectors.

[0064] Furthermore, attention is calculated on visual features, initialized query features, and contextual embedding vectors using a text recognition model to obtain visual attention features and query attention features. The fused features include both visual attention features and query attention features.

[0065] In this embodiment, the text recognition model further includes a masked visual language decoding module. The masked visual language decoding module performs row attention calculations on the visual features of the text image, the initialized query features, and the context embedding vector to obtain visual attention features and query attention features. The initialized query features can be obtained through random initialization and are used to predict the query attention features used for language reconstruction (i.e., generating the next text tag).

[0066] By using the masked visual language decoding module, attention calculations are performed on the visual features of the text image, the initialized query features, and the context embedding vectors of the generated text tags. This effectively integrates the visual information of the image and the linguistic information of the generated text, learning a powerful cross-modal feature representation capability, thereby obtaining high-quality query attention features.

[0067] Step S202: Output the next text tag based on the fusion features.

[0068] In this embodiment, the text recognition model further includes a language reconstruction head. In this step, query attention features are input into the language reconstruction head, and the next text tag is generated based on the query attention features.

[0069] The method in this embodiment inputs the text image to be recognized into a text recognition model for text recognition, and sequentially outputs the text tags in the target text sequence contained in the text image. The process of outputting the next text tag in the target text sequence through the text recognition model includes: extracting visual features of the text image; mapping the generated text tags to context embedding vectors; and performing attention calculations on the visual features, initialized query features, and context embedding vectors to obtain visual attention features and query attention features. This process can effectively integrate the visual information of the image and the linguistic information of the generated text, better achieving cross-modal feature representation. The obtained query attention features are higher-quality multimodal features. Outputting the next text tag based on the query attention features can improve the accuracy of text recognition in complex scenes such as occlusion and blur.

[0070] For example, Figure 3 An example architecture diagram of a text recognition model provided in an embodiment of this application. In an optional embodiment, as shown... Figure 3 As shown, the text recognition model includes: a visual encoding module, a language embedding module, a masked visual language decoding module, and a language reconstruction head.

[0071] The visual encoding module is used to extract the visual features of the input text image. The visual encoding module can be implemented using any visual encoder, such as a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN) and its variants, a Visual Transformer (ViT) and its variants, etc. This embodiment does not impose any specific limitations.

[0072] The language embedding module maps the input text sequence (including generated text tokens) to context embedding vectors. The language embedding module can be implemented using any embedding layer that maps the input text sequence to embedding vectors.

[0073] The masked visual-language decoding module performs attention calculations on the visual features of the text image, the initialized query features, and the context embedding vector to obtain visual attention features and query attention features. This module can be used in visual and language interdisciplinary tasks to decode or generate visual and linguistic features using masking strategies. For example, a Transformer-based decoder could be used. The initialized query features can be obtained through random initialization.

[0074] The language reconstruction head is used to generate the next text tag based on query attention features. The language reconstruction head can consist of one or more linear layers.

[0075] Figure 4 This is a schematic diagram of the structure of a masked visual language decoding layer provided in an exemplary embodiment of this application. In an optional embodiment, the masked visual language decoding module includes multiple stacked masked visual language decoding layers. Figure 4 As shown, each masked visual language decoding layer includes: a visual self-attention layer, a query text cross-attention layer, a visual query cross-attention layer, and a query visual cross-attention layer. ×N d The mask visual language decoding module includes stacked N d N masked visual language decoding layers d It is a positive integer.

[0076] The visual self-attention layer is used to: perform self-attention calculation on the input visual features to obtain intermediate visual features, and output the intermediate visual features to the visual query cross-attention layer and the query visual cross-attention layer.

[0077] The query text cross-attention layer is used to: perform cross-attention calculation on the input query features and context embedding vectors to obtain intermediate query features, and output the intermediate query features to the visual query cross-attention layer and the query visual cross-attention layer.

[0078] The visual query cross-attention layer is used to: take intermediate visual features as query input, intermediate query features as key input and value input, perform cross-attention calculation, and obtain visual attention features.

[0079] The query visual cross-attention layer is used to: take intermediate query features as query input, intermediate visual features as key input and value input, perform cross-attention calculation, and obtain query attention features.

[0080] In this embodiment, the visual self-attention layer, query text cross-attention layer, visual query cross-attention layer, and query visual cross-attention layer can all be implemented using a multi-head attention (MHA) module. Of course, the visual self-attention layer, query text cross-attention layer, visual query cross-attention layer, and query visual cross-attention layer can also be implemented using a single-head attention module, but this would reduce the efficiency of attention computation to some extent.

[0081] based on Figure 4 The masked visual language decoding module shown above, specifically includes the following steps in step S202: First, through a visual self-attention layer, self-attention calculation is performed on visual features to obtain intermediate visual features. Second, through a query text cross-attention layer, cross-attention calculation is performed on query features and context embedding vectors to obtain intermediate query features. Third, through a visual query cross-attention layer, cross-attention calculation is performed using intermediate visual features as query input, intermediate query features as key input, and intermediate query features as value input to obtain visual attention features. Fourth, through a query visual cross-attention layer, cross-attention calculation is performed using intermediate query features as query input, intermediate visual features as key input, and intermediate query features as value input to obtain query attention features.

[0082] Specifically, the visual features of text images Initialized query features (represented as) ) and context embedding vector F l The first-layer mask visual language decoding layer performs attention calculations, and outputs the visual attention features of the first layer (represented as...). ) and query attention features (represented as For any other layer (denoted as the nth layer) of the masked visual language decoding layer, the input is the visual attention feature output from the previous layer (the (n-1th layer) of the masked visual language decoding layer (represented as...). ) and query attention features (represented as ), and the context embedding vector F l The output is the visual attention features obtained after attention calculation (represented as...). ) and query attention features (represented as The visual attention features output by the last layer (represented as...) ) and query attention features (represented as ), which serve as the final visual attention feature and query attention feature.

[0083] It should be noted that during the text recognition process, although visual attention features and query attention features are obtained, only the query attention features are used for language reconstruction to generate the next text tag; visual attention features are not used for visual reconstruction.

[0084] The scheme in this embodiment uses a masked visual language decoding module to perform self-attention calculation on the input visual features to obtain intermediate visual features. It then performs cross-attention calculation on the input query features and context embedding vectors to obtain intermediate query features. Using the intermediate visual features as query input and the intermediate query features as key and value inputs, cross-attention calculation is performed to obtain visual attention features. Finally, using the intermediate query features as query input and the intermediate visual features as key and value inputs, cross-attention calculation is performed to obtain query attention features. This effectively integrates the visual information of the text image with the linguistic information of the generated text, better achieving cross-modal feature representation. The obtained query attention features are higher-quality multimodal features, thereby improving the model's text recognition accuracy in complex scenarios such as occlusion and blur.

[0085] This application also provides a text recognition model training method. By obtaining a visual-language joint modeling model as the base model for the text recognition model, and acquiring a training set including image samples and corresponding text sequences, the visual-language joint modeling model is trained in two stages: In the first stage, the visual-language joint modeling model is trained with the visual-language joint modeling task as the training objective to obtain a first-stage model; in the second stage, the first-stage model is trained with the autoregressive text generation task as the training objective to obtain the aforementioned text recognition model. Using the visual-language joint reconstruction task as a bridge, the features of both visual and semantic modalities are simultaneously modeled and aligned, effectively integrating visual and semantic information. This allows the model to learn powerful cross-modal representation capabilities, helping it achieve better performance in complex scenes with occlusion and blur, thereby improving the model's text recognition accuracy in complex scenes with occlusion and blur.

[0086] Figure 5This is a flowchart illustrating a text recognition model training method provided as an exemplary embodiment of this application. The execution entity in this embodiment is the server in the aforementioned system architecture. Figure 5 As shown, the specific steps of this method are as follows:

[0087] Step S501: Obtain the visual-language joint modeling model and training set. The sample data in the training set includes: image samples and text sequences corresponding to the image samples.

[0088] In this embodiment, a visual-language joint modeling model is obtained. This model can simultaneously achieve visual reconstruction and language reconstruction. It can adopt an existing multimodal model structure or be built based on an existing multimodal model structure.

[0089] For example, by adding task heads for visual reconstruction and / or language reconstruction to multimodal models such as Contrastive Language-Image Pre-training (CLIP) and Simple Visual Language Model (SVLM), an initial visual-language joint modeling model can be obtained. For multimodal models that already include a visual reconstruction head, adding a language reconstruction head yields the initial visual-language joint modeling model. For multimodal models that already include a language reconstruction head, adding a visual reconstruction head yields the initial visual-language joint modeling model. For multimodal models that do not include either a visual reconstruction head or a language reconstruction head (such as multimodal representation models), adding both visual and language reconstruction heads yields the initial visual-language joint modeling model.

[0090] The training set used in this embodiment includes image samples and corresponding text sequences. Image samples refer to text image samples, and text sequences refer to the text sequences containing the content text within the text images. The training set used in this embodiment can be an existing text recognition model's training set, or it can be a training set constructed according to actual application requirements.

[0091] Optionally, the images in the training set can be preprocessed before training. Specifically, data augmentation can be performed on the image samples in the training set to obtain richer image samples. Data augmentation methods include, but are not limited to, inverting, Gaussian blurring, and Poisson noise. Additionally, the server can resize all image samples in the training set (including those obtained through augmentation) to a fixed size and normalize the pixel values ​​to a preset range. The fixed size and preset range can be configured and adjusted according to actual needs; for example, the fixed size can be 32*128, and the preset range can be [-1,1] or [0,1], without specific limitations here.

[0092] Step S502: Using the training set, with the visual-language joint modeling task as the training objective, perform the first stage of training on the visual-language joint modeling model to obtain the first-stage model.

[0093] In this embodiment, a training set is used to train the visual-language joint modeling model in two stages.

[0094] In the first stage, the visual-language joint modeling task is used as the training objective, specifically including two parts: visual reconstruction task and language reconstruction task. The visual-language joint modeling model is jointly trained on the visual reconstruction task and the language reconstruction task. After the first stage of training is completed, the first-stage model is obtained.

[0095] In visual reconstruction tasks, the model is trained to reconstruct occluded image patches using uncluttered image patches from image samples and uncluttered text from text sequences. In language reconstruction tasks, the model is trained to reconstruct occluded text using uncluttered text from text sequences and uncluttered image patches from image samples.

[0096] By jointly training visual reconstruction tasks and language reconstruction tasks, visual information and language information (i.e. semantic information) can be effectively integrated, and the model can learn a powerful cross-modal feature representation capability.

[0097] Step S503: Using the training set, with the autoregressive text generation task as the training objective, train the first-stage model in the second stage to obtain the text recognition model.

[0098] In practical applications, text recognition tasks are essentially autoregressive text generation tasks. The process of generating the text recognition result involves multiple time steps. In the first time step, the next text tag is generated based on a given starting tag. In subsequent time steps, the next text tag is generated based on the already generated text tags. This continues until an ending tag is generated, yielding the complete text recognition result.

[0099] In the second stage, the text recognition task is trained using the autoregressive text generation task as the training objective, based on the architecture of the first stage. Optionally, since the text recognition task does not require the use of a visual reconstruction head, visual reconstruction can be disabled in the second stage of training (e.g., the visual reconstruction head can be removed or configured to be disabled), focusing on language reconstruction.

[0100] The method in this embodiment uses a visual-language joint modeling model as the base model and trains it in two stages using a training set to obtain a text recognition model. In the first stage, the visual-language joint modeling model is trained with a visual-language joint modeling task as the training objective. This allows the model to effectively integrate visual information from images and linguistic information from text sequences, learning powerful cross-modal feature representation capabilities and improving its text recognition ability in complex scenes such as occlusion and blur. In the second stage, based on the architecture of the first stage, an autoregressive text generation task is trained, focusing the model on language reconstruction and improving its text recognition ability. Through these two stages of training, the accuracy and robustness of the text recognition model are improved when applied to complex scenes such as occlusion and blur.

[0101] Figure 6 An example architecture diagram of the visual-language joint modeling model provided in this application embodiment. In an optional embodiment, as shown... Figure 6 As shown, the visual-language joint modeling model includes: a visual encoding module, a language embedding module, a masked visual-language decoding module, a visual reconstruction head, and a language reconstruction head.

[0102] The visual encoding module is used to extract the initial visual features of the input image. The visual encoding module can be implemented using any visual encoder, such as a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN) and its variants, a Visual Transformer (ViT) and its variants, etc. This embodiment does not impose any specific limitations.

[0103] The language embedding module maps input text sequences to context embedding vectors. The language embedding module can be implemented using any embedding layer that maps input text sequences to embedding vectors.

[0104] The masked visual-language decoding module performs attention calculations on visual features, initialized query features, and context embedding vectors to obtain visual attention features and query attention features. This module can be used in interdisciplinary visual and language tasks, leveraging masking strategies to decode or generate visual and linguistic features. For example, a Transformer-based decoder can be used. The initialized query features can be obtained through random initialization.

[0105] A visual reconstruction head is used to reconstruct occluded image patches in an image based on visual attention features, thus obtaining a visual reconstruction result. A visual reconstruction head can be composed of one or more linear layers.

[0106] A language reconstruction head is used to reconstruct masked text markers in a text sequence based on query attention features, thus obtaining the language reconstruction result. Alternatively, a language reconstruction head is used to generate a prediction result for a text sequence based on query attention features. A language reconstruction head can consist of one or more linear layers.

[0107] The following is based on Figure 6 The architecture of the visual-language joint modeling model is shown, and the training process in the first stage is explained in detail.

[0108] Figure 7 A flowchart illustrating visual-language joint modeling training provided for an exemplary embodiment of this application. (See attached flowchart.) Figure 7 As shown, in the aforementioned step S502, the training set is used to train the visual-language joint modeling model in the first stage, with the visual-language joint modeling task as the training objective, to obtain the first-stage model. The specific implementation process is as follows:

[0109] Step S701: Obtain image samples and corresponding text sequences from the training set.

[0110] The training set used in this embodiment includes image samples and corresponding text sequences. Image samples refer to text image samples, and text sequences refer to the text sequences containing the content text within the text images. The training set used in this embodiment can be an existing text recognition model's training set, or it can be a training set constructed according to actual application requirements.

[0111] Step S702: Cover at least one image block in the image sample, and encode the covered image sample through the visual coding module to obtain the first visual feature.

[0112] In the first stage, the training objective is a joint visual-language modeling task, specifically comprising two parts: a visual reconstruction task and a language reconstruction task. The joint visual-language modeling model is trained on both tasks. Through joint training on these two tasks, visual and linguistic information (i.e., semantic information) can be effectively integrated, enabling the model to learn powerful cross-modal feature representation capabilities.

[0113] In visual reconstruction tasks, models are trained to reconstruct occluded image patches using occluded image patches in image samples and occluded text in text sequences.

[0114] In this step, for image samples (i.e., text images) I in the training set, a fixed image patch size (p) is used. h ,p w The image sample I is segmented into multiple segments of size (p). h ,p w Image blocks of image sample I. At least one image block in image sample I is masked, and the masked image block is designated as an invisible image block, while the unmasked image block is designated as a visible image block.

[0115] Optionally, when covering at least one image block in image sample I, at least one image block in the image sample can be randomly covered according to a configured visual occlusion ratio, such that the proportion of the covered invisible image blocks is equal to the configured visual occlusion ratio, or the proportion of the covered invisible image blocks is close to the visual occlusion ratio (e.g., the difference is less than a first threshold). The visual occlusion ratio can be configured and adjusted according to the needs of the actual application scenario and empirical values; for example, the visual occlusion ratio can be configured to 0.75, and no specific limitation is made here. The first threshold can also be configured and adjusted according to the needs of the actual application scenario and empirical values, and no specific limitation is made here.

[0116] Optionally, when occluding at least one image block in image sample I, a corresponding number of image blocks can be randomly selected from the image blocks of the image sample according to the configured number of visual occlusions, and these selected blocks can be used as the occluded invisible image blocks. The number of visual occlusions can be configured and adjusted according to the needs of the actual application scenario and empirical values. For example, the number of visual occlusions can be 1, 2, 3, 5, etc., and no specific limitation is made here.

[0117] Furthermore, the visible image patches are input into the visual encoding module for encoding, resulting in feature maps corresponding to each visible image patch. For invisible image patches, the feature map corresponding to each invisible image patch is set as a learnable mask feature. This yields feature maps for all image patches contained in image sample I. The first visual feature contains feature maps for all image patches of the image sample. The learnable mask features are continuously trained and iterated during the first stage of training.

[0118] For example, the first visual feature is obtained by encoding the masked image sample through a visual encoding module. This process can be expressed as the following equation (1):

[0119]

[0120] in, Indicates primary visual features. * This represents the set of visible image patches. ViT(i|i∈I) * This includes feature maps of visible image blocks obtained after encoding by the visual encoding module. Including feature maps of invisible image patches, i.e., learnable mask features.

[0121] Step S703: Cover at least one text tag in the text sequence corresponding to the masked image sample, and map the masked text sequence into an embedding vector through the language embedding module to obtain the first context embedding vector.

[0122] In language reconstruction tasks, models are trained to reconstruct occluded text using occluded text in text sequences and occluded image patches in image samples.

[0123] In this step, for the text sequence corresponding to image sample I in the training set, at least one text marker in the text sequence is replaced with mask text (a special text marker set in advance) to cover at least one text marker in the text sequence corresponding to the image sample.

[0124] Optionally, when masking at least one text marker in the text sequence corresponding to the image sample, at least one text marker in the text sequence can be replaced with masked text according to a configured text masking ratio to obtain a masked text sequence. The proportion of the replaced text markers is equal to the text masking ratio, or the proportion of the replaced text markers is close to the text masking ratio (e.g., the difference is less than a second threshold). The text masking ratio can be configured and adjusted according to the needs of the actual application scenario and empirical values; for example, the text masking ratio can be configured to 0.2, without specific limitations here. The second threshold can also be configured and adjusted according to the needs of the actual application scenario and empirical values, without specific limitations here.

[0125] Optionally, when masking at least one text marker in the text sequence corresponding to the image sample, a corresponding number of text markers can be randomly selected from the text sequence corresponding to the image sample and replaced with masked text, based on the configured text masking quantity. The text masking quantity can be configured and adjusted according to the needs and experience of the actual application scenario; for example, the text masking quantity can be 1, 2, etc., and no specific limitation is made here.

[0126] Furthermore, the masked text sequence is input into the language embedding module for mapping to obtain the first context embedding vector. For example, this process can be represented by the following equation (2):

[0127] F l =Emb(t|t∈T)(2)

[0128] Among them, F l Let Emb(t|t∈T) be the first context embedding vector, which includes the embedding vectors of each text tag obtained by the language embedding module. Let T represent the text sequence corresponding to the image sample.

[0129] Step S704: Through the masked visual language decoding module, attention calculation is performed on the first visual feature, the initialized first query feature, and the first context embedding vector to obtain the first visual attention feature and the first query attention feature.

[0130] The first query feature can be obtained through random initialization and is used to predict the first query attention feature for language reconstruction. The first visual attention feature is used for visual reconstruction.

[0131] After obtaining the first visual feature and the first context embedding vector, the first visual feature, the initialized first query feature, and the first context embedding vector are input into the masked visual language decoding module for attention calculation, thereby decoding to obtain the first visual attention feature and the first query attention feature. The specific implementation of the masked visual language decoding module will be described in detail in the following embodiments.

[0132] By using the masked visual language decoding module, attention calculations are performed on the first visual features, the initialized first query features, and the first context embedding vector. This effectively integrates the visual information of the image and the linguistic information of the text sequence, learning a powerful cross-modal feature representation capability, thereby obtaining high-quality first visual attention features and first query attention features.

[0133] Step S705: Using the visual reconstruction head, reconstruct the occluded image patch based on the first visual attention features to obtain the visual reconstruction result.

[0134] In this step, the first visual attention feature is input into the visual reconstruction head to reconstruct the occluded image patch in image sample I (i.e., visual reconstruction) and obtain the visual reconstruction result. The visual reconstruction result includes the prediction result of the occluded image patch in image sample I.

[0135] Step S706: Using the language reconstruction head, reconstruct the masked text tags based on the first query attention features to obtain the language reconstruction result.

[0136] In this step, the first query attention feature is input into the language reconstruction head to reconstruct the masked text markers in the text sequence T corresponding to image sample I (i.e., language reconstruction), and obtain the language reconstruction result. The language reconstruction result includes the prediction result of the masked text markers in the text sequence T.

[0137] Step S707: Calculate the reconstruction task loss based on the visual reconstruction results and the language reconstruction results.

[0138] After obtaining the visual reconstruction results and the language reconstruction results, in this step, the reconstruction task loss is calculated based on the visual reconstruction results and the language reconstruction results according to the configured reconstruction task loss function.

[0139] In this step, the visual reconstruction loss is calculated based on at least one image patch and the visual reconstruction result; the language reconstruction loss is calculated based on at least one text tag and the language reconstruction result; and the reconstruction task loss is determined based on the visual reconstruction loss and the language reconstruction loss.

[0140] The visual reconstruction loss can be calculated using the Mean Square Error (MSE) loss function, or other commonly used loss functions in visual reconstruction tasks. The language reconstruction loss can be calculated using the Cross-Entropy Loss (CEL) loss function, or other commonly used loss functions in language reconstruction tasks.

[0141] For example, the visual reconstruction loss can be calculated based on at least one image patch and the visual reconstruction result using the following equation (3):

[0142]

[0143] Among them, L v This indicates visual reconstruction loss. M v |M represents the set of invisible image patches that are covered. v | indicates the number of invisible image patches that are obscured. i∈M v Refers to any invisible image patch. i This represents the visual reconstruction result of the invisible image block i. This represents the truth value of the invisible image block i.

[0144] For example, the language reconstruction loss can be calculated based on at least one text tag and the language reconstruction result using the following equation (4):

[0145]

[0146] Among them, L l This indicates language reconstruction loss. M l Indicates the text mark that is covered, |M l | indicates the number of text tags that are covered. i∈M l Refers to any obscured text mark. i This represents the language reconstruction result of the masked text tag i. This represents the truth value of the text tag i that is being covered. This represents the cross-entropy loss (CEL).

[0147] Furthermore, the reconstruction task loss is obtained by weighted summing of the visual reconstruction loss and the language reconstruction loss. This process can be represented as: L = λ v L v +λ l L l Where L represents the reconstruction task loss. λ v and λ l λ represents the weighting coefficients for visual reconstruction loss and language reconstruction loss, respectively. v and λ l It can be configured and adjusted according to the needs and experience of the actual application scenario, for example, λ v and λ l All values ​​can be set to 1.0; no specific limit is specified here.

[0148] In an optional embodiment, the masked visual language decoding module includes multiple stacked masked visual language decoding layers. After obtaining the first visual feature and the first context embedding vector, the first visual feature... The first query feature initialized (represented as) ) and the first context embedding vector F l The first-layer mask visual language decoding layer performs attention calculations, and outputs the visual attention features of the first layer (represented as...). ) and query attention features (represented as For any other layer (denoted as the nth layer) of the masked visual language decoding layer, the input is the visual attention feature output from the previous layer (the (n-1th layer) of the masked visual language decoding layer (represented as...). ) and query attention features (represented as ), and the first context embedding vector F l The output is the visual attention features obtained after attention calculation (represented as...). ) and query attention features (represented as ).

[0149] Optionally, during visual reconstruction in step S705, visual reconstruction is performed using the visual reconstruction head based on the first visual attention features output by the masked visual language decoding module (i.e., the last masked visual language decoding layer) to obtain a visual reconstruction result. Correspondingly, during language reconstruction in step S706, language reconstruction is performed using the language reconstruction head based on the first query attention features output by the masked visual language decoding module (i.e., the last masked visual language decoding layer) to obtain a language reconstruction result.

[0150] Optionally, during visual reconstruction in step S705, visual reconstruction can be performed based on the visual attention features output by at least one intermediate layer (excluding the last layer) in the masked visual language decoding module. Correspondingly, during language reconstruction in step S706, language reconstruction can be performed based on the query attention features output by at least one intermediate layer (excluding the last layer) in the masked visual language decoding module through the language reconstruction head.

[0151] For example, in step S705, the visual attention features output from each layer of the masked visual language decoding module (such as...) are... The visual reconstruction head is input into each layer to perform visual reconstruction, obtaining the visual reconstruction results for each layer. In step S706, the query attention features (such as...) output from each layer of the masked visual language decoding module are... Each language reconstruction head is input to perform language reconstruction, and the language reconstruction results corresponding to each layer are obtained.

[0152] For example, multiple target layers are selected from the masked visual language decoding module. In step S705, the visual attention features output by each target layer are input into the visual reconstruction head for visual reconstruction to obtain the corresponding visual reconstruction result. In step S706, the query attention features output by each target layer are input into the language reconstruction head for language reconstruction to obtain the corresponding language reconstruction result.

[0153] For example, based on the visual attention features output from the nth layer in the masked visual language decoding module The process of visual reconstruction can be represented by the following equation (5):

[0154]

[0155] Among them, v nRepresents visual attention features based on the output of the nth layer. The visual reconstruction results. Head v (·) indicates the processing procedure of the visual reconstruction head.

[0156] For example, based on the query attention features output from the nth layer in the masked visual language decoding module The process of language reconstruction can be represented by the following equation (6):

[0157]

[0158] Among them, l n This represents the query attention feature based on the output of the nth layer. The language reconstruction results. Head l (·) indicates the processing procedure of the language reconstruction head.

[0159] Furthermore, based on at least one image patch and the visual reconstruction results corresponding to each layer, the visual reconstruction loss is calculated, which can be achieved using the following equation (7):

[0160]

[0161] Among them, L v This indicates visual reconstruction loss. M v |M represents the set of invisible image patches that are covered. v | indicates the number of invisible image patches that are obscured. i∈M v Refers to any invisible image patch. N represents the set of masked visual language decoding layers for visual reconstruction, and k∈N represents any masked visual language decoding layer for visual reconstruction. This represents the visual reconstruction result of the invisible image block i obtained by visual reconstruction based on the output of the k-th layer. This represents the truth value of the invisible image block i.

[0162] For example, the language reconstruction loss can be calculated based on at least one text tag and the language reconstruction results corresponding to each layer, using the following equation (8):

[0163]

[0164] Among them, L l This indicates language reconstruction loss. M l Indicates the text mark that is covered, |M l | indicates the number of text tags that are covered. i∈M l Refers to any masked text mark. N represents the set of masked visual language decoding layers for visual reconstruction, and k∈N represents any masked visual language decoding layer for visual reconstruction. This represents the language reconstruction result of the masked text tag i obtained by language reconstruction based on the output of the k-th layer. This represents the truth value of the text tag i that is being covered. This represents the cross-entropy loss.

[0165] Step S708: Based on the reconstruction task loss, train the visual-language joint modeling model to obtain the first-stage model.

[0166] After obtaining the reconstruction task loss, the parameters and learnable mask features of the visual-language joint modeling model are adjusted through backpropagation based on the reconstruction task loss. After the first stage of training, a first-stage model is obtained.

[0167] It should be noted that the strategies used in the first stage of training in this embodiment, including but not limited to the optimization algorithm, learning rate, and batch size, can be configured and adjusted according to the needs of the actual application scenario, and are not specifically limited here. For example, the batch size can be set to 4×8, and the initial learning rate can be set to 7e-4, i.e., 7×10. -4 .

[0168] In this embodiment, the method uses the visual-language joint modeling task as the training objective in the first stage of training the model. Specifically, it includes two parts: a visual reconstruction task and a language reconstruction task. The model is jointly trained using both tasks. Through this joint training, the visual reconstruction process considers textual context information, and the language reconstruction process considers visual context. This effectively integrates visual and linguistic information, enabling the model to learn powerful cross-modal feature representation capabilities. This improves the model's accuracy in text recognition, especially in complex scenarios such as occlusion and blurring, significantly enhancing its accuracy and robustness.

[0169] based on Figure 4 The masked visual language decoding module shown above, specifically includes the following steps in the implementation process of step S704:

[0170] A first intermediate visual feature is obtained by performing self-attention calculation on the first visual feature through a visual self-attention layer; a first intermediate query feature is obtained by performing cross-attention calculation on the first query feature and the first context embedding vector through a query text cross-attention layer; a first intermediate query feature is obtained by performing cross-attention calculation on the first intermediate visual feature as query input and the first intermediate query feature as key input and value input through a visual query cross-attention layer; and a first query attention feature is obtained by performing cross-attention calculation on the first intermediate query feature as query input and the first intermediate visual feature as key input and value input through a query visual cross-attention layer.

[0171] Specifically, the first visual features The first query feature initialized (represented as) ) and the first context embedding vector F l The first-layer mask visual language decoding layer performs attention calculations, and outputs the visual attention features of the first layer (represented as...). ) and query attention features (represented as For any other layer (denoted as the nth layer) of the masked visual language decoding layer, the input is the visual attention feature output from the previous layer (the (n-1th layer) of the masked visual language decoding layer (represented as...). ) and query attention features (represented as ), and the first context embedding vector F l The output is the visual attention features obtained after attention calculation (represented as...). ) and query attention features (represented as The visual attention features output by the last layer (represented as...) ) and query attention features (represented as ), which serve as the final first visual attention feature and the first query attention feature.

[0172] Figure 8 A flowchart of the masked visual language decoding layer provided for an exemplary embodiment of this application is shown below. Figure 8 As shown, the processing flow of the nth layer mask visual language decoding layer is as follows:

[0173] Step S801: Through the visual self-attention layer, perform self-attention calculation on the visual attention features output by the (n-1)th layer to obtain the intermediate visual features of the nth layer.

[0174] In this step, the visual attention features output from layer (n-1) are... Input the nth layer of the masked visual language decoding layer into the visual self-attention layer, perform self-attention calculation, and obtain the intermediate visual features of the nth layer (which can be represented as...). This process can be represented as:

[0175] Step S802: By using the query text cross-attention layer, cross-attention calculation is performed on the query attention features output by layer n-1 and the first context embedding vector to obtain the intermediate query features of layer n.

[0176] In this step, the query attention features output from layer (n-1) are... and the first context embedding vector F l The query text of the nth layer mask visual language decoding layer is input to the cross-attention layer for cross-attention calculation, resulting in the intermediate query features of the nth layer (which can be represented as...). This process can be represented as:

[0177] Where, m q,l The attention mask used by the query text cross-attention layer is the same for all different mask visual language decoding layers.

[0178] In this embodiment, the attention mask m used by the query text cross-attention layer q,l It can be determined using any of the following methods:

[0179] The first method involves obtaining the attention mask corresponding to the text sequence; setting the value corresponding to the masked text marker in the attention mask to the first value, which is then used as the attention mask for the query text cross-attention layer. The first value can be 0, or it can be any other symbol representing something hidden / masked / invisible in the masking operation; no specific limitations are imposed here.

[0180] The second approach involves obtaining the attention mask corresponding to the text sequence; shuffling the attention mask, and setting the value corresponding to the masked text marker in the shuffled mask matrix to the first value, which is then used as the attention mask for the query text cross-attention layer. This second approach utilizes Permuted Language Modeling (PLM) technology, which enables better context modeling.

[0181] The attention mask corresponding to the text sequence can be implemented using the attention mask commonly used in autoregressive text generation tasks, such as... Figure 9 The attention mask corresponding to the text sequence "L→O→A→N" shown.

[0182] When shuffling the attention mask corresponding to a text sequence, the values ​​corresponding to each text tag in the attention mask can be randomly shuffled to obtain a shuffled mask matrix. For example, Figure 9 The text sequence corresponding to the scrambled mask matrix shown is "O→A→N→L".

[0183] Optionally, by setting the value corresponding to the masked text marker to 0 in the attention mask or scrambled mask matrix corresponding to the text sequence, the masking mask matrix corresponding to the masked text marker can be obtained. By performing a logical AND operation between the attention mask or scrambled mask matrix corresponding to the text sequence and the masking mask matrix, the attention mask used by the cross-attention layer of the query text can be obtained (e.g., ...). Figure 9 (As shown). The masking matrix has the same dimension as the attention mask corresponding to the text sequence. The masked text markers in this matrix have a value of 0 to prevent the leakage of masked information. For example... Figure 9 As shown, assuming the letter "A" in the text sequence "L→O→A→N" is masked, the masking matrix is ​​as follows: Figure 9 As shown in the image.

[0184] Optionally, to improve the robustness of language modeling, the same image sample and text sequence can be repeatedly input into the visual-language joint modeling model for joint visual-language reconstruction. In each of the multiple joint visual-language reconstructions, the attention mask m used by the query text cross-attention layer, determined using the first method described above, is used in at least one of the reconstructions. q,l The attention mask m used in the query text cross-attention layer is determined at least once using the second method described above in the joint visual-language reconstruction. q,l In the joint reconstruction of different visual languages, a second method can be used to determine different attention masks m. q,l .

[0185] For example, by setting the number of permutations, the second method involves randomly shuffling the attention mask corresponding to the text sequence multiple times (the number of permutations), and setting the values ​​corresponding to the masked text markers in the shuffled mask matrix to 0, thus obtaining multiple corresponding attention masks m. q,l Based on the number of permutations, the same image sample and text sequence are repeatedly input into the visual-language joint modeling model multiple times (permutation number + 1) to perform visual-language joint reconstruction. Each reconstruction uses a different m... q,l The number of permutations can be configured according to actual application requirements; for example, the number of permutations can be 6, and no specific limit is set here.

[0186] Step S803: Through the visual query cross-attention layer, the intermediate visual features of the nth layer are used as query input, and the intermediate query features of the nth layer are used as key input and value input to perform cross-attention calculation, so as to obtain the visual attention features of the nth layer.

[0187] The process of this step can be represented as follows: in, This represents the visual attention features of the nth layer.

[0188] Step S804: By querying the visual cross-attention layer, the intermediate query features of the nth layer are used as query inputs, and the intermediate visual features of the nth layer are used as key and value inputs, and cross-attention calculation is performed to obtain the query attention features of the nth layer.

[0189] The process of this step can be represented as follows: in, This represents the query attention feature of the nth layer.

[0190] In this embodiment, the masked visual language decoding module performs sufficient attention calculations on the first visual feature, the initialized first query feature, and the first context embedding vector input to decode the first visual attention feature and the first query attention feature. The specific implementation of the masked visual language decoding module will be described in detail in later embodiments.

[0191] The masked visual language decoding module comprises multiple stacked masked visual language decoding layers. Each masked visual language decoding layer includes a visual self-attention layer, a query text cross-attention layer, a visual query cross-attention layer, and a query visual cross-attention layer. The visual self-attention layer enables interaction between visual contexts, the query text cross-attention layer enables interaction between linguistic contexts, and the visual query cross-attention layer and the query visual cross-attention layer enable interaction between visual and linguistic modalities. This effectively integrates the visual information of images and the linguistic information of text sequences, learning powerful cross-modal feature representation capabilities, thereby obtaining high-quality first visual attention features and first query attention features.

[0192] Figure 10 A training framework diagram for the first stage provided in an exemplary embodiment of this application. (See diagram below.) Figure 10 As shown, the visual-language joint modeling model includes: a visual encoding module, a language embedding module, a masked visual-language decoding module, a visual reconstruction head, and a language reconstruction head. In the first stage of training for the visual-language joint modeling task, some image patches in the image samples are masked (e.g., ...). Figure 10 The black rectangular blocks on the input image represent the masked image blocks. These blocks are then encoded by the visual encoding module to obtain the initial visual features (i.e., the first visual features) of the image sample. Masking part of the text markers in the text sequence corresponding to the image sample (e.g.) Figure 10 The “lo#n” shown in the image (where # is a masking marker used to cover text tags) is then input into the language embedding module for mapping to obtain the first context embedding vector F. lThe initial visual features of the image samples. and the first context embedding vector F l The input mask visual language decoding module performs decoding processing to obtain the first visual attention feature F. v And the first query attention feature F q The first visual attention feature F v The visual reconstruction head is input to perform visual reconstruction, and the visual reconstruction result is obtained. The first query attention feature F is then applied. q The input language reconstruction head is used for language reconstruction to obtain the language reconstruction result. Based on the ground truth values ​​of the corresponding image samples in the visual reconstruction result, the visual reconstruction loss L is calculated. v Based on the language reconstruction results and the ground truth values ​​of the corresponding text sequences, the language reconstruction loss L is calculated. l According to visual reconstruction loss L v and language reconstruction loss L l Train the visual-language joint modeling model.

[0193] It should be noted that in practical applications, when inputting image patches from image samples into the visual encoding module, the first positional code of the image patch must also be input. This first positional code contains the position of the image patch within the image sample. Similarly, when inputting text sequences into the language embedding module, the second positional code corresponding to the text sequence must also be input. This second positional code contains the positional information of each text marker within the text sequence.

[0194] Figure 11 This is a flowchart illustrating the training process for an autoregressive text generation task provided as an exemplary embodiment of this application. Figure 11 As shown, in step S503 above, the training set is used to train the first-stage model in the second stage with the autoregressive text generation task as the training objective, to obtain the text recognition model. The specific implementation process is as follows:

[0195] Step S1101: Remove or disable the visual reconstruction head of the first-stage model.

[0196] When training the first-stage model obtained from the first stage of training in the second stage, the training objective is autoregressive text generation, and a visual reconstruction head is not required. Therefore, in the second stage, the visual reconstruction head of the first-stage model can be removed or disabled, while keeping the architecture of other modules in the first-stage model unchanged.

[0197] Step S1102: For image samples in the training set, the image samples are encoded by the visual encoding module to obtain the second visual features.

[0198] In the second stage of training, there is no need to mask the image samples and text sequences. The image samples in the training set are input into the visual encoding module for encoding to obtain the second visual features.

[0199] Step S1103: Map the text sequence corresponding to the image sample to an embedding vector through the language embedding module to obtain the second context embedding vector.

[0200] In this step, the text sequence corresponding to the image sample is input into the language embedding module, and the language embedding module maps the text sequence corresponding to the image sample into an embedding vector to obtain the second context embedding vector.

[0201] Step S1104: Through the masked visual language decoding module, attention is calculated on the second visual feature, the initialized second query feature, and the second context embedding vector to obtain the second visual attention feature and the second query attention feature.

[0202] The second query feature can be obtained through random initialization and is used to predict the second query attention feature for language reconstruction. The second visual attention feature is used for visual reconstruction.

[0203] After obtaining the first visual feature and the first context embedding vector, in this step, the second visual feature, the initialized second query feature, and the second context embedding vector are input into the mask visual language decoding module for attention calculation, so as to decode the second visual attention feature and the second query attention feature.

[0204] By using the masked visual language decoding module to perform attention calculations on the second visual features, the initialized second query features, and the second context embedding vector, the visual information of the image and the linguistic information of the text sequence can be effectively integrated, and a powerful cross-modal feature representation capability can be learned, thereby obtaining high-quality second visual attention features and second query attention features.

[0205] The implementation principle of this step is similar to that of step S704 mentioned above, except that the attention mask m used by the query text cross-attention layer is different. q,l Unlike the first stage, the specific implementation principle is detailed in the aforementioned embodiments, and is not specifically limited here.

[0206] In the second stage, the attention mask m used by the query text cross-attention layer q,l This can be the attention mask corresponding to the current sequence, or a mask matrix obtained by scrambling the attention masks corresponding to the text sequence. In other words, the mask matrix used is a sentence with all values ​​of 1.

[0207] Step S1105: Generate the prediction result of the text sequence based on the second query attention features through the language reconstruction head.

[0208] In this step, the second query attention feature is input into the language reconstruction head to perform autoregressive generation on the text sequence corresponding to the image sample, and the prediction result of the text sequence is obtained.

[0209] Step S1106: Calculate the autoregressive text generation task loss based on the prediction results of the text sequence.

[0210] In this step, the cross-entropy loss function value is calculated based on the prediction results of the text sequence, and is used as the loss for the autoregressive text generation task.

[0211] The calculation principle for the autoregressive text generation task loss in this step is consistent with the principle for calculating the language reconstruction loss in the aforementioned embodiments. Please refer to the relevant content in the aforementioned embodiments for details, which will not be repeated here. The difference lies in that, in the second stage, the entire text sequence is used as the target of language reconstruction, and the prediction result of the text sequence is used as the language reconstruction result.

[0212] Step S1107: Train a one-stage model based on the autoregressive text generation task loss to obtain a text recognition model.

[0213] After obtaining the autoregressive text generation task loss, the parameters of the first-stage model (excluding the visual reconstruction head) are adjusted through backpropagation based on the autoregressive text generation task loss. After the second stage of training, the text recognition model is obtained.

[0214] It should be noted that the strategies used for the second stage of training in this embodiment, including but not limited to the optimization algorithm, learning rate, and batch size, can be configured and adjusted according to the needs of the actual application scenario, and are not specifically limited here. For example, the batch size can be set to 4×8, and the initial learning rate can be set to 1e-4, i.e., 1×10⁻⁴. -4 .

[0215] In this embodiment, after the first stage of training of the visual-language joint modeling model, the visual reconstruction head of the first-stage model is removed or disabled based on the architecture of the first-stage model. The second stage of training is then carried out with the autoregressive text generation task as the training objective, so that the model focuses more on language reconstruction and can further improve the model's text recognition ability.

[0216] Figure 12 A diagram illustrating the training framework for the second stage provided in an exemplary embodiment of this application. (See diagram for example.) Figure 12 As shown, in the second stage of training, the visual reconstruction head is removed or disabled, while the architecture of other modules in the first-stage model remains unchanged. In the training of the autoregressive text generation task in the second stage, image samples are input into the visual encoding module for encoding to obtain the initial visual features (i.e., the second visual features) of the image samples. The text sequence corresponding to the image sample is input into the language embedding module for mapping, resulting in the second context embedding vector F. l,2 The initial visual features of the image samples. Second context embedding vector F l,2 The input mask visual language decoding module performs decoding processing to obtain the second visual attention feature F. v,2 Second query attention feature F q,2 The second query attention feature F q,2 The input language reconstruction head performs language reconstruction (and autoregressive text generation) to obtain the predicted text sequence. Based on the predicted text sequence and the ground truth text sequence, the autoregressive text generation task loss L′ is calculated. l The loss L′ for the autoregressive text generation task is calculated. l Train a one-stage model (excluding the visual reconstruction head).

[0217] It should be noted that in practical applications, when inputting image patches from image samples into the visual encoding module, the first positional code of the image patch must also be input. This first positional code contains the position of the image patch within the image sample. Similarly, when inputting text sequences into the language embedding module, the second positional code corresponding to the text sequence must also be input. This second positional code contains the positional information of each text marker within the text sequence.

[0218] Figure 13 This is a flowchart illustrating a text recognition method provided as another exemplary embodiment of this application. The execution entity in this embodiment is a server running a text recognition model. Figure 13 As shown, the specific steps of this method are as follows:

[0219] Step S1301: In response to the call request to the text recognition model, obtain the text image to be recognized.

[0220] In this embodiment, the server provides an Application Programming Interface (API) for the text recognition model to external end-user devices. End-user devices can use the text recognition capabilities of the text recognition model by calling the API.

[0221] When a device needs to perform text recognition, it sends an API call request to the server for the text recognition model. This API call request includes the text image to be recognized. The server responds to the API call request and retrieves the text image to be recognized.

[0222] Step S1302: Input the text image and the starting marker of the text sequence to be predicted into the text recognition model for text recognition, and generate the text markers in the target text sequence contained in the text image in sequence.

[0223] The process of outputting the next text tag in the target text sequence through the text recognition model includes: using the generated text tag as context information, performing cross-modal feature extraction and fusion on the text image and context information to obtain fused features; and outputting the next text tag based on the fused features.

[0224] Specifically, visual features of the text image are extracted; the generated text tags are mapped to context embedding vectors; attention is calculated on the visual features, initialized query features, and context embedding vectors to obtain visual attention features and query attention features; the fused features include both visual attention features and query attention features. Further, the next text tag is output based on the query attention features.

[0225] For the specific implementation of this step, please refer to the relevant content in the foregoing embodiments, which will not be repeated here.

[0226] In this embodiment, the training process of the text recognition model includes: acquiring a visual-language joint modeling model and a training set, wherein the sample data in the training set includes: image samples and text sequences corresponding to the image samples; using the training set, with the visual-language joint modeling task as the training objective, performing a first-stage training on the visual-language joint modeling model to obtain a first-stage model; and using the training set, with the autoregressive text generation task as the training objective, performing a second-stage training on the first-stage model to obtain the text recognition model.

[0227] In this embodiment, the two-stage training process of the text recognition model is detailed in the relevant content of the aforementioned embodiments and will not be repeated here. The trained text recognition model includes: a visual encoding module, a language embedding module, a masked visual language decoding module, and a language reconstruction head.

[0228] In the inference process using the text recognition model, an autoregressive pattern is employed to generate the text sequence. In the first iteration, the text image is input into the visual encoding module of the text recognition model, and the start tag is input into the language embedding module. Inference is performed through various modules of the text recognition model, and finally, the language reconstruction head outputs the first text tag of the text sequence.

[0229] In subsequent iterations, the text image is input into the visual encoding module of the text recognition model, and the generated text tag sequence is input into the language embedding module. The various modules of the text recognition model perform inference, and finally, the language reconstruction head outputs the next text tag for the text sequence. This process continues until an end tag is generated, at which point a complete text sequence is obtained as the text recognition result.

[0230] Step S1304: Return the target text sequence.

[0231] After obtaining the target text sequence of the text image, the server returns the target text sequence to the end device.

[0232] The implementation principle and technical effects of this embodiment are detailed in the foregoing embodiments and will not be repeated here.

[0233] Figure 14 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Figure 14 As shown, the server includes a memory 1401 and a processor 1402. The memory 1401 stores computer-executable instructions and can be configured to store various other data to support operations on the server. The processor 1402 is communicatively connected to the memory 1401 and executes the computer-executable instructions stored in the memory 1401 to implement the technical solutions provided in any of the above-described method embodiments. Their specific functions and the technical effects they achieve are similar and will not be repeated here.

[0234] Optional, such as Figure 14 As shown, the server also includes other components such as a firewall 1403, a load balancer 1404, a communication component 1405, and a power supply component 1406. Figure 14 The diagram only shows some components and does not mean that the server only includes... Figure 14 The components shown. Figure 14 This example uses a cloud server deployed in the cloud as an example, but the server can also be deployed locally. This embodiment does not make any specific limitations here.

[0235] This application also provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the method of any of the foregoing embodiments. The specific functions and technical effects to be achieved are not described here.

[0236] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method of any of the foregoing embodiments. The computer program is stored in a readable storage medium, and at least one processor of the server can read the computer program from the readable storage medium. The execution of the computer program by the at least one processor causes the server to perform the technical solution provided in any of the above method embodiments. The specific functions and the technical effects that can be achieved are not described here.

[0237] This application provides a chip, including a processing module and a communication interface. The processing module is capable of executing the technical solution of the server in the aforementioned method embodiments. Optionally, the chip further includes a storage module (e.g., a memory), which stores instructions. The processing module executes the instructions stored in the storage module, and the execution of the instructions stored in the storage module causes the processing module to execute the technical solution provided in any of the aforementioned method embodiments.

[0238] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0239] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules from at least one processor.

[0240] The memory may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk drive, and may also be a USB flash drive, external hard drive, read-only memory, disk, or optical disc. The aforementioned memory can be object storage service (OSS). The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk, or optical disc.

[0241] The aforementioned communication components are configured to facilitate wired or wireless communication between the device housing the communication components and other devices. The device housing the communication components can access wireless networks based on communication standards, such as mobile hotspots (WiFi), 2G, 3G, 4G / Long Term Evolution (LTE), 5G, or combinations thereof. In one exemplary embodiment, the communication components receive broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication components also include a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be based on Radio Frequency Identification (RFID), infrared, Ultra Wide Band (UWB), Bluetooth, and other technologies. The aforementioned power supply components provide power to various components of the device housing the power supply components. The power supply components may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device housing the power supply components.

[0242] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0243] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside within an application-specific integrated circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components within an electronic device or host device.

[0244] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0245] The order of the embodiments described above is merely for illustrative purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The sequence numbers are merely used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types. "Multiple" means two or more, unless otherwise explicitly specified.

[0246] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0247] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0248] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A text recognition method, characterized by, The method comprises: inputting a text image to be recognized into a text recognition model for text recognition, and sequentially outputting text labels in a target text sequence contained in the text image; wherein the process of outputting a next text label in the target text sequence by the text recognition model comprises: extracting a visual feature of the text image; mapping the generated text label into a context embedding vector; performing attention calculation on the visual feature, the initialized query feature and the context embedding vector to obtain visual attention features and query attention features, and the fusion feature comprises the visual attention features and the query attention features; 2. The method of claim 1, wherein, Accordingly, the process of outputting a next text label according to the fusion feature comprises: outputting a next text label according to the query attention feature. The text recognition model comprises a visual encoding module, a language embedding module, a mask visual language decoding module and a language reconstruction head. The visual encoding module is configured to extract a visual feature of an input text image. The language embedding module is configured to map a generated text label into a context embedding vector.

3. The method of claim 2, wherein, The mask visual language decoding module is configured to perform attention calculation on the visual feature, the initialized query feature and the context embedding vector to obtain visual attention features and query attention features. The language reconstruction head is configured to generate a next text label according to the query attention feature. The mask visual language decoding module comprises a visual self-attention layer, a query text cross-attention layer, a visual query cross-attention layer and a query visual cross-attention layer. The mask visual language decoding module performs attention calculation on the visual feature, the initialized query feature and the context embedding vector to obtain visual attention features and query attention features, which comprises: performing self-attention calculation on the visual feature by the visual self-attention layer to obtain an intermediate visual feature; 4. The method of claim 3, wherein, performing cross-attention calculation on the query feature and the context embedding vector by the query text cross-attention layer to obtain an intermediate query feature; performing cross-attention calculation on the intermediate visual feature as query input, the intermediate query feature as key input and value input by the visual query cross-attention layer to obtain visual attention features; performing cross-attention calculation on the intermediate query feature as query input, the intermediate visual feature as key input and value input by the query visual cross-attention layer to obtain query attention features. The method further comprises: receiving a text image sent by an end-side device; or 5. The method of claim 1, wherein, ​ ​ ​ An input image sent by a receiving-end device is received, text detection is performed on the input image, and a region containing text in the input image is determined; the region containing text is segmented from the input image to obtain the text image.

6. The method according to any one of claims 1-5, characterized in that, The training process of the text recognition model comprises: A visual language joint modeling model and a training set are constructed, and sample data in the training set comprises: an image sample, and a text sequence corresponding to the image sample; In a first stage, the visual language joint modeling model is trained using the training set for a visual language joint modeling task to obtain a one-stage model; In a second stage, the one-stage model is trained using the training set for an autoregressive text generation task to obtain a text recognition model.

7. The method of claim 6, wherein, The visual language joint modeling model comprises: a visual encoding module, a language embedding module, a masked visual language decoding module, a visual reconstruction head, and a language reconstruction head; The visual encoding module is configured to extract initial visual features of an input image; The language embedding module is configured to map an input text sequence into a context embedding vector; The masked visual language decoding module is configured to perform attention calculation on the initial visual features, an initialized query feature, and the context embedding vector to obtain visual attention features and query attention features; The visual reconstruction head is configured to reconstruct a covered image block in the image according to the visual attention features to obtain a visual reconstruction result; The language reconstruction head is configured to reconstruct a covered text token in the text sequence according to the query attention features to obtain a language reconstruction result, or to generate a prediction result of the text sequence according to the query attention features.

8. The method of claim 7, wherein, In the first stage, the visual language joint modeling model is trained using the training set for a visual language joint modeling task to obtain a one-stage model, which comprises: For the image sample in the training set, at least one image block in the image sample is covered, the visual encoding module is used to encode the covered image sample to obtain first visual features; At least one text token in the text sequence is covered, and the language embedding module is used to map the covered text sequence into an embedding vector to obtain a first context embedding vector; The masked visual language decoding module is used to perform attention calculation on the first visual features, an initialized first query feature, and the first context embedding vector to obtain first visual attention features and first query attention features; The visual reconstruction head is used to reconstruct the covered image block according to the first visual attention features to obtain a visual reconstruction result, and the language reconstruction head is used to reconstruct the covered text token according to the first query attention features to obtain a language reconstruction result; A reconstruction task loss is calculated according to the visual reconstruction result and the language reconstruction result; The parameters of the visual language joint modeling model are adjusted according to the reconstruction task loss to obtain the one-stage model.

9. The method of claim 8, wherein, The at least one image block in the image sample is covered, and the covered image sample is encoded by the visual coding module to obtain a first visual feature, including: cutting the image sample into a plurality of image blocks; According to the visual covering ratio, at least one image block in the image sample is covered, the covered image block is taken as an invisible image block, and the un-covered image block is taken as a visible image block; input the visible image block into the visual coding module for encoding to obtain the feature map corresponding to each visible image block, and set the feature map corresponding to the invisible image block as a learnable mask feature, and the first visual feature includes the feature map of the visible image block and the feature map of the invisible image block.

10. The method of claim 8, wherein, The at least one text token in the text sequence is covered, and the covered text sequence is mapped into an embedding vector by the language embedding module to obtain a first context embedding vector, including: According to the text covering ratio, at least one text token in the text sequence is replaced with a mask text to obtain a masked text sequence; input the masked text sequence into the language embedding module for mapping to obtain the first context embedding vector.

11. The method of claim 8, wherein, The mask visual language decoding module includes: a visual self-attention layer, a query text cross-attention layer, a visual query cross-attention layer, a query visual cross-attention layer, Through the mask visual language decoding module, attention calculation is performed on the first visual feature, the initialized first query feature and the first context embedding vector to obtain a first visual attention feature and a first query attention feature, including: Through the visual self-attention layer, self-attention calculation is performed on the first visual feature to obtain a first intermediate visual feature; Through the query text cross-attention layer, cross-attention calculation is performed on the first query feature and the first context embedding vector to obtain a first intermediate query feature; Through the visual query cross-attention layer, the first intermediate visual feature is input as the query, and the first intermediate query feature is input as the key and the value, and cross-attention calculation is performed to obtain the first visual attention feature; Through the query visual cross-attention layer, the first intermediate query feature is input as the query, and the first intermediate visual feature is input as the key and the value, and cross-attention calculation is performed to obtain the first query attention feature.

12. The method of claim 11, wherein, Further comprising: obtain the attention mask corresponding to the text sequence; Set the value corresponding to the covered text token in the attention mask to a first value as the attention mask used by the query text cross-attention layer.

13. The method of claim 11, wherein, Further comprising: obtain the attention mask corresponding to the text sequence; Shuffle the attention mask; Set the value corresponding to the covered text token in the shuffled mask matrix to a first value as the attention mask used by the query text cross-attention layer.

14. The method of claim 8, wherein, The visual reconstruction loss is calculated according to the at least one image block and the visual reconstruction result, including: According to the at least one image block and the visual reconstruction result, a visual reconstruction loss is calculated; According to the at least one text mark and the language reconstruction result, a language reconstruction loss is calculated; According to the visual reconstruction loss and the language reconstruction loss, a reconstruction task loss is determined.

15. The method of claim 7, wherein, In the second stage, using the training set, the one-stage model is trained for an autoregressive text generation task to obtain a text recognition model, including: Removing or disabling the visual reconstruction head of the one-stage model; For image samples in the training set, the image samples are encoded by the visual encoding module to obtain second visual features; The text sequence of the image sample is mapped into an embedding vector by the language embedding module to obtain a second context embedding vector; The second visual features, the initialized second query features and the second context embedding vector are subjected to attention calculation by the masked visual language decoding module to obtain second visual attention features and second query attention features; The second query attention features are used to generate a prediction result of the text sequence by the language reconstruction head; According to the prediction result of the text sequence, an autoregressive text generation task loss is calculated; According to the autoregressive text generation task loss, the one-stage model is trained to obtain a text recognition model.

16. A text recognition method characterized by, Including: In response to a call request for the text recognition model, a text image to be recognized is obtained; The text image and the starting mark of the text sequence to be predicted are input into the text recognition model for text recognition to sequentially generate text marks in a target text sequence contained in the text image; The target text sequence is returned; Wherein, the process of outputting the next text mark in the target text sequence by the text recognition model includes: The generated text mark is used as context information, and the text image and the context information are subjected to cross-modal feature extraction and fusion to obtain a fusion feature; The next text mark is output according to the fusion feature.

17. The method of claim 16, wherein, The generated text mark is used as context information, and the text image and the context information are subjected to cross-modal feature extraction and fusion to obtain a fusion feature, including: The visual features of the text image are extracted; The generated text mark is mapped into a context embedding vector; The visual features, the initialized query features and the context embedding vector are subjected to attention calculation to obtain visual attention features and query attention features, and the fusion feature includes the visual attention features and the query attention features; Correspondingly, the next text mark is output according to the query attention features. Including:

18. A server, characterized by At least one processor; And The memory is connected in communication with the at least one processor; Wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the server to perform the method of any one of claims 1-17. The computer-readable storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, the method of any one of claims 1-17 is implemented.

19. A computer-readable storage medium, characterized in that, ​ 20. A computer program product comprising a computer program, characterized in that, The computer program, which when executed by the processor, implements the method as claimed in any of claims 1-17.

Images