A language perception-based scene text recognition pre-training method and system

Abstract

The present application relates to the technical field of scene text recognition, and provides a scene text recognition pre-training method and system based on language perception. The scene text recognition method comprises the following steps: obtaining a guide view based on an obtained input image; segmenting the input image and the guide view respectively, and inputting the segmented images into a full connection layer in sequence to obtain input image embedding representation and guide view embedding representation; performing random mask on the input image embedding representation, inputting the image patches without mask into a first encoder to obtain first visible mark features and first CLS features; inputting the guide view embedding representation into a second encoder to obtain second visible mark features and second CLS features; inserting a learnable mask mark at a corresponding mask position based on the first visible mark features, and inputting the first visible mark features after mask insertion, the first CLS features, the second visible mark features and the second CLS features into a decoder to obtain a prediction result.

Description

Technical Field

[0001] This invention relates to the field of scene text recognition technology, and in particular to a pre-training method and system for scene text recognition based on language perception. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In the era of deep learning, scene text recognition heavily relies on large-scale labeled data for training. In practical applications, constructing a labeled dataset covering multiple scenes and languages ​​typically requires millions of dollars in manual annotation costs. This strongly supervised paradigm is not only costly but also leads to a potential decrease in recognition accuracy when the model faces data distribution shifts. To alleviate these problems, researchers have designed self-supervised agent tasks that incorporate the characteristics of scene text images. For example, sequence contrast learning methods improve upon traditional global contrast learning strategies by constructing positive and negative sample pairs at the sequence element level to obtain the sequence structure features of scene text. Masked image modeling methods obtain the visual structure of scene text by reconstructing the occluded original pixels. However, the above methods have the following problems:

[0004] Existing sequence contrast learning methods primarily focus on aligning local features at the fragment or character level, lacking the ability to model the overall language structure of the text. Because they rely solely on local feature alignment, the models suffer from insufficient semantic relationships between characters when processing long text sequences. Furthermore, local features may be misaligned.

[0005] Masked image modeling methods overemphasize visual pattern reconstruction while neglecting the linguistic coherence of text. The reconstruction process mainly relies on local visual features, failing to adequately model the global context of characters within the same word.

[0006] Therefore, existing methods fail to fully consider the characteristics of scene text recognition, which requires the simultaneous use of visual and linguistic information, resulting in the inability to effectively integrate linguistic priors and a decrease in recognition accuracy. Summary of the Invention

[0007] To address the technical problems of existing technologies, such as the separation of visual and linguistic features and insufficient generalization ability, this invention provides a pre-training method and system for scene text recognition based on language perception. The method involves data augmentation of the input image to generate a guiding view, inputting the masked original image and the guiding view into the encoder for feature extraction, calculating the mean squared error loss of the CLS-labeled features of the two branches, inserting learnable mask labels at the corresponding positions of the mask branches, inputting the mask branch features and guiding view features into the decoder, constructing the reconstruction loss, and reconstructing the masked image. This invention innovatively integrates visual and linguistic information, solving the problem that existing self-supervised methods struggle to simultaneously model character structure and inter-character relationships, achieving synergistic optimization of visual and linguistic features, improving the model's adaptability to complex scenes, and further enhancing the accuracy of scene text recognition.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] The first aspect of the present invention provides a pre-training method for scene text recognition based on language awareness.

[0010] A language-aware scene text recognition pre-training method includes:

[0011] Based on the acquired input image, a guide view is obtained; the guide view has the same text content as the input image but a different visual representation.

[0012] The input image and the guide view are segmented separately to obtain several input image patches and guide view patches, which are then arranged and fed into a fully connected layer to obtain the input image embedding representation and the guide view embedding representation.

[0013] The input image embedding representation is randomly masked, and the unmasked image patch is input into the first encoder to obtain the first visible marker feature and the first CLS feature; the guided view embedding representation is input into the second encoder to obtain the second visible marker feature and the second CLS feature.

[0014] Based on the first visible marker feature, a learnable mask marker is inserted at the corresponding mask position. The first visible marker feature, the first CLS feature, the second visible marker feature, and the second CLS feature after the mask is inserted are input into the decoder to obtain the decoding result.

[0015] The first encoder, second encoder, and decoder are trained using the unmasked image patch corresponding to the input image, the guided view embedding representation, and the decoding result.

[0016] Furthermore, the method for obtaining a guide view based on the acquired input image includes: processing the input image using data augmentation techniques that perform color transformation and / or geometric transformation to obtain the guide view.

[0017] Furthermore, the first encoder and the second encoder share parameters.

[0018] Furthermore, in the decoder, the first visible marker feature after masking is combined with the learnable mask marker inserted at the corresponding position as a query vector, and the second visible marker feature is used as a key vector and a value vector.

[0019] Furthermore, during model training, alignment loss is used to align the first CLS features and the second CLS features.

[0020] Furthermore, during model training, based on the prediction results, the reconstruction loss is used to constrain the consistency between the predicted mask position and the corresponding reconstructed target.

[0021] A second aspect of the present invention provides a language-aware scene text recognition pre-training system.

[0022] A language-aware scene text recognition pre-training system includes:

[0023] An image processing module is configured to: obtain a guide view based on an acquired input image; the guide view has the same text content as the input image but a different visual representation;

[0024] The image segmentation module is configured to segment the input image and the guide view respectively, obtain several input image patches and guide view patches, and then arrange them and input them into a fully connected layer to obtain the input image embedding representation and the guide view embedding representation.

[0025] The encoder module is configured to: randomly mask the input image embedding representation and input the unmasked image patch into the first encoder to obtain the first visible marker feature and the first CLS feature; and input the guided view embedding representation into the second encoder to obtain the second visible marker feature and the second CLS feature.

[0026] The decoder module is configured to: insert learnable mask markers at corresponding mask positions based on the first visible marker features, and input the first visible marker features, the first CLS features, the second visible marker features, and the second CLS features after the mask insertion into the decoder to obtain the decoding result;

[0027] The training module is configured to train the first encoder, the second encoder, and the decoder using the unmasked image patch corresponding to the input image, the guided view embedding representation, and the decoding result.

[0028] A third aspect of the present invention provides a computer device comprising:

[0029] A processor, adapted to execute computer programs;

[0030] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the language-aware scene text recognition pre-training method described in the first aspect above.

[0031] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and to execute steps in the language-aware scene text recognition pre-training method described in the first aspect above.

[0032] The fifth aspect of the present invention provides a computer program product or computer program.

[0033] This invention provides a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in the language-aware scene text recognition pre-training method described in the first aspect above.

[0034] Compared with the prior art, the beneficial effects of the present invention are:

[0035] This invention provides a language-aware scene text recognition pre-training method and system. The method involves obtaining a guiding view based on an acquired input image; the guiding view has the same text content as the input image but different visual representations. The input image and guiding view are segmented to obtain several input image patches and guiding view patches, which are then arranged and input into a fully connected layer to obtain input image embedding representations and guiding view embedding representations. The input image embedding representations are randomly masked, and the unmasked image patches are input into a first encoder to obtain first visible marker features and first CLS features. The guiding view embedding representation is input into a second encoder to obtain second visible marker features and second CLS features. Based on the first visible marker features, learnable mask markers are inserted at corresponding mask positions. The masked first visible marker features, first CLS features, second visible marker features, and second CLS features are input into a decoder to obtain the decoding result, achieving deep collaborative optimization of visual and linguistic features. The first encoder, second encoder, and decoder are trained using the unmasked image patches corresponding to the input image, the guiding view embedding representation, and the decoding result. This invention enables the learning of universal representations encompassing both visual semantics and linguistic structure during the pre-training stage, significantly enhancing the model's generalization ability in cross-scene and cross-linguistic text recognition tasks. Experimental results demonstrate that, on standard benchmark datasets, this invention significantly improves recognition accuracy compared to existing state-of-the-art methods, possessing significant theoretical and practical value. Attached Figure Description

[0036] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0037] Figure 1 This is a comparative diagram illustrating whether or not prior art has language information assistance, as shown in the embodiments of the present invention;

[0038] Figure 2 This is a schematic diagram comparing two existing self-supervised learning methods with the self-supervised learning method described in this invention, as shown in the embodiments of this invention.

[0039] Figure 3 This is a comparative diagram showing the visualization of the attention at the corresponding query position when two existing self-supervised learning methods and the self-supervised learning method of the present invention process the same image, as illustrated in the embodiments of the present invention.

[0040] Figure 4 This is a flowchart illustrating a language-aware scene text recognition pre-training method according to an embodiment of the present invention.

[0041] Figure 5This is a schematic diagram of the overall framework of the scene text recognition pre-training method based on language awareness, as shown in an embodiment of the present invention.

[0042] Figure 6 This is a comparative diagram showing the attention visualization at the corresponding query position between the existing method and the method of the present invention as illustrated in the embodiments of the present invention.

[0043] Figure 7 This is a schematic diagram of the structure of a scene text recognition pre-training system based on language awareness, as shown in an embodiment of the present invention.

[0044] Figure 8 This is a schematic diagram of the structure of a computer device shown in an embodiment of the present invention. Detailed Implementation

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0046] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0047] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0048] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems according to various embodiments of this disclosure. It should be noted that each block in a flowchart or block diagram may represent a module, segment, or portion of code, which may include one or more executable instructions for implementing the logical functions specified in the various embodiments. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0049] Scene text images contain both visual and linguistic information, both of which are crucial for accurate recognition. Figure 1 This is a comparative diagram illustrating whether or not prior art has language information assistance, as shown in the embodiments of the present invention. Figure 1 When only visual input is available, the visual model outputs "chastmas," which does not match the visual input. However, with the assistance of linguistic information, the visual and linguistic models work together to output "christmas," which matches the visual input. This demonstrates that linguistic information can effectively supplement visual features and improve recognition performance.

[0050] Figure 2 This is a comparative diagram illustrating two existing self-supervised learning methods (sequence contrast learning and masked image modeling) and the self-supervised learning method described in this invention, as shown in the embodiments of this invention; see reference. Figure 2 Under self-supervised pre-training settings, existing methods (such as sequence contrast learning and masked image modeling) have significant limitations due to the inability to utilize language models: sequence contrast learning focuses excessively on local feature alignment, neglecting global language structure; masked image modeling relies heavily on local visual structure for reconstruction, lacking language perception capabilities. Therefore, these methods fail to effectively model the visual-language collaborative relationship, resulting in limited performance in downstream tasks.

[0051] To address the aforementioned issues, this invention provides an innovative self-supervised framework—Linguistics-aware Masked Image Modeling (LMIM). The core idea of ​​this framework is to model linguistic information by exploring the co-occurrence relationships between characters under a self-supervised setting, and then organically integrate this information into the visual feature learning process.

[0052] Figure 3 This is a comparative diagram illustrating the visualization of attention at corresponding query positions when two existing self-supervised learning methods (sequence contrast learning and masked image modeling) and the self-supervised learning method described in this invention process the same image, as shown in the embodiments of this invention. (Refer to...) Figure 3 As can be seen, the LMIM proposed in this invention can capture both character structure and inter-character relationships simultaneously, achieving a better fusion of visual and linguistic information compared to existing methods.

[0053] The language-aware scene text recognition pre-training method of the present invention will be described in detail below with reference to the accompanying drawings:

[0054] Figure 4 This is a flowchart illustrating a language-aware scene text recognition pre-training method according to an embodiment of the present invention. Figure 5This is a schematic diagram illustrating the overall framework of a language-aware scene text recognition pre-training method according to an embodiment of the present invention. (Refer to...) Figure 4 , Figure 5 The method includes:

[0055] Step 1: Based on the acquired input image, obtain the guide view; the guide view has the same text content as the input image but a different visual presentation;

[0056] Specifically, an input image X for training is obtained, with dimensions H×W×C (height, width, and channels, respectively). A guiding view X2 is obtained using data augmentation techniques such as color transformation and geometric transformation. The guiding view X2 has the same text content as the original input image X but a different visual appearance.

[0057] Step 2: Segment the input image and the guide view respectively to obtain several input image patches and guide view patches, and then arrange them and input them into a fully connected layer to obtain the input image embedding representation and the guide view embedding representation;

[0058] Specifically, to input the Transformer encoder, the input image X and guide view X2 need to be cut into 4×4 pixel image patches, resulting in several input image patches and guide view patches. All patches are arranged in raster order (image patches are arranged first by column from left to right, then by row from top to bottom) (the number of patches is L), and then connected to a fully connected layer to obtain the input image embedding representation E. X and guide view embedded representation E X2 .

[0059] Step 3: Randomly mask the input image embedding representation, and input the unmasked image patch into the first encoder to obtain the first visible marker feature and the first CLS feature; input the guided view embedding representation into the second encoder to obtain the second visible marker feature and the second CLS feature;

[0060] Specifically, the input image embedding representation E X A random mask (80% masking) is applied, and then the unmasked input image patch (L×20%) is fed into the Transformer encoder (ViT-Small) to obtain the first visible marker feature F. X and the first CLS feature F Xcls Embed the guide view in the representation E X2 The second visible marker feature F is obtained by directly inputting it into the encoder without masking (sharing parameters with the previous encoder). X2 Second CLS feature F X2cls .

[0061] In some embodiments, the first CLS feature F obtained from the two branchesXcls Second CLS feature F X2cls These all represent global features of the input image, which can be considered global language features. Because the generated guide view maintains the same text content, the language features are consistent. The CLS features of the two branches are aligned using mean squared error loss to constrain language consistency; this is the alignment loss L_{align}.

[0062] Step 4: Based on the first visible marker feature, insert a learnable mask marker at the corresponding mask position, and input the first visible marker feature, the first CLS feature, the second visible marker feature and the second CLS feature after the mask is inserted into the decoder to obtain the decoding result.

[0063] Specifically, a decoder is designed first, consisting of three parts: self-attention, cross-attention, and a feedforward network. Specifically, the first visible marker feature is first processed by the self-attention module to obtain the output. This output, along with the second visible marker feature, is then fed into the cross-attention module to obtain the final output. Finally, the output is passed through the feedforward network to obtain the final output of the decoder layer. This invention uses cross-attention technology to reconstruct a masked view guided by linguistic information provided by the guided view. It receives input from two branches: the features obtained from the masked view branch serve as the "query," and the features obtained from the guided view branch serve as the "key" and "value." The predicted value of the output feature is then determined.

[0064] Step 5: Train the first encoder, the second encoder, and the decoder using the unmasked image patch corresponding to the input image, the guided view embedding representation, and the decoding result.

[0065] To reconstruct the features of the mask location, based on the first visible marker feature F X Insert a learnable mask marker F at the corresponding mask position. mask This is then input into the decoder as a "query". The second visible marker feature F X2 As the "key" and "value", the mean squared error loss is used to constrain the consistency between the predicted mask position and the corresponding reconstructed target based on the output prediction value. This loss is L_{recon}.

[0066] In this embodiment, the model parameters are optimized by jointly optimizing the total loss function L = L_{recon} + L_{align}.

[0067] This invention generates views containing the same text content but with different visual appearances using data augmentation techniques, providing linguistic information guidance for masked image reconstruction. By aligning the CLS marker features of the two branches through mean squared error loss, the consistency of features in different visual appearance views is forced, thereby decoupling linguistic information independent of visual features. A specially designed decoder structure (self-attention layer, cross-attention layer, and feedforward neural network) is employed, organically integrating linguistic information into the reconstruction process based on a cross-attention mechanism, calculating the mean squared error loss between the predicted mask position and the reconstructed target. Specifically, in the cross-attention mechanism, the features obtained from the mask branch serve as the "query," while the features obtained from the guiding view branch serve as the "key" and "value," achieving synergistic optimization of visual and linguistic features and improving the accuracy of scene text recognition.

[0068] To comprehensively evaluate the effectiveness of this invention, a systematic experimental scheme was designed, covering two typical language scenarios: Chinese and English, to verify the model performance from multiple dimensions. The Chinese pre-training data consisted of approximately 11 million images crawled and cleaned from the web, covering most real-world scenarios. The English data used approximately 10 million images provided by Union14M-U as pre-training data. The Chinese data was fine-tuned and tested on a dataset containing four scenarios (Scene, Web, Document, and Handwriting). The English data was fine-tuned on Union14M-L and tested on seven text types (Curve, Multi-Oriented, Artistic, Contextless, Salient, Multi-Words, and General) and six commonly used benchmarks (IIIT-5K, IC13, SVT, IC15, SVTP, and CUTE) within Union14M.

[0069] The data collection and processing in this embodiment of the invention should strictly comply with the requirements of relevant laws and regulations.

[0070] Table 1 Comparison of the present invention with existing methods in Chinese benchmarks.

[0071]

[0072]

[0073] Table 2 Comparison of the present invention with existing methods based on the English Union 14M benchmark.

[0074]

[0075]

[0076] Table 3 Comparison of the present invention with existing methods on six commonly used English benchmarks.

[0077]

[0078] Table 1 shows the comparison between the model described in this invention and existing methods on Chinese benchmarks. Compared to other languages, Chinese relies more on linguistic knowledge to understand its content. Therefore, this invention shows a significant advantage on Chinese datasets, especially in the Scene and Web subsets. Table 2 shows the performance on the Union14M benchmark for English, and Table 3 shows the performance on six commonly used benchmarks. Significant improvements are achieved in most subsets, demonstrating the effectiveness of this invention in modeling visual and linguistic information. Figure 6 The attention distribution of the model trained by the method of the present invention at different query positions is shown. The visualization results show that the method of the present invention not only preserves the local character structure but also effectively models the relationship between characters.

[0079] The above combination Figure 4 , Figure 5 The language-aware scene text recognition pre-training method provided by the embodiments of the present invention has been described in detail. Next, the language-aware scene text recognition pre-training system provided by the embodiments of the present invention will be described in conjunction with the accompanying drawings.

[0080] Figure 7 This is a schematic diagram illustrating the structure of a language-aware scene text recognition pre-training system according to an embodiment of the present invention. (Refer to...) Figure 7 The system described in this invention includes:

[0081] An image processing module is configured to: obtain a guide view based on an acquired input image; the guide view has the same text content as the input image but a different visual representation;

[0082] The image segmentation module is configured to segment the input image and the guide view respectively, obtain several input image patches and guide view patches, and then arrange them and input them into a fully connected layer to obtain the input image embedding representation and the guide view embedding representation.

[0083] The encoder module is configured to: randomly mask the input image embedding representation and input the unmasked image patch into the first encoder to obtain the first visible marker feature and the first CLS feature; and input the guided view embedding representation into the second encoder to obtain the second visible marker feature and the second CLS feature.

[0084] The decoder module is configured to: insert learnable mask markers at corresponding mask positions based on the first visible marker features, and input the first visible marker features, the first CLS features, the second visible marker features, and the second CLS features after the mask insertion into the decoder to obtain the decoding result;

[0085] The training module is configured to train the first encoder, the second encoder, and the decoder using the unmasked image patch corresponding to the input image, the guided view embedding representation, and the decoding result.

[0086] In some embodiments, the image processing module is specifically configured to: process the input image using data augmentation techniques that perform color transformation and / or geometric transformation to obtain a guide view.

[0087] In some possible implementations, the first encoder and the second encoder share parameters.

[0088] In some embodiments, the decoder module is specifically configured to: combine the first visible marker feature after masking with the learnable mask marker inserted at the corresponding position as a query vector, and the second visible marker feature as a key vector and a value vector.

[0089] In some possible implementations, the system further includes a model training module configured to: during model training, align the first CLS feature and the second CLS feature using an alignment loss; and during model training, based on the prediction results, use a reconstruction loss to constrain only the consistency between the predicted mask position and the corresponding reconstructed target.

[0090] According to embodiments of the present invention, the language-aware scene text recognition pre-training system can correspond to the execution of the methods described in the embodiments of the present invention, and the above and other operations and / or functions of each module of the language-aware scene text recognition pre-training system are respectively for implementing Figure 4 , Figure 5 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0091] See Figure 8 The diagram shows the structure of a computer device, which includes a processor, a communication interface, and a computer-readable storage medium. The processor, communication interface, and computer-readable storage medium are connected via a bus or other means. The communication interface is used to receive and send data. The computer-readable storage medium can be stored in the computer device's memory. The computer-readable storage medium stores computer programs, including program instructions, and the processor executes the program instructions stored in the computer-readable storage medium. The processor (or CPU, Central Processing Unit) is the computing and control core of the computer device, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement the corresponding steps in the embodiment of the language-aware scene text recognition pre-training method.

[0092] This embodiment provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the processing system of the computer device.

[0093] Furthermore, this storage space also contains one or more instructions suitable for loading and execution by the processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM memory or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0094] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes one or more instructions stored in the computer-readable storage medium to implement the corresponding steps in the above-described embodiment of the language-aware scene text recognition pre-training method.

[0095] This embodiment provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding steps in the above-described embodiment of the language-aware scene text recognition pre-training method.

[0096] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0097] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0098] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0099] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0100] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0101] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A pre-training method for scene text recognition based on language perception, characterized in that, include: Based on the acquired input image, a guide view is obtained; The guide view has the same text content as the input image but a different visual presentation; The input image and the guide view are segmented to obtain several input image patches and guide view patches, which are then arranged and input into a fully connected layer to obtain the input image embedding representation and the guide view embedding representation. This includes: cutting the input image and the guide view into 4×4 pixel image patches, inputting the image patches into a Transformer encoder to obtain several input image patches and guide view patches, arranging all patches in raster order, and arranging the image patches in the order of first column from left to right and then row from top to bottom, and then connecting them to a fully connected layer to obtain the input image embedding representation and the guide view embedding representation. The input image embedding representation is randomly masked, and the unmasked image patch is input into the first encoder to obtain the first visible marker feature and the first CLS feature; the guide view embedding representation is input into the second encoder to obtain the second visible marker feature and the second CLS feature, including: randomly masking the input image embedding representation, inputting the unmasked input image patch into the Transformer encoder to obtain the first visible marker feature and the first CLS feature, and directly inputting the guide view embedding representation without masking into the encoder that shares parameters with the previous encoder to obtain the second visible marker feature and the second CLS feature; Based on the first visible marker feature, learnable mask markers are inserted at the corresponding mask positions. The first visible marker feature, the first CLS feature, the second visible marker feature, and the second CLS feature after mask insertion are input into the decoder to obtain the decoding result. This includes designing a decoder, which consists of three parts: self-attention, cross-attention, and a feedforward network. Specifically, the first visible marker feature is first processed by the self-attention module to obtain the output. This output, together with the second visible marker feature, is then fed into the cross-attention module to obtain the output. The output is then processed by the feedforward network to obtain the final output of the decoder layer. Cross-attention technology is used to guide the reconstruction of the mask view by utilizing the language information provided by the guided view. The inputs of two branches are received: the features obtained from the mask branch are used as queries, and the features obtained from the guided view branch are used as keys and values. The predicted values ​​of the features are output. The first encoder, second encoder, and decoder are trained using the unmasked image patch corresponding to the input image, the guided view embedding representation, and the decoding result.

2. The scene text recognition pre-training method based on language awareness according to claim 1, characterized in that, The method for obtaining a guide view based on the acquired input image includes: processing the input image using data augmentation techniques that perform color transformation and / or geometric transformation to obtain the guide view.

3. The scene text recognition pre-training method based on language perception according to claim 1, characterized in that, The first encoder and the second encoder share parameters.

4. The scene text recognition pre-training method based on language perception according to claim 1, characterized in that, The decoder combines the first visible marker feature after masking with the learnable mask marker inserted at the corresponding position as the query vector, and the second visible marker feature as the key vector and value vector.

5. The scene text recognition pre-training method based on language awareness according to claim 1, characterized in that, During model training, alignment loss is used to align the first CLS features and the second CLS features.

6. The scene text recognition pre-training method based on language awareness according to claim 1, characterized in that, During model training, based on the prediction results, the reconstruction loss is used to constrain the consistency between the predicted mask position and the corresponding reconstructed target.

7. A scene text recognition pre-training system based on language perception, characterized in that, include: The image processing module is configured to: obtain a guide view based on the acquired input image; The guide view has the same text content as the input image but a different visual presentation; The image segmentation module is configured to segment the input image and the guide view respectively, obtain several input image patches and guide view patches, and then arrange them and input them into a fully connected layer to obtain the input image embedding representation and the guide view embedding representation. The module includes: cutting the input image and the guide view into 4×4 pixel image patches, inputting the image patches into a Transformer encoder to obtain several input image patches and guide view patches, arranging all patches in raster order, and arranging the image patches in the order of first column from left to right and then row from top to bottom, and then connecting them to a fully connected layer to obtain the input image embedding representation and the guide view embedding representation. An encoder module is configured to: randomly mask the input image embedding representation and input the unmasked image patch into a first encoder to obtain a first visible marker feature and a first CLS feature; and input the guide view embedding representation into a second encoder to obtain a second visible marker feature and a second CLS feature, including: randomly masking the input image embedding representation, inputting the unmasked input image patch into a Transformer encoder to obtain the first visible marker feature and the first CLS feature, and directly inputting the guide view embedding representation without masking into an encoder that shares parameters with the previous encoder to obtain the second visible marker feature and the second CLS feature; The decoder module is configured to: insert learnable mask markers at corresponding mask positions based on the first visible marker features; input the first visible marker features, first CLS features, second visible marker features, and second CLS features after mask insertion into the decoder to obtain the decoding result; including: designing the decoder, which consists of three parts: self-attention, cross-attention, and feedforward network; specifically: the first visible marker features are first processed by the self-attention module to obtain the output; the output and the second visible marker features are then fed into the cross-attention module to obtain the output; and the output is then processed by the feedforward network to obtain the final output of the decoder layer; cross-attention technology is used to guide the reconstruction of the mask view by using language information provided by the guiding view; the input of two branches is received, the features obtained by the mask branch are used as queries, and the features obtained by the guiding view branch are used as keys and values; the predicted value of the features is output. The training module is configured to train the first encoder, the second encoder, and the decoder using the unmasked image patch corresponding to the input image, the guided view embedding representation, and the decoding result.

8. A computer device, characterized in that, A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the steps of the language-aware scene text recognition pre-training method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and execute the steps of the language-aware scene text recognition pre-training method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps in the language-aware scene text recognition pre-training method as described in any one of claims 1-6.

Images