Self-supervised scene text recognition method based on mask next scale prediction

Abstract

The present application relates to the technical field of character recognition, and discloses a self-supervised scene character recognition method based on mask next scale prediction, comprising: acquiring an unannotated scene character image, and constructing a small-scale enhanced view, a large-scale enhanced view and a local enlarged view; inputting a shared coding network to extract image block features and global semantic vectors; predicting high-resolution features based on small-scale view features, and calculating next scale prediction loss; under the double guidance of small-scale layout features and predicted features, restoring features of an occluded area of the local enlarged view, and calculating mask reconstruction loss; aligning global semantic vectors of each view to calculate multi-scale language alignment loss; and optimizing the recognition model by combining the three losses. By coupling the global prior of next scale prediction and the local constraint of mask reconstruction, the problem of limited field of view of single scale modeling and attention dispersion is solved, semantic drift is prevented, and the accuracy and robustness of recognition are improved.

Description

Technical Field

[0001] This invention relates to the field of text recognition technology, and in particular to a self-supervised scene text recognition method based on masked next-scale prediction. Background Technology

[0002] Scene text recognition (STR) aims to transcribe text from natural scene images into computer-editable text sequences, and is a key component of document digitization, assisted reading, and visual understanding systems. Although deep learning-based methods have made significant progress, they typically rely on large amounts of labeled data.

[0003] To reduce reliance on labeled data, self-supervised learning (SSL) has become a research hotspot in recent years. Currently, mainstream SSL methods are mainly based on masked image modeling (MIM), which involves learning visual representations by reconstructing masked image patches. Existing methods typically employ a single-scale encoder structure, taking a fixed-resolution image as input, and training the network by predicting pixels or features in the masked regions, enabling it to extract image features.

[0004] The existing technology still has the following shortcomings: 1. Lack of multi-scale hierarchical structure modeling capability. Most existing methods in the field of self-supervised text recognition operate on a single spatial scale and cannot capture the inherent hierarchical characteristics of text in a scene, i.e., the evolution from global layout to word structure, and then to fine-grained character strokes.

[0005] 2. Simple Next-Scale Prediction (NSP) leads to attention divergence. Although NSP can simulate the evolution from coarse to fine, when applied directly, the encoder's attention will diverge spatially because all tokens are globally visible during the prediction process, tending to focus on irrelevant background areas rather than text structure.

[0006] 3. Limited perspective of MIM methods. While the simple MIM method forces a focus on the local, it lacks a global layout awareness, resulting in a "short-sighted" perspective and an inability to establish long-distance structural dependencies.

[0007] 4. Semantic inconsistency across scales. During multi-resolution learning, features extracted at different scales are prone to semantic drift, leading to changes in the textual meaning corresponding to the features. Summary of the Invention

[0008] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention provides a self-supervised scene text recognition method based on masked next-scale prediction.

[0009] A self-supervised scene text recognition method based on masked next-scale prediction includes: Obtain unlabeled scene text images, and construct small-scale enhanced views, large-scale enhanced views, and local magnified views based on the scene text images. Perform partial occlusion processing on the local magnified views to obtain masked magnified views. The visible regions of the small-scale enhanced view, the large-scale enhanced view, and the masked magnified view are respectively input into a shared coding network to extract the corresponding image patch feature sequences and global semantic vectors; The next-scale self-supervised prediction step is performed by upsampling the image patch feature sequence of the small-scale augmented view and inputting it into the next-scale prediction decoder to obtain the prediction features. The target features of the large-scale augmented view are extracted through the parameter-frozen teacher coding network, and the next-scale prediction loss is calculated based on the target features of the large-scale augmented view. The mask reconstruction self-supervised step is performed, and the visible region features of the magnified mask view are combined with the mask markers and input into the mask reconstruction decoder. Under the guidance of the image patch feature sequence of the small-scale enhanced view and the predicted features, the reconstructed features are output, and the mask reconstruction loss is calculated based on the target features of the local magnified view. The mask reconstruction decoder includes a first decoding block and a second decoding block in series; Guided by the image patch feature sequence of the small-scale enhanced view and the predicted features, the reconstructed features are output, including: The visible area features of the magnified mask view combined with the mask markers to form the feature sequence to be recovered are input into the first decoding block, and the image block feature sequence of the small-scale enhanced view is used as guiding information for feature interaction. The output of the first decoding block is input into the second decoding block, and the predicted features are used as guiding information to perform feature interaction to obtain the reconstructed features; Based on the global semantic vectors of the small-scale enhanced view, the large-scale enhanced view, and the mask magnification view, calculate the multi-scale language alignment loss; By combining the next-scale prediction loss, the mask reconstruction loss, and the multi-scale language alignment loss, the shared coding network is subjected to self-supervised joint optimization to obtain a recognition model for scene text recognition.

[0010] Furthermore, the construction of a small-scale enhanced view, a large-scale enhanced view, and a masked magnified view based on the scene text image includes: After normalizing the scene text image, multiple aligned scale sequences are constructed according to a preset ratio; Randomly extract adjacent first-scale and second-scale images from the multiple aligned scale sequences, and apply data augmentation to them respectively to obtain the small-scale augmented view and the large-scale augmented view. After partially magnifying and cropping the scene text image, the partially magnified view is obtained; the partially magnified view is divided according to a preset image block size, and image blocks of a preset proportion are randomly occluded to obtain the masked magnified view.

[0011] Furthermore, the aligned scale sequence includes four scales that increase by a factor of 2, and the preset occlusion area is 80%.

[0012] Furthermore, the data augmentation includes geometric perturbations and appearance perturbations; The small-scale enhanced view and the large-scale enhanced view employ different random enhancement parameters.

[0013] Further, the step of upsampling the image patch feature sequence of the small-scale enhanced view and inputting it into the next-scale prediction decoder to obtain prediction features, and calculating the next-scale prediction loss based on the target features of the large-scale enhanced view, includes: The spatial length of the image patch feature sequence of the small-scale enhanced view is upsampled to be consistent with the number of image patches in the large-scale enhanced view by bicubic interpolation to obtain intermediate features. The intermediate features are input into the next-scale prediction decoder, which outputs the predicted features. The large-scale augmented view is input into a parameter-frozen teacher coding network to extract target features; The mean square error between the predicted features and the target features of the large-scale enhanced view is calculated to obtain the next-scale prediction loss.

[0014] Further, the calculation of the mask reconstruction loss based on the target features of the magnified local view includes: The local magnified view is input into the parameter-frozen teacher coding network to extract target features; Extract the features corresponding to the occlusion location from the reconstructed features; The mean square error between the reconstructed features corresponding to the occlusion location and the target features is calculated to obtain the mask reconstruction loss.

[0015] Further, the calculation of multi-scale language alignment loss based on the global semantic vectors of the small-scale enhanced view, the large-scale enhanced view, and the mask magnification view includes: Calculate the first mean square error between the global semantic vector of the magnified mask view and the global semantic vector of the small-scale enhanced view; Calculate the second mean square error between the global semantic vector of the masked magnified view and the global semantic vector of the large-scale enhanced view; The first mean square error and the second mean square error are summed to obtain the multi-scale language alignment loss.

[0016] Furthermore, the shared coding network adopts a visual Transformer network, and both the next-scale prediction decoder and the mask reconstruction decoder adopt a network structure that includes self-attention layers and cross-attention layers.

[0017] Furthermore, the resulting recognition model is used for scene text recognition, including: The text image of the scene to be recognized is obtained and input into the optimized and trained shared coding network to extract recognition features; The identified features are input into a text decoder for transcription, and a computer-editable text sequence is output.

[0018] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: 1. By predicting the mapping from low-resolution context to high-resolution features, the encoder is forced to learn the hierarchical evolution from coarse layout to fine strokes, thereby significantly improving the model's ability to recognize and perceive extreme scale changes and blurred text; explicit cross-scale structural modeling is achieved, which improves the ability to perceive blurred and scale-changing text.

[0019] 2. It innovatively combines next-scale prediction (NSP) and masked image modeling (MIM). NSP provides global structural priors, while MIM provides local fine-grained constraints. The two complement each other, overcoming the attention diffusion phenomenon caused by NSP alone, and achieving coherent and accurate focusing of the model on the text region. A dual-paradigm coupled self-supervised framework is constructed to effectively solve the attention diffusion problem.

[0020] 3. A multi-scale language alignment (MLA) module was designed to explicitly align global semantic representations at different scales and views in the feature space and in a single forward propagation. This effectively prevents semantic drift during multi-resolution learning and improves the robustness of features. By introducing a multi-scale language alignment mechanism, the semantic consistency of cross-scale feature extraction is guaranteed.

[0021] 4. Experimental data show that it performs well in handling extreme scale changes and layout deformations, achieving state-of-the-art performance (SOTA) on the challenging Union14M benchmark and multiple standard STR datasets. Compared with existing single-scale self-supervised methods, it significantly improves the accuracy and robustness of text recognition in complex scenes.

[0022] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the framework structure of the character recognition method of the present invention; Figure 2 This is a query-based attention visualization diagram of the last encoder layer of this invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but cannot be used to limit the scope of this invention.

[0026] Figure 1 The overall structure and main modules of the self-supervised scene text recognition method based on masked next-scale prediction of the present invention are intuitively demonstrated.

[0027] The purpose of this invention is to design a unified self-supervised learning framework, namely Masked Next-Scale Prediction (MNSP). The method of this invention aims to improve the accuracy and robustness of scene text recognition by coupling the tasks of "next-scale prediction" and "masked image reconstruction," explicitly modeling the cross-scale structural evolution of text in a scene while using local constraints from mask reconstruction to correct the attention distribution. This ensures that the model can capture global layout priors while accurately focusing on local fine-grained features, thereby improving the accuracy and robustness of scene text recognition.

[0028] This invention proposes a self-supervised scene text recognition training method based on "mask-based next-scale prediction". This method is designed for text images in natural scenes. Without relying on manually labeled text content, it jointly trains the encoding network through two tasks: "feature prediction from low resolution to high resolution" and "feature reconstruction after local occlusion". This allows the network to simultaneously learn the overall layout of the text, word structure, and character stroke details, thereby obtaining a high-quality visual representation suitable for scene text recognition.

[0029] The overall process of this invention includes: multi-scale view construction, feature extraction via shared coding network, learning cross-scale structural evolution via next-scale prediction branch, learning local detail recovery via guided mask reconstruction branch, maintaining semantic consistency via multi-scale language alignment branch, and joint optimization of the three losses.

[0030] The specific implementation of the functional modules of the technical solution of this invention will be described in detail below.

[0031] (I) Construction of Input Images and Multi-Scale Sequences Let I be the input unlabeled scene text image. First, perform basic normalization on the image, scaling its dimensions to the commonly used standard size of 32×128 for text recognition while maintaining pixel value normalization. Then, construct four aligned scale sequences at a ratio of 2: .

[0032] Where adjacent scales satisfy, when hour, .

[0033] During training, predictions are not made using all scales simultaneously, but rather from adjacent scales. Randomly and uniformly select a pair This ensures that the prediction relationship "from coarser to finer scale" is covered throughout the entire training process.

[0034] The present invention also constructs a locally magnified view for mask reconstruction. This view is derived from the original image. The image was magnified to 125% of its original size and partially cropped to its original size. The resolution was 64×256, which is consistent with the largest scale in the four-scale sequence.

[0035] The purpose of magnification is to enable the network to recover finer-grained character outlines and stroke information even when the network is occluded.

[0036] (ii) Data augmentation and generation of three training views To avoid the network from "shortcut learning" by only remembering a fixed appearance, this invention generates three training views with clear sources and different functions for the same original image.

[0037] The first view is a small-scale enhanced view, meaning the image is first scaled down to... Then, random data augmentation is applied to obtain... ; The second view is a large-scale enhanced view, meaning the image is first scaled up to... Then, random data augmentation is applied to obtain... ; The third view is a magnified mask view, meaning the image is first magnified to a local size. Then, randomly occlude 80% of the image patches at the image patch level to obtain... .

[0038] The data augmentation preferably includes two types: geometric perturbation and appearance perturbation.

[0039] Geometric perturbations include at least one or more of slight rotation, perspective or affine distortion, and scale jitter, used to simulate changes in shooting angle and text layout distortion. Appearance perturbations include at least one or more of the following: brightness variation, contrast variation, blur, noise, and color perturbation, used to simulate lighting, defocusing, and imaging noise in real-world scenes.

[0040] Small-scale augmented views and large-scale augmented views use different stochastic augmentation parameters, thereby forcing the network to learn stable representations that are independent of appearance changes but related to textual semantics.

[0041] For a magnified view of the mask First, the image is divided into fixed blocks of 4×4. Then, 80% of the divided image blocks are randomly sampled and occluded.

[0042] The occluded areas are uniformly replaced with preset mask markers, while the unoccluded areas retain the original image blocks. This results in... Retaining only 20% of the visible area significantly increases the difficulty of reconstruction, forcing the network to combine global layout with local context to recover the occluded text details.

[0043] (III) Shared coding network and image patch feature representation This invention employs a small visual Transformer network as a shared encoding network, preferably a 12-layer Transformer encoder. All branches share the same online encoder to ensure that different training tasks ultimately apply to the same set of scene text representations.

[0044] Let the side length of the image patch be p=4. When the input scale is... At that time, the number of image patches at this scale is denoted as .

[0045] Where h is the height and w is the width.

[0046] Therefore, adjacent scales satisfy The encoding network outputs a corresponding image patch feature vector for each image patch, and also outputs a global semantic vector to represent the semantic meaning of the text in the entire image.

[0047] right After the three views are fed into the shared coding network, the corresponding image patch feature sequences and global semantic vectors can be obtained. For the masked magnified view, only the unoccluded image patches are fed into the coding network, and the occluded positions are added with mask markers in the subsequent reconstruction stage.

[0048] (iv) Next-scale prediction branch The purpose of the next-scale prediction branch is: given a smaller scale Based on the text features, predict adjacent larger scales. The high-resolution features that should appear are obtained by this branch. This branch works directly in the feature space without pixel-level magnification, thus enabling more stable learning of the cross-scale correspondence between "text layout - word structure - character details".

[0049] Specifically, let the shared coding network enhance the small-scale view. The output is .

[0050] After removing the global semantic category markers, only the image patch feature sequences are retained, and they are upsampled to the same level using bicubic interpolation. The corresponding length yields the intermediate features. .

[0051] Then Input the next-scale prediction decoder and output the predicted features. .

[0052] At the same time, large-scale enhanced views Input the frozen teacher coding network to obtain teacher target features. The next-scale prediction loss is defined as: .

[0053] The above loss indicates that the smaller the mean square error between the predicted features and the teacher's target features, the better the network can infer the high-resolution text structure from the low-resolution context alone.

[0054] Because high-resolution features contain richer character outlines and stroke information, in order to complete the prediction task, the network must retain enough local details of the text in the lower-scale input. Therefore, the shared encoding network is forced to learn a stronger fine representation ability.

[0055] The teacher coding network used in this paper specifically refers to the MAERec pre-trained encoder. MAERec is a previously disclosed technology (self-supervised character recognition). This invention uses the pre-trained weights obtained by training with this technology to initialize an encoder, namely the teacher coding network.

[0056] (v) Mask reconstruction branch guided by next-scale prediction While models can learn global structures across scales by relying solely on next-scale predictions, their attention is easily diverted to background regions. Therefore, this invention introduces a mask reconstruction branch guided by next-scale predictions to impose stronger text region constraints on the network for local occlusion recovery tasks.

[0057] First, zoom in on the mask view. Unoccluded image patches are fed into a shared coding network to obtain visible image patch features; then, mask markers are added to the occluded locations to form a complete feature sequence to be recovered. Then input it into the mask to rebuild the decoder.

[0058] The mask reconstruction decoder preferably comprises two cascaded decoding blocks.

[0059] The first decoding block utilizes a small-scale enhanced view. coding features As guiding information, the occluded area is given priority to obtain layout-level priors such as "text line position, word shape, and character spacing"; The second decoding block utilizes the next-scale prediction result obtained in the previous section. As guiding information, it enables the occluded area to obtain detailed prior information such as "character edges, stroke direction, and local outline".

[0060] Therefore, mask reconstruction does not simply rely on surrounding pixels for local completion, but rather completes the restoration under the dual constraints of "low-scale layout guidance + high-scale prediction guidance". This inherits the global structural information provided by the next-scale prediction, while the occlusion restoration task draws the network's attention back to the local regions that truly determine the text recognition result.

[0061] Let the output of the mask reconstruction decoder be The original unoccluded image Input the frozen teacher coding network to obtain target features .like Indicates the first The mask reconstruction loss is defined as follows: (The loss is determined by the presence or absence of an image patch.) .

[0062] The aforementioned losses are calculated only for the obscured areas, thus ensuring that online learning focuses on the core issue of "how to recover the obscured text area".

[0063] (vi) Multi-scale language alignment branch Because the appearance of the same text image varies significantly under small-scale, large-scale, and local occlusion conditions, a shared encoding network can easily learn semantically inconsistent representations on different views without additional constraints. Therefore, this invention sets up a multi-scale language alignment branch to explicitly constrain the global semantic vectors of the three views.

[0064] Specifically, small-scale enhanced views are extracted separately. Large-scale enhanced view and mask zoom view The global semantic vector, denoted as , and To maintain semantic consistency between the magnified view and the two scale views through mean squared error loss constraint masking, it is defined as follows: .

[0065] By adopting this "dual anchor point" alignment method, instead of aligning only with small scales or only with large scales, we can simultaneously take into account scale invariance and high-frequency detail sensitivity, thereby preventing semantic drift.

[0066] (vii) Joint training objectives and implementation process This invention employs an end-to-end joint training approach, with the overall loss function being: .

[0067] in, Used for constrained cross-scale structure prediction. Used to constrain local occlusion recovery These three elements work together to constrain semantic consistency across views. This enables the shared coding network to simultaneously possess global layout understanding, local detail recovery capabilities, and cross-scale semantic stability.

[0068] The preferred implementation method is as follows: The shared coding network uses a 12-layer visual Transformer small network; the image patch size is 4×4; the basic input resolution is 32×128; and the random occlusion ratio during mask reconstruction is 80%. Both the next-scale prediction decoder and the mask reconstruction decoder can adopt a two-layer "self-attention-cross-attention-feedforward network" structure, and the output feature dimension is preferably 384.

[0069] The AdamW optimizer is preferred for the pre-training phase, with an initial learning rate of 3×10⁻⁶. -4 The batch size is 512, and the training lasts for 10 cycles. The first cycle is used for learning rate warm-up, and cosine annealing is used to adjust the learning rate.

[0070] In downstream text recognition tasks, the pre-trained shared encoder network can be combined with a 6-layer Transformer decoder; the AdamW optimizer with a learning rate of 1×10⁻⁶ is preferred for fine-tuning. -4 The batch size is 512, the training duration is 10 cycles, and the maximum output character length is 25. The above implementation parameters are preferred embodiments and do not constitute a limitation on the scope of protection of this invention.

[0071] Implementation steps of the method of the present invention Step 1: Obtain unlabeled scene text images and construct four scale sequences that increase by a factor of 2, as well as a magnified mask view for local reconstruction.

[0072] Step 2: Apply random data augmentation to the small-scale view and the large-scale view, and randomly occlude 80% of the area of ​​the magnified view according to image blocks to obtain three training views.

[0073] Step 3: Input the three training views into the shared coding network to extract image patch features and global semantic vectors.

[0074] Step 4: Based on the small-scale view features, after upsampling and the next-scale prediction decoder, predict the high-resolution features of the adjacent larger scale, and use the output of the teacher coding network as supervision to calculate the next-scale prediction loss.

[0075] Step 5: Combine the visible image patch features of the magnified mask view with the mask markers to form the sequence to be restored. Under the dual guidance of small-scale layout features and next-scale prediction features, restore the features of the occluded area and calculate the mask reconstruction loss.

[0076] Step 6: Align the global semantic vectors of the small-scale view, large-scale view, and magnified mask view, and calculate the multi-scale language alignment loss.

[0077] Step 7: Sum the three losses and jointly optimize the shared encoding and decoding networks to obtain a self-supervised model for pre-training of scene text recognition.

[0078] Step 8: Use the trained model obtained in Step 7 to perform text recognition.

[0079] The inventors conducted verification experiments using the technical solution of this invention to evaluate the effectiveness of MNSP. The self-supervised model designed in this invention was pre-trained on the large-scale unlabeled dataset Union14M-U (containing 4 million samples) and fine-tuned on the labeled dataset Union14M-L.

[0080] The tests were conducted on six mainstream scene text datasets: IIIT5K contains 3000 images, mainly regular text; IC13 contains 1015 images; SVT contains 647 images; IC15 contains 1811 images, mostly low-quality and multi-directional text; SVTP contains 645 perspective transformation images; and CUTE contains 288 curved text images.

[0081] Finally, it was evaluated on the challenging Union14M benchmark.

[0082] Table 1 shows the performance comparison of the present invention with other methods on the Union14M benchmark. The results demonstrate that MNSP achieves an average accuracy of 86.2%, with significant improvements, particularly in the scale-dependent Curve and Multi-Oriented categories. Table 2 shows the performance comparison of the present invention on six standard STR datasets. The present invention achieves an average accuracy of 96.7%, outperforming several mainstream methods, demonstrating its effectiveness in handling extreme scale changes and layout deformations. Figure 2 The results of the attention visualization of the present invention are shown. Compared with NSP (attention divergence) and MIM (visual limitation), it can be found that the present invention can focus on the entire text area coherently and accurately, effectively solving the attention diffusion phenomenon of NSP.

[0083] Table 1. Performance comparison of MNSP (and NSP) and other methods on the Union14M benchmark dataset.

[0084] It should be noted that all text recognizers were trained using Union14M-L. MNSP-20ep refers to 20 rounds of pre-training. This refers to the independent NSP module method set in this invention.

[0085] Table 2 Performance comparison of MNSP and other methods on six commonly used STR benchmarks

[0086] It should be noted that the symbol † indicates the use of the revised label.

[0087] Figure 2 This is a visualization of query-based attention in the last encoder layer of this invention. The red boxes in the figure mark the query locations.

[0088] MNSP demonstrates consistent attention across the entire text region; MIM's attention is highly localized; NSP's attention is too scattered, and these patterns are consistent across both regular and irregular text images.

[0089] from Figure 2 The results clearly demonstrate that MNSP can apply effective attention to the entire text region in both irregular and regular text, improving the accuracy and precision of recognition.

[0090] This approach addresses the limitations of existing single-scale self-supervised methods in STR tasks and the attention diffusion phenomenon of pure NSP methods. The proposed MNSP framework achieves the unification of cross-scale structural modeling and fine-grained local perception through the complementary mechanism of "NSP global guidance" and "MIM local constraints". The designed multi-scale language alignment (MLA) module explicitly constrains semantic consistency in the feature space, thereby improving the robustness of features.

[0091] In summary, the technical solution of the present invention has the following innovative points and beneficial effects compared with the prior art or a combination of prior art.

[0092] 1. Explicit cross-scale structural modeling: By predicting the mapping from low-resolution context to high-resolution features, the encoder is forced to learn the hierarchical evolution from coarse layout to fine strokes, which significantly improves the model's ability to perceive scale changes and blurred text.

[0093] 2. A self-supervised framework with dual paradigm coupling is proposed: This invention innovatively combines Next-Scale Prediction (NSP) and Masked Image Modeling (MIM). NSP provides global structural priors, while MIM provides local fine-grained constraints. The two complement each other, effectively solving the attention diffusion problem caused by NSP alone, and achieving precise focusing on text regions.

[0094] 3. Multi-scale Linguistic Alignment (MLA): The MLA module is introduced to explicitly align semantic representations ([CLS] tokens) at different scales and views in a single forward propagation, ensuring semantic consistency across scales and preventing semantic drift.

[0095] 4. Excellent performance and robustness: Experiments show that the present invention achieves state-of-the-art performance (SOTA) on the Union14M benchmark and six standard STR datasets, especially under extreme scale changes and layout deformations, and exhibits stronger robustness than existing methods.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A self-supervised scene text recognition method based on masked next-scale prediction, characterized in that, include: Obtain unlabeled scene text images, and construct small-scale enhanced views, large-scale enhanced views, and local magnified views based on the scene text images. Perform partial occlusion processing on the local magnified views to obtain masked magnified views. The visible regions of the small-scale enhanced view, the large-scale enhanced view, and the masked magnified view are respectively input into a shared coding network to extract the corresponding image patch feature sequences and global semantic vectors; The next-scale self-supervised prediction step is performed by upsampling the image patch feature sequence of the small-scale augmented view and inputting it into the next-scale prediction decoder to obtain the prediction features. The target features of the large-scale augmented view are extracted through the parameter-frozen teacher coding network, and the next-scale prediction loss is calculated based on the target features of the large-scale augmented view. The mask reconstruction self-supervised step is performed, and the visible region features of the magnified mask view are combined with the mask markers and input into the mask reconstruction decoder. Under the guidance of the image patch feature sequence of the small-scale enhanced view and the predicted features, the reconstructed features are output, and the mask reconstruction loss is calculated based on the target features of the local magnified view. The mask reconstruction decoder includes a first decoding block and a second decoding block in series; Guided by the image patch feature sequence of the small-scale enhanced view and the predicted features, the reconstructed features are output, including: The visible area features of the magnified mask view combined with the mask markers to form the feature sequence to be recovered are input into the first decoding block, and the image block feature sequence of the small-scale enhanced view is used as guiding information for feature interaction. The output of the first decoding block is input into the second decoding block, and the predicted features are used as guiding information to perform feature interaction to obtain the reconstructed features; based on the global semantic vectors of the small-scale enhanced view, the large-scale enhanced view, and the mask magnification view, the multi-scale language alignment loss is calculated; By combining the next-scale prediction loss, the mask reconstruction loss, and the multi-scale language alignment loss, the shared coding network is subjected to self-supervised joint optimization to obtain a recognition model for scene text recognition.

2. The self-supervised scene text recognition method based on masked next-scale prediction according to claim 1, characterized in that, The construction of a small-scale enhanced view, a large-scale enhanced view, and a masked magnified view based on the scene text image includes: After normalizing the scene text image, multiple aligned scale sequences are constructed according to a preset ratio; Randomly extract adjacent first-scale and second-scale images from the multiple aligned scale sequences, and apply data augmentation to them respectively to obtain the small-scale augmented view and the large-scale augmented view. After partially magnifying and cropping the scene text image, the partially magnified view is obtained; the partially magnified view is divided according to a preset image block size, and image blocks of a preset proportion are randomly occluded to obtain the masked magnified view.

3. The self-supervised scene text recognition method based on masked next-scale prediction according to claim 2, characterized in that, The aligned scale sequence includes four scales that increase by a factor of 2, and the preset occlusion area is 80%.

4. The self-supervised scene text recognition method based on masked next-scale prediction according to claim 2, characterized in that, The data augmentation includes geometric perturbations and appearance perturbations; The small-scale enhanced view and the large-scale enhanced view employ different random enhancement parameters.

5. The self-supervised scene text recognition method based on masked next-scale prediction according to claim 1, characterized in that, The step of upsampling the image patch feature sequence of the small-scale enhanced view and inputting it into the next-scale prediction decoder to obtain prediction features, and calculating the next-scale prediction loss based on the target features of the large-scale enhanced view, includes: The spatial length of the image patch feature sequence of the small-scale enhanced view is upsampled to be consistent with the number of image patches in the large-scale enhanced view by bicubic interpolation to obtain intermediate features. The intermediate features are input into the next-scale prediction decoder, which outputs the predicted features. The large-scale augmented view is input into a parameter-frozen teacher coding network to extract target features; The mean square error between the predicted features and the target features of the large-scale enhanced view is calculated to obtain the next-scale prediction loss.

6. The self-supervised scene text recognition method based on masked next-scale prediction according to claim 1, characterized in that, The calculation of the mask reconstruction loss based on the target features of the magnified local view includes: The local magnified view is input into the parameter-frozen teacher coding network to extract target features; Extract the features corresponding to the occlusion location from the reconstructed features; The mean square error between the reconstructed features corresponding to the occlusion location and the target features is calculated to obtain the mask reconstruction loss.

7. The self-supervised scene text recognition method based on masked next-scale prediction according to claim 1, characterized in that, The calculation of multi-scale language alignment loss based on the global semantic vectors of the small-scale enhanced view, the large-scale enhanced view, and the mask magnification view includes: Calculate the first mean square error between the global semantic vector of the magnified mask view and the global semantic vector of the small-scale enhanced view; Calculate the second mean square error between the global semantic vector of the masked magnified view and the global semantic vector of the large-scale enhanced view; The first mean square error and the second mean square error are summed to obtain the multi-scale language alignment loss.

8. The self-supervised scene text recognition method based on masked next-scale prediction according to claim 1, characterized in that, The shared coding network adopts a visual Transformer network, and both the next-scale prediction decoder and the mask reconstruction decoder adopt a network structure that includes self-attention layers and cross-attention layers.

9. The self-supervised scene text recognition method based on masked next-scale prediction according to claim 1, characterized in that, The obtained recognition model is used for scene text recognition, including: The text image of the scene to be recognized is obtained and input into the optimized and trained shared coding network to extract recognition features; The identified features are input into a text decoder for transcription, and a computer-editable text sequence is output.

Images