Scene text recognition method based on multi-modal vision-language fusion

Abstract

The invention relates to a scene text recognition method based on multi-modal vision-language fusion, and the method comprises the steps: constructing a scene text recognition network which comprises a vision encoder, a language model adapter, and a language-vision cross enhancement decoder; the language-vision cross enhancement decoder comprises a feature fusion module and a full connection layer; obtaining a model training data set and a candidate character set, and training the scene text recognition network based on the model training data set and the candidate character set to obtain a trained scene text recognition network; samples in the model training data set are scene text images; and inputting the scene text image to be recognized and the candidate character set into the trained scene text recognition network to obtain a text recognition result. According to the method and the device, the recognition precision of complex text scenes is improved, and particularly, higher recognition accuracy is shown in similar character, shielding and low-resolution image scenes.

Description

Technical Field

[0001] This application relates to the field of computer vision, and more specifically, to a scene text recognition method based on multimodal vision-language fusion. Background Technology

[0002] Existing scene text recognition methods, such as TESTR, primarily rely on visual features for character prediction. However, these methods have limitations in several aspects. First, the character decoder classifies based solely on image features, lacking utilization of linguistic context (such as syntax and semantics). This leads to confusion when dealing with visually similar characters, such as "r" and "n", or "b" and "d". Because these characters are visually very similar, it is often difficult to accurately distinguish them based solely on image features, thus affecting the overall recognition accuracy. Furthermore, existing methods also lack robustness. In real-world scenarios, text images are often affected by factors such as occlusion, blurring, or low resolution, which degrade image quality, making it difficult for purely visual models to accurately extract character features. When the model makes predictions on these low-quality images, errors may accumulate, ultimately leading to a significant drop in the accuracy of the recognition results. Summary of the Invention

[0003] To overcome at least one deficiency in the prior art, this application provides a scene text recognition method based on multimodal vision-language fusion.

[0004] Firstly, a scene text recognition method based on multimodal vision-language fusion is provided, including: A scene text recognition network is constructed, which includes a visual encoder, a language model adapter, and a language-visual cross-enhanced decoder; the language-visual cross-enhanced decoder includes a feature fusion module and a fully connected layer. Obtain the model training dataset and candidate character set, and train the scene text recognition network based on the model training dataset and candidate character set to obtain the trained scene text recognition network; the samples in the model training dataset are scene text images; During training, samples are input to the visual encoder to extract visual features; candidate character sets are input to the language model adapter to obtain language embeddings; visual features and language embeddings are input to the feature fusion module, aligned based on a dynamic cross-attention mechanism, and feature fusion is performed to obtain fused features; the fused features are passed through a fully connected layer to output the character probability at each spatial location in the sample; The text image of the scene to be recognized and the candidate character set are input into the trained scene text recognition network to obtain the text recognition result, which is the character probability at each spatial location in the text image of the scene to be recognized.

[0005] In one embodiment, the feature fusion module is used to implement the following functions: The visual features are convolved and 2D sinusoidal positional encoding is added to obtain the processed visual features; The processed visual features undergo a linear transformation to generate a query vector; Language embeddings are processed through linear variables to generate key vectors; Language embeddings are value vectors; The query vector, key vector, and value vector are input into a dynamic cross-attention mechanism to obtain weighted language features. Visual features and weighted linguistic features are concatenated to obtain concatenated features; The concatenated features are input into two fully connected layers and then passed through a Sigmoid function to obtain two gated weights. The fusion result is obtained by fusing visual features and weighted language features based on two gated weights; The spliced ​​features are projected through learnable weights and added to the fusion result to obtain the fused features.

[0006] In one embodiment, the loss function used during training is:

[0007]

[0008] in, For loss function, For CTC losses, For cross-entropy loss, These are the weighting coefficients; The attenuation rate, To decay the initial training cycle, This is the current training cycle.

[0009] In one embodiment, the language model adapter uses the BERT model.

[0010] Secondly, a scene text recognition device based on multimodal vision-language fusion is provided, comprising: The network construction module is used to build a scene text recognition network, which includes a visual encoder, a language model adapter, and a language-visual cross-enhancement decoder; the language-visual cross-enhancement decoder includes a feature fusion module and a fully connected layer. The training module is used to acquire the model training dataset and candidate character set, and to train the scene text recognition network based on the model training dataset and candidate character set to obtain the trained scene text recognition network; the samples in the model training dataset are scene text images; During training, samples are input to the visual encoder to extract visual features; candidate character sets are input to the language model adapter to obtain language embeddings; visual features and language embeddings are input to the feature fusion module, aligned based on a dynamic cross-attention mechanism, and feature fusion is performed to obtain fused features; the fused features are passed through a fully connected layer to output the character probability at each spatial location in the sample; The recognition module is used to input the scene text image to be recognized and the candidate character set into the trained scene text recognition network to obtain the text recognition result, which is the character probability at each spatial position in the scene text image to be recognized.

[0011] Compared with the prior art, this application has the following beneficial effects: This application deeply integrates semantic priors with visual features, which solves the problem of the separation of language and visual information in the prior art. Especially when processing complex and ambiguous text, it achieves character-level semantic alignment through a dynamic cross-attention mechanism, which significantly improves the robustness of text recognition and improves the recognition accuracy in complex text scenes. In particular, it shows higher recognition accuracy in scenes with similar-shaped characters, occlusion and low-resolution images. Attached Figure Description

[0012] This application can be better understood by referring to the description given below in conjunction with the accompanying drawings, which, together with the detailed description below, are incorporated in and form part of this specification. In the drawings: Figure 1 A flowchart of a scene text recognition method based on multimodal vision-language fusion is shown. Figure 2 A schematic diagram of a scene text recognition network is shown. Detailed Implementation

[0013] Exemplary embodiments of the present application will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of the actual embodiments are described in the specification. However, it should be understood that many embodiment-specific decisions can be made in the development of any such actual embodiment to achieve the developer’s specific objectives, and these decisions may vary as the embodiments differ.

[0014] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the device structure closely related to the solution according to this application is shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0015] It should be understood that this application is not limited to the described embodiments by virtue of the following description with reference to the accompanying drawings. In this document, embodiments may be combined with each other, features may be substituted or borrowed between different embodiments, and one or more features may be omitted in one embodiment, where feasible.

[0016] This application provides a scene text recognition method based on multimodal vision-language fusion. Figure 1 A flowchart of a scene text recognition method based on multimodal vision-language fusion is shown. (See attached image) Figure 1 The method mainly includes the following steps: Step S1: Construct a scene text recognition network. The scene text recognition network includes a visual encoder, a language model adapter, and a language-visual cross-enhancement decoder. The language-visual cross-enhancement decoder includes a feature fusion module and a fully connected layer.

[0017] Figure 2 A schematic diagram of a scene text recognition network is shown.

[0018] Step S2: Obtain the model training dataset and candidate character set. Train the scene text recognition network based on the model training dataset and candidate character set to obtain the trained scene text recognition network. The samples in the model training dataset are scene text images.

[0019] The training set includes the SynthText dataset and the ICDAR2017 MLT dataset, while the test set includes the Total-Text dataset and the ICDAR2015 dataset. The SynthText dataset is a synthetic dataset for scene text detection and recognition, using 800,000 synthetic images covering 10 fonts, 5 background complexities, and text lengths ranging from 5 to 25 characters. These text instances encompass a wide variety of fonts, colors, sizes, and orientations, providing rich training data for scene text recognition tasks. This dataset is generated by rendering text onto randomly selected natural images from the COCO dataset, aiming to provide a large amount of labeled data for scene text recognition tasks.

[0020] The ICDAR2017 MLT dataset contains text instances with various arrangements, including horizontal, vertical, curved, and rotated text, and includes nine languages ​​(Chinese, English, Japanese, Korean, etc.). The training set contains 7200 images, and the test set contains 1800. These diverse text arrangements simulate complex text environments in the real world. Each text instance is accompanied by precise bounding box annotations, which were performed by professionals to ensure accuracy and consistency.

[0021] The Total-Text dataset is a publicly available dataset for scene text detection and recognition. It consists of images from various domains, including billboards, signs, road signs, and posters, aiming to provide a comprehensive challenge for scene text detection and recognition. Text instances in the Total-Text dataset are highly challenging because they can be affected by factors such as lighting variations, occlusion, blurring, font variations, and complex backgrounds. These factors make scene text detection and recognition tasks more difficult.

[0022] The ICDAR 2015 dataset is a public dataset for scene text detection and recognition, released by the International Conference on Document Analysis and Recognition (ICDAR). This dataset aims to provide a comprehensive challenge for scene text detection and recognition, encompassing various text types and diverse scenes.

[0023] Meanwhile, based on the character semantic prior of the pre-trained language model, a vocabulary containing all possible characters is constructed. The vocabulary is generated by the sub-word segmentation algorithm of the language model and embedded with character-level semantic features to obtain a candidate character set.

[0024] During training, samples are input to the visual encoder to extract visual features; candidate character sets are input to the language model adapter to obtain language embeddings; visual features and language embeddings are input to the feature fusion module, aligned based on a dynamic cross-attention mechanism, and feature fusion is performed to obtain fused features; the fused features are passed through a fully connected layer to output the character probability at each spatial location in the sample.

[0025] Specifically, the visual encoder uses ResNet-50 to extract multi-scale features from the image. The language model adapter uses the BERT model to convert the semantic knowledge of the pre-trained language model into character-level embeddings, thus obtaining the language embeddings.

[0026] Step S3: Input the scene text image to be recognized and the candidate character set into the trained scene text recognition network to obtain the text recognition result, which is the character probability at each spatial location in the scene text image to be recognized.

[0027] This embodiment deeply integrates semantic priors with visual features, solving the problem of the separation of language and visual information in the prior art. Especially when processing complex and ambiguous text, it achieves character-level semantic alignment through a dynamic cross-attention mechanism, which significantly improves the robustness of text recognition and the recognition accuracy in complex text scenes. In particular, it shows higher recognition accuracy in scenes with similar-looking characters, occlusion, and low-resolution images.

[0028] In one embodiment, the feature fusion module is used to implement the following functions: The visual features are convolved and then 2D sinusoidal positional encoding is added to obtain the processed visual features; here, the visual features are mapped to the same dimension as the language embedding through 1×1 convolution. Then, 2D sinusoidal positional encoding is added to preserve spatial location information.

[0029] Two-dimensional position encoding is defined as:

[0030] in, The pixel representing the spatial location (h, w) has an encoding dimension of d, where D represents the total encoding dimension.

[0031] Two-dimensional position coding employs piecewise nonlinear frequency scaling to enhance spatial awareness, in which... Used to constrain magnitude, cross terms Enhance the correlation of local locations and improve the modeling ability of curved or rotated text.

[0032] The processed visual features undergo a linear transformation to generate a query vector Q; Language embeddings are processed through linear variables to generate a key vector K; Language embeddings are represented as value vectors V; The query vector Q, key vector K, and value vector V are input into a dynamic cross-attention mechanism to obtain weighted language features. ; visual features and weighted language features The pieces are then stitched together to obtain the stitching features; The concatenated features are input into two fully connected layers and then passed through a sigmoid function to obtain two gated weights. , :

[0033] in, This represents the Sigmoid function. and These are the weights and biases of the first fully connected layer, respectively. and These are the weights and biases of the second fully connected layer, respectively. This indicates splicing characteristics.

[0034] Based on two gating weights , By fusing visual features and weighted linguistic features, a fusion result is obtained. :

[0035] in, This indicates element-wise multiplication.

[0036] The spliced ​​features are projected through learnable weights and added to the fusion result to obtain the fused features.

[0037] Finally, the fused features are mapped to the character category probability distribution through a fully connected layer, and the character probability at each spatial location is output.

[0038] Furthermore, the total loss is calculated based on a hybrid loss function, which is the sum of the dynamically weighted CTC loss and the cross-entropy loss. The parameters of the visual encoder, language adapter, and language-visual cross-enhancement decoder (LVCD) modules are jointly optimized through backpropagation. During training, the loss function used is:

[0039]

[0040] in, For loss function, For CTC losses, For cross-entropy loss, These are the weighting coefficients; The attenuation rate, , To decay the initial training cycle, , This is the current training cycle.

[0041] When epoch≤τ =0.7; When epoch>τ, for each additional epoch, reduce The lower limit is 0.3.

[0042] This application uses a lightweight language model, employing a pruned or distilled BERT-base to reduce computational overhead. On the curved text dataset Total-Text, word-level accuracy is improved by 4.2% (from 82.1% to 86.3%).

[0043] In occluded text scenarios, the character error rate (CER) decreased by 6.8% (from 12.5% ​​to 5.7%), and on datasets such as Total-Text and CTW1500, the error rate for similar-looking characters decreased by 15% to 20%.

[0044] Employing the same inventive concept as the scene text recognition method based on multimodal vision-language fusion, this embodiment also provides a corresponding scene text recognition device based on multimodal vision-language fusion, comprising: The network construction module is used to build a scene text recognition network, which includes a visual encoder, a language model adapter, and a language-visual cross-enhancement decoder; the language-visual cross-enhancement decoder includes a feature fusion module and a fully connected layer. The training module is used to acquire the model training dataset and candidate character set, and to train the scene text recognition network based on the model training dataset and candidate character set to obtain the trained scene text recognition network; the samples in the model training dataset are scene text images; During training, samples are input to the visual encoder to extract visual features; candidate character sets are input to the language model adapter to obtain language embeddings; visual features and language embeddings are input to the feature fusion module, aligned based on a dynamic cross-attention mechanism, and feature fusion is performed to obtain fused features; the fused features are passed through a fully connected layer to output the character probability at each spatial location in the sample; The recognition module is used to input the scene text image to be recognized and the candidate character set into the trained scene text recognition network to obtain the text recognition result, which is the character probability at each spatial position in the scene text image to be recognized.

[0045] The scene text recognition device based on multimodal vision-language fusion in this embodiment has the same inventive concept as the scene text recognition method based on multimodal vision-language fusion described above. Therefore, the specific implementation of this device can be found in the embodiment section of the scene text recognition method based on multimodal vision-language fusion described above, and its technical effects correspond to the technical effects of the above method, so they will not be repeated here.

[0046] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A scene text recognition method based on multimodal vision-language fusion, characterized in that, include: A scene text recognition network is constructed, which includes a visual encoder, a language model adapter, and a language-visual cross-enhanced decoder. The language-visual cross-enhanced decoder includes a feature fusion module and a fully connected layer; Obtain the model training dataset and the candidate character set, and train the scene text recognition network based on the model training dataset and the candidate character set to obtain the trained scene text recognition network; The samples in the model training dataset are scene text images; During training, the samples are input into the visual encoder to extract visual features; The candidate character set is input into the language model adapter to obtain language embeddings; the visual features and the language embeddings are input into the feature fusion module, aligned based on a dynamic cross-attention mechanism, and then fused to obtain fused features; the fused features are passed through a fully connected layer to output the character probability at each spatial location in the sample; The scene text image to be recognized and the candidate character set are input into the trained scene text recognition network to obtain the text recognition result, which is the character probability at each spatial position in the scene text image to be recognized.

2. The method as described in claim 1, characterized in that, The feature fusion module is used to implement the following functions: The visual features are convolved and 2D sinusoidal positional encoding is added to obtain the processed visual features; The processed visual features undergo a linear transformation to generate a query vector; The language embedding is processed through linear variables to generate a key vector; The language embedding is used as a value vector; The query vector, the key vector, and the value vector are input into a dynamic cross-attention mechanism to obtain weighted language features; The visual features and the weighted language features are concatenated to obtain the concatenated features; The spliced ​​features are respectively input into two fully connected layers and passed through the Sigmoid function to obtain two gated weights; The visual features and the weighted language features are fused based on the two gating weights to obtain the fusion result; The splicing features are projected using learnable weights and added to the fusion result to obtain the fused features.

3. The method as described in claim 1, characterized in that, The loss function used during training is: in, For loss function, For CTC losses, For cross-entropy loss, These are the weighting coefficients; The attenuation rate, To decay the initial training cycle, This is the current training cycle.

4. The method as described in claim 1, characterized in that, The language model adapter uses the BERT model.

5. A scene text recognition device based on multimodal vision-language fusion, characterized in that, include: A network construction module is used to construct a scene text recognition network, which includes a visual encoder, a language model adapter, and a language-visual cross-enhanced decoder. The language-visual cross-enhanced decoder includes a feature fusion module and a fully connected layer; The training module is used to acquire the model training dataset and the candidate character set, and to train the scene text recognition network based on the model training dataset and the candidate character set to obtain the trained scene text recognition network. The samples in the model training dataset are scene text images; During training, the samples are input into the visual encoder to extract visual features; The candidate character set is input into the language model adapter to obtain language embeddings; the visual features and the language embeddings are input into the feature fusion module, aligned based on a dynamic cross-attention mechanism, and then fused to obtain fused features; the fused features are passed through a fully connected layer to output the character probability at each spatial location in the sample; The recognition module is used to input the scene text image to be recognized and the candidate character set into the trained scene text recognition network to obtain the text recognition result, which is the character probability at each spatial position in the scene text image to be recognized.

Images