Scene text recognition method and system based on multi-granularity training of global vector

Abstract

The present invention discloses a scene text recognition method and system based on multi-granularity training of global vectors, and relates to the technical field of scene text recognition. The method comprises the steps of: obtaining an image to be recognized, cropping the image to be recognized to obtain a part containing text; performing scene recognition on the part containing text using a scene text recognition model to obtain a recognition result, wherein the scene text recognition model is trained in stages, and the steps include: preprocessing the training samples, encoding the preprocessed samples using an encoder, extracting a global vector containing multi-granularity information, decoding the global vector using a decoder, and obtaining a recognition result. The present invention can better utilize information of different granularities of an image, fully learn the information of the image from easy to difficult, and can also achieve better results for complex texts and curved texts.

Description

Technical Field

[0001] The present invention relates to the technical field of scene text recognition, and in particular to a scene text recognition method and system based on multi-granularity training of global vectors. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Scene Text Recognition is an important task in computer vision, which aims to read the text and content of a given cropped scene image. These words usually appear in daily life scenes such as road signs, billboards, store signs, vehicles, and product packaging. Because it provides rich semantic information, it is widely used in autonomous driving, visual question answering, augmented reality and other fields.

[0004] Scene text can be used to help vehicles recognize text information in road signs and traffic signs to assist driving decisions. Real-time translation applications can be realized by recognizing and translating text in natural scenes, such as language translation tools during travel. Warehouse management processes can also be optimized by recognizing text on product packaging. In document processing, scene text recognition can extract valuable information from document scans containing images to support subsequent data analysis.

[0005] Currently, scene text recognition technology faces many challenges. Compared with printed text, scene text is highly diverse in font, size, color, and language, and usually lacks standardized specifications. In addition, complex backgrounds such as irregular textures, color changes, shadows, and lighting differences increase the difficulty of text area detection and recognition. Changes in shooting angles often cause text perspective deformation, tilt, or bending, further increasing the complexity of recognition. In some cases, text may be difficult to recognize due to partial occlusion, blur, or out of focus, especially in low-resolution images, where recognition accuracy will drop significantly.

[0006] Early scene text recognition methods mainly relied on manually designed features, such as edge detection and shape recognition. These methods have achieved certain results in simple scenes, but their performance is limited in complex scenes. In recent years, deep learning technologies such as convolutional neural networks (CNNs) and sequence models (such as RNNs and LSTMs) have been widely used, significantly improving the performance of scene text recognition. End-to-end methods combined with attention mechanisms have become mainstream. They can handle text of variable length and have better robustness to complex backgrounds and text deformations.

[0007] The modern scene text detection and recognition process usually includes: firstly, detecting the text area in the image through the target detection algorithm (such as Faster R-CNN or EAST), then cropping the area and recognizing the character sequence through the character recognition model. Commonly used recognition models include convolutional recurrent neural network (CRNN) and Transformer model based on Attention mechanism.

[0008] Most deep learning methods treat scene text recognition as a sequence tagging or sequence transcription problem. With the development of deep learning, many deep learning methods including CTC-based, Attention-based, and Transformer-based have achieved good performance in scene text recognition, but they will produce attention drift, which will greatly damage the performance and cause serious accuracy loss. In contrast, CTC-based methods usually assign a unique output tag to each expected character and integrate a special "blank" tag to solve the alignment problem. However, this method often requires a lot of post-processing to manage blank and repeated characters.

[0009] At present, in addition to the alignment problem between visual markers and character embeddings, traditional scene text recognition methods are not comprehensive in extracting and mining information in images, cannot perform targeted extraction of multi-granular information, cannot improve recognition efficiency, and have poor adaptability to complex tasks. Summary of the invention

[0010] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a scene text recognition method and system based on multi-granularity training of global vectors, which avoids the alignment problem between visual markers and character embeddings that is common in traditional methods, can better utilize the information of different granularities of images, and fully learn the information of images from easy to difficult, so as to quickly and accurately recognize cropped scene images, and can also achieve better results for complex text and curved text.

[0011] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0012] A first aspect of the present invention provides a scene text recognition method based on multi-granularity training of a global vector, comprising the following steps:

[0013] Obtain an image to be recognized, and crop the image to be recognized to obtain a portion containing text;

[0014] A scene text recognition model is used to perform scene recognition on a part containing text to obtain a recognition result, wherein the scene text recognition model is trained in stages, and the steps include: preprocessing the training samples, encoding the preprocessed samples by an encoder, extracting a global vector containing multi-granularity information, and decoding the global vector by a decoder to obtain a recognition result; using the error voting method to divide the training samples into five difficulty levels, training from easy to difficult, and gradually adding more difficult data to the training data to obtain better generalization ability, and using a cosine annealing heating restart learning rate scheduling strategy to adapt the staged training process.

[0015] Furthermore, the specific steps of preprocessing the training samples are: geometric transformation, random shearing and random noise addition to the images of the training samples.

[0016] Furthermore, the specific steps of extracting the global vector containing multi-granularity information are as follows:

[0017] The Vision Transformer module is used to extract multi-granularity features from the image to obtain multi-granularity features;

[0018] A global average pooling layer is used to integrate the extracted multi-granular features with the fine-grained and global features of the image into a single vector as the global vector;

[0019] The global vector is fused with the context information for decoder processing.

[0020] Furthermore, the specific steps of fusing the global vector with the context information are:

[0021] Utilize the global channel-aware context module to extract context information and context location information;

[0022] The feature fusion module is used to fuse the context information, context location information and the corresponding global vector.

[0023] Furthermore, the specific steps of using the Vision Transformer module to extract multi-granularity features from images are as follows:

[0024] Divide the image into small blocks of different sizes, perform convolution operations on the small blocks and combine them with position encoding;

[0025] During the convolution process, the extraction process of each granularity information is independent, and after the extraction is completed, it passes through a transformer-based encoder.

[0026] Furthermore, the decoder adopts a language decoder, which superimposes a two-layer masked multi-head self-attention mechanism and a feedforward network.

[0027] Furthermore, the decoder decodes the global vector in a step-by-step decoding manner to gradually capture different levels of features and information at different levels.

[0028] Furthermore, the decoder introduces a random dropout mechanism, which randomly drops some neurons during training, so that the scene text recognition model uses different subsets for calculation in each forward propagation.

[0029] Furthermore, during the training of the scene text recognition model, cross entropy loss and L1 loss are combined as loss functions.

[0030] A second aspect of the present invention provides a scene text recognition system based on multi-granularity training of a global vector, comprising:

[0031] A data acquisition module is configured to acquire an image to be recognized and crop the image to be recognized to obtain a portion containing text;

[0032] A scene recognition module is configured to perform scene recognition on a portion containing text using a scene text recognition model to obtain a recognition result;

[0033] Among them, the scene recognition module includes a model training module, which is configured to train the scene text recognition model in stages, and the steps include: preprocessing the training samples, encoding the preprocessed samples with an encoder, extracting a global vector containing multi-granularity information, and decoding the global vector with a decoder to obtain a recognition result; using the error voting method to divide the training samples into five difficulty levels, training from easy to difficult, and gradually adding more difficult data to the training data to obtain better generalization ability, and using a cosine annealing heating restart learning rate scheduling strategy to adapt to the staged training process.

[0034] One or more of the above technical solutions have the following beneficial effects:

[0035] The present invention discloses a scene text recognition method and system based on multi-granularity training of global vectors. Character decoding is achieved by decoding using global tags and introducing sequence information through a global channel-aware context module combined with position embedding, thereby avoiding the alignment problem between visual tags and character embeddings that is prevalent in traditional methods. The method can quickly and accurately recognize cropped scene images, and can also achieve good results for complex text and curved text.

[0036] The present invention innovatively designs a separated multi-granularity Vision Transformer module, which can extract richer information of different granularities from images. In this module, the extraction and combination processes of information of different granularities are separated to ensure that each layer can focus on information of a specific scale, thereby improving the recognition ability of complex backgrounds, text deformation and irregular image areas.

[0037] The present invention innovatively designs and uses a global channel-aware context module, which can diffuse the information in the global vector to different positions by calculating the weights of different channels.

[0038] By superimposing the self-attention module, the model can gradually capture features and information at different levels at different levels, thereby enhancing the expressive power of the input features layer by layer.

[0039] The present invention innovatively introduces L1 loss into the text recognition task and combines it with cross entropy loss to weaken the weight of contextual semantics and increase the influence of the image information itself.

[0040] The present invention classifies the training data according to its difficulty and uses a phased training method to help the model better generalize and adapt to complex tasks.

[0041] The present invention performs parallel training during training and performs step-by-step decoding during testing, thereby realizing autoregressive scene text recognition and accelerating the training speed.

[0042] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0044] Figure 1 This is a flow chart of a scene text recognition method based on multi-granularity training of global vectors in Embodiment 1 of the present invention;

[0045] Figure 2 This is a structural diagram of a scene text recognition model in Embodiment 1 of the present invention;

[0046] Figure 3 4 is a structural diagram of a separated multi-granularity Vision Transformer module in Embodiment 1 of the present invention;

[0047] Figure 4 1 is a structural diagram of a Transformer encoder in Embodiment 1 of the present invention;

[0048] Figure 5 4 is a structural diagram of a global channel-aware context module in Embodiment 1 of the present invention;

[0049] Figure 6 This is a structural diagram of a language decoder in Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0050] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0051] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "include" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or their combinations;

[0052] Embodiment 1:

[0053] Embodiment 1 of the present invention provides a scene text recognition method based on multi-granularity training of global vectors, which specifically includes the following steps:

[0054] Step 1: Get the image to be recognized, and crop the image to be recognized to obtain the part containing the text.

[0055] Step 2: Use the scene text recognition model to perform scene recognition on the part containing the text and obtain the recognition result. Figure 1 As shown, the text part contained in the picture is first obtained, the cropped image is preprocessed to obtain more diverse data, and then the preprocessed picture is input into the encoder to obtain the global vector containing multi-granularity information corresponding to the picture. The decoder decodes the global vector to obtain the recognition result.

[0056] Among them, the present invention innovatively uses a staged training method to train the scene text recognition model in stages. By inputting the training data into the model in stages according to difficulty, the learning difficulty of the model is gradually increased, imitating the human learning process.

[0057] Specifically, it gradually increases the difficulty of tasks from simple to complex, thereby helping the model to better generalize and adapt to complex tasks.

[0058] The present invention uses the method of error voting to divide the training samples into five difficulty levels, conducts training from easy to difficult, and gradually adds more difficult data to the training data, thereby obtaining better generalization ability.

[0059] The specific process of the error voting method is: use multiple models to infer the data, and use the voting method to classify the data instances according to the number of errors generated by each data instance in the model reasoning process, and divide them into five categories from easy to difficult.

[0060] To adapt to the staged training, the present invention adopts the learning rate scheduling strategy of cosine annealing heating restart, as shown in the following formula: .

[0061] in, For the time step The learning rate at each moment, is the minimum learning rate set, is the maximum learning rate set, is the time step in the current restart cycle, is the length of the current restart cycle. In each restart cycle, the maximum learning rate and the minimum learning rate are reduced. Through the learning rate scheduling strategy of cosine annealing heating restart, through the dynamic adjustment of the learning rate, especially the periodic restart of the learning rate, the model can jump out of the local optimal point in the later stage of training and explore new solutions. A learning rate that is too low may cause the model to fall into the local optimal point. By restarting the learning rate, the model can continue to explore the parameter space, which helps to find a better global optimal point. Cosine annealing can quickly reduce the learning rate in the early stage of model training, speed up the convergence of the model, and prevent the model from staying in a bad local optimal point through restarting.

[0062] In order to facilitate autoregressive decoding and speed up training, the present invention separates the training and reasoning processes of the model. During the training process, the real label is used as context information after passing through the embedding layer, and feature fusion is performed with the embedding of the position information and the output through the feature fusion module, so that the model can better combine context information and image information. In the language decoder, each mask multi-head self-attention module gradually generates outputs inside, and outputs all time steps at one time. In the reasoning stage, the present invention uses step-by-step decoding, and uses the output of each time step model as context information after passing through the embedding layer. At each time step, it needs to pass through the complete language decoder and output head part, and the recognition result of the current time step can be obtained before the reasoning of the next time step can be performed.

[0063] The specific training steps are as follows: Figure 2 As shown, including:

[0064] Step 2.1: Preprocess the training samples.

[0065] Step 2.1.1: Use the image augmentation module to perform geometric transformations on the images of training samples, perform random cropping, and randomly add noise, etc., to improve the generalization ability of the model and increase sample diversity. At the same time, the labels of the training set are mapped to corresponding codes using a dictionary.

[0066] In a specific implementation, since images in actual application scenarios may be tilted, blurred, offset, or have different colors, in order to improve the generalization ability of the model, increase sample diversity, and reduce the occurrence of model overfitting, the present invention introduces an image augmentation module to effectively improve the performance of the recognition model. Combined with the actual situation of scene text images, the image augmentation module uses a series of operations to retain both the overall structure of the characters and the local feature information. In the following transformation, , is the original coordinate of the pixel point, , The transformed coordinates are:

[0067] 1. Geometric transformation: Considering the scene text image, random rotation angle, random scaling, random rotation, and random distortion. Since the image in the actual application scenario may have the phenomenon of character tilt or overall text tilt, rotating the original image by a certain angle with the center of the image as the rotation point can simulate this tilt phenomenon. The formula is as follows, where is the rotation angle, is the coordinate of the center point of the image.

[0068] ,

[0069] .

[0070] 2. Random scaling: Considering that the sizes of scene text images may be different, random scaling is used to change the input images into different sizes. The formula is as follows: For the scaling ratio.

[0071] ,

[0072] .

[0073] 3. Random shearing: An affine transformation that changes the shape of an image by stretching or compressing certain parts of the image horizontally or vertically. It does not change the parallel line relationship of the image, but it changes the angle and proportion in the image, causing the image content to tilt. The formula is as follows: , Indicates the image shear angle along the x-axis and y-axis.

[0074] ,

[0075] .

[0076] 4. Random distortion: Random distortion is a more complex transformation of the image, usually changing the shape or scale of a local area in the image rather than the geometry of the entire image. Distortion is not just stretching or tilting, but can also include twisting, curling, or compressing local areas of the image. The formula is as follows, For point The distance to the center of the image, is the distortion coefficient.

[0077] ,

[0078] .

[0079] 5. Image Gaussian noise: In order to reduce image noise and reduce its detail level, Gaussian noise is added to the input image to simulate noise interference in the image. It adds random noise generated from Gaussian distribution to each pixel of the image. The formula is as follows, is the original image, is the image after adding noise, The mean is 0 and the variance is Gaussian noise.

[0080] .

[0081] 6. Motion blur processing: simulate the motion effect of an image at a certain angle and distance. Through this motion blur effect, the training data can be closer to the blurred images in actual usage scenarios, thereby improving the robustness and generalization ability of the model. The formula for generating the blur kernel is as follows: is the size of the blur kernel, and the value of the blur kernel is normalized:

[0082] .

[0083] Represents the rotation matrix The formula for rotating the blur kernel is as follows: is the rotation angle, and Respectively represent the offset of the image center after rotation:

[0084] .

[0085] Then the local area blurring effect is achieved through the convolution operation of the following formula: Represents the original image point The pixel value at Represents the point after the convolution operation Pixel value at:

[0086] .

[0087] Finally, the pixels beyond the valid range are cropped. The formula is shown below:

[0088] .

[0089] After this series of operations, the image will show a motion blur effect.

[0090] 7. Gaussian blur processing: Construct a Gaussian pyramid and use the convolution kernel to downsample the image multiple times to achieve the blurring operation on the image. The so-called Gaussian blur is the process of weighted averaging the entire image. The value of each pixel is obtained by weighted averaging the pixel values ​​of itself and other pixels in the neighborhood.

[0091] 8. Random color adjustment: Randomly increase the brightness, contrast, saturation and hue of the image to enhance color diversity.

[0092] Since the input scene text area has different lengths, it needs to be scaled to the size required by the model. Therefore, when processing the scene text image, its width and height are first scaled in equal proportion. The scaling process uses bilinear interpolation to scale the width and height of the image to the same width and height as the input image required by the model. In the present invention, the image is scaled to a height of 32 pixels, a length of 128 pixels, and a channel number of 3. Bilinear interpolation is performed through the four known neighboring points. , , , Corresponding to the points in the upper left, lower left, upper right, and lower right corners respectively, to calculate the value of the unknown point, first follow and The two horizontal lines are linearly interpolated, and the formula is as follows:

[0093] ,

[0094] .

[0095] in, and Along and Direction linear interpolation results.

[0096] Then interpolate along the y direction, the formula is as follows:

[0097] .

[0098] in, is the interpolation result along the y direction.

[0099] For labels, map the labels corresponding to each image to the index of the characters in the corresponding dictionary file. Because the autoregressive method is used, the start marker is added to the beginning of the input sequence in the training set. <sos>, add an end marker to the end of the target sequence index <eos>, followed by a blank filler <pad>Unify the length of all tags so that the starting and ending points of decoding can be clearly defined, and the sequence length can be unified for batch processing.

[0100] After preprocessing, the separated multi-granularity Vision Transformer module proposed in the present invention is used to extract vectors that can represent features of different granularities of the image, and the vector features are fused into a global vector. Then, the text of the image is obtained through the attention-based decoder of the global channel-aware context module. The specific steps are as follows:

[0101] Step 2.2: Encode the preprocessed samples using an encoder to extract a global vector containing multi-granularity information.

[0102] Step 2.2.1: Use the Vision Transformer module to extract multi-granularity features from the image to obtain multi-granularity features.

[0103] Step 2.2.1.1: Divide the image into small blocks of different sizes, perform convolution operations on the small blocks and combine them with position encoding.

[0104] Step 2.2.1.2: During the convolution process, the extraction process of each granularity information is independent, and after the extraction is completed, it passes through the transformer-based encoder.

[0105] The purpose of this step is to extract multi-granular features of the image, which can reflect the important information and content in the image.

[0106] Transformer usually refers to a model architecture based on the self-attention mechanism, which first performed well in the field of natural language processing and recently began to shine in the field of image processing. The Transformer model is different from the traditional convolutional neural network (CNN) or recurrent neural network (RNN). It is mainly composed of a self-attention layer. The self-attention mechanism can find the dependencies between each position in the input sequence, allowing the model to pay attention to different positions in the input sequence at the same time. Vision Transformer (ViT) is a model that applies the Transformer architecture to computer vision tasks, using the Transformer's self-attention mechanism to process image data.

[0107] In order to better extract information from images, this paper innovatively designs a separate multi-granularity feature extraction module based on Vision Transformer. Multi-granularity usually refers to considering different levels of detail or scales when processing data. This method can help the model better capture the complexity and diversity of information and its basic structure. Figure 3 Specifically, this module divides the image into multiple small patches of different sizes, and each patch after segmentation is regarded as an independent processing unit to facilitate subsequent feature extraction and encoding. These small patches are then flattened and combined with position encoding, and input into the Transformer encoder to obtain a result containing multi-granular semantics of the image.

[0108] First, the input image is operated through convolution kernels of different sizes. Here, convolution kernels of sizes 8×8, 16×16, and 32×32 are used, with step sizes of 8, 16, and 32 respectively. These convolution kernels divide the image into 64, 16, and 4 regions respectively. The number of input channels of the convolution kernel is 3, and the number of output channels is set to 768. The formula of this process is as follows, where represents the convolution operation, It means flattening the last two dimensions of the convolution output, that is, flattening the small squares after the image is divided.

[0109] ,

[0110] ,

[0111] .

[0112] in, , , They represent different patch embeddings, that is, the result of flattening the small blocks obtained by convolution kernels of different sizes on the input image.

[0113] In this process, three convolution kernels are used for convolution separately to ensure that the extraction of information at each granularity is separated from each other. This separate extraction strategy enables the model to fully capture the fine-grained and global semantic information in the image, ensuring that each layer can focus on information at a specific scale, thereby improving the recognition ability of complex backgrounds, text deformation, and irregular image areas.

[0114] The three patches of different sizes are combined with the class token (cls tag) initialized with Gaussian distribution. The formula is as follows:

[0115] .

[0116] Afterwards, the merged Add the position code to get . Specifically, Figure 3 As shown, these three patches of different sizes and the class token initialized with Gaussian distribution ( Mark) merge, The position of the marker is recorded as 0. The dimension is 64*768, and the positions of these 64 1-dimensional vectors are recorded as 1-64 respectively. The dimension is 16*768, and the positions are recorded as 65-80. The dimension is 4*768, the position is 81-84, and the merged And the sinusoidal position code generated according to its position mark is added.

[0117] The formula is as follows:

[0118] .

[0119] in, Encode the position.

[0120] Here, learnable position encoding is used to introduce the position information between patches. Because the Transformer model itself does not have the ability to understand the position of elements in a sequence or image, through position encoding, the model can distinguish patches at different positions and understand the position of the patch in the image, so as to better understand the structure and context information of the image. Input into the Transformer encoder and get the output of the separated Multi-granularity Vision Transformer (MVit) module.

[0121] The Transformer encoder process is as follows Figure 4 As shown in Figure 1, it can efficiently extract and encode the input data. Inside the encoder, the output of each patch interacts and fuses with the output of other patches to extract higher-level features. In this process, the weights of features at each granularity and between each block are dynamically adjusted to achieve feature fusion at different granularities.

[0122] like Figure 4 As shown in the figure, the encoder first normalizes the input and then sends it to the multi-head attention. The encoder performs a residual connection between the output of the multi-head attention and the input of the encoder to avoid losing important information in the deep network. The formula is as follows:

[0123] ,

[0124] .

[0125] in, Represents normalized After the operation, it is sent to multi-head attention The intermediate parameters obtained in express The output obtained after normalization and passing through the multi-layer perceptron.

[0126] The multi-head attention mechanism is the core part of the Transformer model, which allows the model to focus on other positions in the input sequence while processing the data at each position, thereby capturing the dependencies between the data. In the multi-head attention mechanism, each attention head calculates the attention score independently, and then the results of multiple attention heads are merged.

[0127] The data is then processed by a multi-layer perceptron (MLP) after normalization. The multi-layer perceptron is a feed-forward neural network that can further extract features and transform the input data. In the present invention, the multi-layer perceptron includes two linear layers and a ReLU activation function. The first linear layer converts the dimension of the input data to the dimension of the middle layer, the activation function is used to increase the nonlinearity, and the second linear layer converts the dimension of the middle layer back to the original dimension. After being processed by the multi-layer perceptron, the data is normalized and residually connected again. This step has the same purpose as the previous normalization and residual connection, which is to maintain the stability of the data and the flow of information.

[0128] Step 2.2.2: Use the global average pooling layer to integrate the extracted multi-granular features with the fine-grained and global features of the image into a single vector as the global vector.

[0129] In a specific implementation, this embodiment uses global average pooling to aggregate and integrate the outputs of separate multi-granularity VisionTransformer modules, thereby combining the fine-grained and global features of the image into a single vector, which serves as a global vector and contains the multi-granularity global semantic information of the image.

[0130] Specifically, after an image is processed by the separated multi-granularity Vision Transformer module of the present invention, the output dimension is 85×768. The present invention uses global average pooling to process the output. The formula is as follows, thereby obtaining a global vector with a dimension of 1×768.

[0131] .

[0132] in, It is the global average pooling operation After the output.

[0133] This vector contains local information in the image as well as global context information. The features of each granularity are integrated in the feature fusion process to form a global vector containing multi-granularity information, providing a more comprehensive input for the subsequent decoder.

[0134] Step 2.2.3: Fuse the global vector with the context information for decoder processing.

[0135] Step 2.2.3.1: Extract context information and context position information using the global channel-aware context module. This module designed by the present invention disperses the semantic information in the global vector output by the encoder to various positions. The module calculates the weight of each channel through the global context and directly applies it to each position of the input, thereby obtaining a result that is easy for the decoder to process simply and quickly.

[0136] In a specific embodiment, the present invention innovatively designs a global channel-aware context module, whose structure is as follows: Figure 5 As shown in Figure 1, this module can use global context information to weight the features of the input sequence and dynamically adjust the feature expression of each position in the input sequence, thereby improving the expressiveness of the features and spreading the global information to each position. The global vector first passes through a linear layer and is converted into a new feature representation. The linearly transformed features are then added to the position embedding. The purpose of position embedding is to add position information to the input data, because neural networks are usually position-invariant, that is, they cannot recognize the order of the input data. By expanding the global vector with a dimension of 1×768 to After dimensions, the sum dimension is ×768 position embedding addition to get the output , the formula is as follows:

[0137] .

[0138] in, Add one to the maximum length of the label, because the output of the model includes the end token <eos>, set the output dimension of the model to the maximum length of the label plus 1. Adding position information can provide the model with order information of each position in the sequence, so that the model can distinguish elements at different positions in the sequence, thereby effectively capturing the structure and order dependency in the sequence.

[0139] The result after adding the position embedding is globally averaged pooled along the length of the sequence to obtain the global context vector. The formula is as follows:

[0140] .

[0141] in, is the global context vector, express The first tokens.

[0142] This vector contains the global features of the entire sequence. The two fully connected layers in the module are equivalent to a simple two-layer perceptron structure. Through the intermediate nonlinear activation (the present invention uses the ReLU function), nonlinear factors can be introduced into the network to help the model learn more complex feature expressions. This process can be expressed as:

[0143] .

[0144] in, is the output of the linear layer, , is the weight matrix of the linear layer, , is the bias vector.

[0145] The context weights are obtained by applying the softmax function to the output of the linear layer. The softmax function is a commonly used multi-classification output layer activation function that maps the input value to a probability distribution between 0 and 1, and the sum of all output values ​​is 1.

[0146] Finally, since the importance of each channel in this task is similar at each position, the global weight can be used for unified processing, so as to better utilize the original global vector and position information. The calculated context weight is applied to each position of the input, and the weight is adjusted by element-by-element multiplication. After passing through the normalization layer and the linear layer, the formula is as follows to obtain the feature :

[0147] ,

[0148] .

[0149] in, for pass After the function The intermediate parameters obtained after element-by-element multiplication, is the weight matrix, is the bias vector.

[0150] Step 2.2.3.2: Use the feature fusion module to fuse the context information, context position information, and the corresponding global vector to form a more comprehensive and in-depth understanding. The feature fusion module can comprehensively consider different feature information to extract more accurate and reliable feature representations. During training, the context information is the label passed through the embedding layer. During testing, step-by-step decoding is used, and each time step uses the output of the previous time step through the embedding layer as the context information.

[0151] In a specific implementation, the feature fusion module fuses the context information and the location information of the context as well as the output of the global channel-aware context module. In the present invention, the context information is first input into this module through the embedding layer, and then added to the linear layer, as shown below:

[0152] .

[0153] in, is the fused feature, is a linear layer, is the embedding of context information, Embedding of location information.

[0154] Step 2.3: Use the decoder to decode the global vector and obtain the recognition result.

[0155] Specifically, in this embodiment, the decoder uses a language decoder to decode the global vector in a step-by-step decoding manner. The decoder superimposes a two-layer masked multi-head self-attention mechanism and a feedforward network to gradually capture features and information at different levels. Figure 6 As shown in the figure, during each stacking process, the input will go through multi-head self-attention, random dropout, layer normalization and feed-forward network, thereby enhancing the expressive power of the input features layer by layer.

[0156] In a specific implementation, the structure of the language decoder is as follows Figure 6 As shown, the present invention stacks multiple layers of multi-head self-attention mechanisms and feedforward networks. Due to the limited depth of the single-layer masked multi-head self-attention, it can only focus on relatively local information when capturing dependencies. By superimposing two layers of attention mechanisms, a progressive learning mechanism is introduced. After the model captures relatively local dependencies in the first layer, it can further model more complex and longer-distance dependencies in the second layer. Compared with using only one layer of multi-head self-attention, the model can gradually capture features and information at different levels at different levels. In each stacking process, the input will go through multi-head self-attention, random dropout, layer normalization, and feedforward network, thereby enhancing the expressive power of the input features layer by layer. The process of masked multi-head self-attention is as follows, where Mask for the future:

[0157] ,

[0158] ,

[0159] .

[0160] in, is the fused feature, is the query vector, is the key vector, is a value vector, To represent the output using the attention mechanism, is the output of the masked multi-head self-attention.

[0161] The masked multi-head self-attention module can dynamically adjust the model's attention to previously generated tokens when generating each new token. The model calculates the query ( )、key( ) and value( ) to capture the dependencies between different tokens, thereby deciding which information to focus on when generating a new token. Because multiple attention heads are used, the model can independently calculate attention weights in multiple different subspaces. Each attention head focuses on a different part of the sequence, thereby achieving richer feature expression. The role of the mask is to ensure that when the model generates a token, it can only focus on the generated tokens before the token, and will not leak information about future tokens that have not yet been generated. This is crucial in language generation tasks, because when generating a word, the model can only use previous information, but not future information.

[0162] The parallelism of the multi-head self-attention mechanism allows the model to calculate the attention weights between the token at each position and all previous tokens in parallel. The model pays attention to multiple information in different subspaces at the same time, thus avoiding the sequential dependency problem in traditional networks. The masked multi-head self-attention module processes each position in the input sequence through adaptive weighted summation, enhancing the model's ability to capture global dependencies. Each attention head dynamically adjusts the weight according to different contextual information, allowing the model to have a deeper understanding of local and global semantic relationships. The masked multi-head self-attention module ensures that the information of the tokens generated in the past can be effectively transferred to the currently generated tokens without leaking future information. This transmission of information flow is the basis for generating reasonable sequences.

[0163] The stacked two-layer masked multi-head self-attention mechanism can capture global dependencies more effectively, helping the model to understand the context more comprehensively when generating sequences. The use of masks ensures that each time a new token is generated, the model only depends on previously generated tokens to avoid leakage of future information. Through parallel computation of multiple attention heads, the model can efficiently process long sequences while learning complex features in different subspaces. Each attention head focuses on a different level in the sequence, thereby improving the model's performance in capturing sequence-level features and overall representation capabilities.

[0164] In order to prevent the model from overfitting, the present invention also introduces a random drop mechanism. Some neurons are randomly dropped during training so that the model uses different subsets for calculation each time it propagates forward, thereby enhancing the generalization ability of the model. Layer normalization is used to standardize the input data to ensure consistent feature distribution. After multi-head self-attention processing, the data is normalized through layers and then enters the feedforward fully connected network. The network consists of two layers of linear transformation, with a nonlinear activation function in the middle to further extract and enhance feature representation.

[0165] In order to stabilize the gradient propagation, both the masked multi-head self-attention and feedforward networks use residual connections. This mechanism passes the input directly to the output, reducing the gradient vanishing problem and helping the network converge faster. In addition, residual connections allow the original information to be passed directly, preventing the loss of important information in deep networks. Residual connections play the role of effectively fusing feature representations at different levels in masked multi-head self-attention and feedforward networks. Ultimately, this design simplifies the learning task of the model, improves the ability to model complex dependencies, and further improves the generalization performance of the model.

[0166] Step 2.4: Finally, the scene text recognition model outputs the probability distribution of each position as a character. The position with the maximum probability is taken and then mapped to character text according to the dictionary. The prediction result is also used as context information in the feature fusion module during the reasoning process.

[0167] In a specific embodiment, the output head portion of the scene text recognition model includes a linear layer and a softmax normalization layer. The linear layer is used to map the output vector of the decoder to a dimension equal to the size of the vocabulary, and transform the implicit representation into a score for each possible output symbol. These scores are converted into probability distributions. The softmax layer converts each score to a value between 0 and 1 and ensures that the sum of the probabilities of all categories is 1. This step is to generate the conditional probability of each category at each time step, and the decoder can then select or sample the most likely output symbol based on these probabilities.

[0168] Step 2.5: During the scene text recognition model training process, cross entropy loss and L1 loss are combined as the loss function.

[0169] In a specific embodiment, the present invention innovatively combines cross entropy loss and L1 loss as the loss function of the model. Calculated by cross entropy loss, the formula is as follows:

[0170] .

[0171] in, For the model The sample The logit value of the class, For the The corresponding true category of the sample The logit value of is the batch size, When using cross loss for reasoning, it is observed that the model pays more attention to context information rather than image information, which is manifested in that the model is more inclined to recognize other words rather than misrecognize a certain letter when it is misrecognized. In order to reduce this tendency, the present invention chooses to add L1 loss as a balance. The second loss function Calculated by L1 loss, the formula is as follows:

[0172] .

[0173] in, is the length of the target value, To identify the length of the result, The output of the model can be made closer to the target.

[0174] The total loss of the scene text recognition model is as follows, where is the trade-off coefficient:

[0175] .

[0176] Embodiment 2:

[0177] Embodiment 2 of the present invention provides a scene text recognition system based on multi-granularity training of global vectors, including:

[0178] A data acquisition module is configured to acquire an image to be recognized and crop the image to be recognized to obtain a portion containing text;

[0179] A scene recognition module is configured to perform scene recognition on a portion containing text using a scene text recognition model to obtain a recognition result;

[0180] Among them, the scene recognition module includes a model training module, which is configured to train the scene text recognition model in stages, and the steps include: preprocessing the training samples, encoding the preprocessed samples with an encoder, extracting a global vector containing multi-granularity information, and decoding the global vector with a decoder to obtain a recognition result; using the error voting method to divide the training samples into five difficulty levels, training from easy to difficult, and gradually adding more difficult data to the training data to obtain better generalization ability, and using a cosine annealing heating restart learning rate scheduling strategy to adapt to the staged training process.

[0181] The steps involved in the above embodiment 2 correspond to those in the method embodiment 1. For the specific implementation method, please refer to the relevant description part of the embodiment 1.

[0182] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0183] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.< / eos> < / pad> < / eos> < / sos>

Claims

1. A scene text recognition method based on multi-granularity training of global vectors, characterized in that: The following steps are involved: Obtain an image to be recognized, and crop the image to be recognized to obtain a portion containing text; A scene text recognition model is used to perform scene recognition on a part containing text to obtain a recognition result, wherein the scene text recognition model is trained in stages, and the steps include: preprocessing the training samples, encoding the preprocessed samples by using an encoder, extracting a global vector containing multi-granularity information, and decoding the global vector by using a decoder to obtain a recognition result; using an error voting method to divide the training samples into five difficulty levels, training from easy to difficult, and gradually adding more difficult data to the training data to obtain better generalization ability, and using a learning rate scheduling strategy of cosine annealing heating restart to adapt to the staged training process; The specific steps to extract the global vector containing multi-granularity information are: The Vision Transformer module is used to extract multi-granularity features from the image to obtain multi-granularity features; A global average pooling layer is used to integrate the extracted multi-granular features with the fine-grained and global features of the image into a single vector as the global vector; Fuse the global vector with context information for decoder processing; The specific steps of fusing the global vector with the context information are: 1) Extract context information and context location information using the global channel-aware context module; This module uses global context information to weight the features of the input sequence, dynamically adjusts the feature expression of each position in the input sequence, and diffuses the global information to each position. The global vector first passes through a linear layer and is converted into a new feature representation. The linearly transformed features are then added to the position embedding, which adds position information to the input data. By expanding the global vector with a dimension of 1×768 to After dimensions, the sum dimension is ×768 position embedding addition to get output , the formula is as follows: ； in, Add one to the maximum length of a tag; The result after adding the position embedding is globally averaged pooled along the length of the sequence to obtain the global context vector. The formula is as follows: ； in, is the global context vector, express The first Tokens; This vector contains the global features of the entire sequence; the two fully connected layers in the module are equivalent to a simple two-layer perceptron structure. Through the nonlinear activation in the middle, nonlinear factors are introduced into the network to help the model learn more complex feature expressions. This process is expressed as: ； in, is the output of the linear layer, , is the weight matrix of the linear layer, , is the bias vector; Apply the softmax function to the output of the linear layer to obtain the context weight; Use global weights for unified processing, apply the calculated context weights to each position of the input, adjust the weights by element-by-element multiplication, and then pass through the normalization layer and the linear layer. The formula is as follows to get the feature : ， ； in, for pass After the function The intermediate parameters obtained after element-by-element multiplication, is the weight matrix, is the bias vector; 2) Use the feature fusion module to fuse the context information, context position information and the corresponding global vector; During training, the context information is the label passed through the embedding layer. During testing, step-by-step decoding is used, and each time step uses the output of the previous time step through the embedding layer as the context information; The feature fusion module fuses the context information, the location information of the context, and the output of the global channel-aware context module. The context information is first passed through the embedding layer and then input into the feature fusion module. After the linear layer, it is added. The formula is as follows: ； in, is the fused feature, is a linear layer, is the embedding of context information, Embedding of location information.

2. The scene text recognition method based on multi-granularity training of global vector as claimed in claim 1, characterized in that: The specific steps of preprocessing the training samples are: geometric transformation, random cutting and random noise addition to the images of the training samples.

3. The scene text recognition method based on multi-granularity training of global vector as claimed in claim 1, characterized in that: The specific steps of using the Vision Transformer module to extract multi-granularity features from images are as follows: Divide the image into small blocks of different sizes, perform convolution operations on the small blocks and combine them with position encoding; During the convolution process, the extraction process of each granularity information is independent, and after the extraction is completed, it passes through a transformer-based encoder.

4. The scene text recognition method based on multi-granularity training of global vector as claimed in claim 1, characterized in that: The decoder uses a language decoder, which superimposes a two-layer masked multi-head self-attention mechanism and a feedforward network.

5. The scene text recognition method based on multi-granularity training of global vector as claimed in claim 4, characterized in that: The decoder decodes the global vector in a step-by-step decoding manner to gradually capture different levels of features and information at different levels.

6. The scene text recognition method based on multi-granularity training of global vector as claimed in claim 1, characterized in that: The decoder introduces a random dropout mechanism, which randomly drops some neurons during training, so that the scene text recognition model uses different subsets for calculation in each forward propagation.

7. The scene text recognition method based on multi-granularity training of global vector as claimed in claim 1, characterized in that: During the training process of the scene text recognition model, cross entropy loss and L1 loss are combined as loss functions.

8. A scene text recognition system based on multi-granularity training of global vectors, using the scene text recognition method based on multi-granularity training of global vectors as claimed in claim 1, characterized in that: include: A data acquisition module is configured to acquire an image to be recognized and crop the image to be recognized to obtain a portion containing text; A scene recognition module is configured to perform scene recognition on a portion containing text using a scene text recognition model to obtain a recognition result; Among them, the scene recognition module includes a model training module, which is configured to train the scene text recognition model in stages, and the steps include: preprocessing the training samples, encoding the preprocessed samples with an encoder, extracting a global vector containing multi-granularity information, and decoding the global vector with a decoder to obtain a recognition result; using the error voting method to divide the training samples into five difficulty levels, training from easy to difficult, and gradually adding more difficult data to the training data to obtain better generalization ability, and using a cosine annealing heating restart learning rate scheduling strategy to adapt to the staged training process.

Images