An image-based text recognition method, device and medium

Abstract

The application provides an image-based text recognition method, device and medium, and relates to the technical field of image recognition. In the method, a text image is preprocessed to obtain a corresponding normalized RGB tensor; an image quality vector is generated based on the normalized RGB tensor and an image quality perception network; an initial feature map is obtained based on the normalized RGB tensor and an image visual feature extraction network; the initial feature map is modulated using a FiLM mechanism based on the image quality vector to obtain an intermediate feature map; a spatial attention mask is multiplied element by element with the intermediate feature map to obtain a target feature map; and the target feature map is sequentially modeled and decoded to obtain a text recognition result. The method effectively alleviates the negative impact of image degradation on text recognition performance, significantly reduces the character error rate, and improves the accuracy of the text recognition result.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to an image-based text recognition method, device, and medium. Background Technology

[0002] Existing image-based text recognition methods typically follow a general paradigm of "visual feature extraction—sequence modeling—character decoding." Methods represented by the classic CRNN (Convolutional Recurrent Neural Network) first use a convolutional neural network (such as ResNet or VGG) to perform multi-level convolutions and downsampling on the entire input image to obtain a two-dimensional feature map. Then, this feature map is sliced ​​horizontally column by column, and each column is fed into a bidirectional RNN (LSTM or GRU) to capture the contextual dependencies between characters. Finally, the character sequence is output through the CTC (Connectionist Temporal Classification) decoding layer. Newer methods such as TrOCR and PARSeq introduce the Vision Transformer as a backbone, dividing the image into fixed-size patches, obtaining the global receptive field through a self-attention mechanism, and then regressively generating characters via a Transformer decoder.

[0003] However, the above method also has the following technical problems:

[0004] The methods described above typically assume that the input image is of good quality, implicitly assuming that "the image has no significant degradation." However, in real-world applications, text images are often affected by various degradation factors, including but not limited to: reduced signal-to-noise ratio due to low illumination, blurring caused by motion or defocusing, block artifacts produced by compression algorithms such as JPEG, decreased local contrast due to uneven lighting, and missing character details caused by insufficient resolution. When one or more of these degradations are present, the visual features extracted by the backbone network (such as convolutional neural networks or Vision Transformers) are often severely distorted, making it difficult for subsequent sequence modeling and decoding modules to recover the correct character semantics. This leads to a significant increase in the character recognition error rate and seriously affects the accuracy of text recognition results. Summary of the Invention

[0005] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows:

[0006] According to a first aspect of the present invention, an image-based text recognition method is provided, the method comprising the following steps:

[0007] S1. Preprocess the text image to obtain the normalized RGB tensor corresponding to the text image; the text image is the part of the target image that corresponds to the text region.

[0008] S2. Based on the normalized RGB tensor and the image quality perception network, generate the image quality vector corresponding to the text image. Each dimension of the image quality vector corresponds to a preset image degradation type in the preset image degradation type set, and the value range of each dimension is [0, 1], which is used to represent the image quality level of the text image under the preset image degradation type corresponding to each dimension. The larger the dimension value, the higher the image quality level of the text image under the preset image degradation type corresponding to that dimension. Among them, the higher the image quality level of the text image under the preset image degradation type corresponding to that dimension, the lower the degradation severity of the preset image degradation type corresponding to that dimension in the text image.

[0009] S3. Based on the normalized RGB tensor and the image visual feature extraction network, obtain the initial feature map corresponding to the text image.

[0010] S4. Based on the image quality vector corresponding to the text image, the FiLM mechanism is used to modulate the initial feature map corresponding to the text image to obtain the intermediate feature map corresponding to the text image.

[0011] S5. Multiply the spatial attention mask element-wise with the intermediate feature map corresponding to the text image to obtain the target feature map corresponding to the text image; the spatial attention mask is obtained based on the image quality vector corresponding to the text image.

[0012] S6. Perform sequence modeling and decoding on the target feature map corresponding to the text image to obtain the text recognition result corresponding to the text image.

[0013] According to a second aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein a computer program is stored in the storage medium, and the computer program is loaded and executed by a processor to implement the aforementioned method.

[0014] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned method.

[0015] The present invention has at least the following beneficial effects:

[0016] This invention provides an image-based text recognition method, device, and medium. The method involves: preprocessing the text image to obtain a normalized RGB tensor; generating an image quality vector based on the normalized RGB tensor and an image quality perception network; obtaining an initial feature map based on the normalized RGB tensor and an image visual feature extraction network; modulating the initial feature map using a FiLM mechanism based on the image quality vector to obtain an intermediate feature map; element-wise multiplying a spatial attention mask with the intermediate feature map to obtain a target feature map; and performing sequence modeling and decoding on the target feature map to obtain the text recognition result. As can be seen, this invention utilizes an image quality perception network to generate structured image quality vectors, explicitly characterizing the severity of degradation of the input text image across various preset image degradation types. Subsequently, based on the image quality vectors, the initial feature map extracted by the image visual feature extraction network is subjected to channel-level adaptive modulation using the FiLM mechanism to obtain an intermediate feature map. Furthermore, based on spatial attention masks, the intermediate feature map is weighted by region, thereby effectively suppressing severely degradation-affected regions and enhancing the representation ability of reliable character regions. This effectively alleviates the negative impact of image degradation on text recognition performance, significantly reduces the character error rate, and improves the accuracy of text recognition results. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating an image-based text recognition method provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar tasks and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0021] Embodiments of the present invention provide an image-based text recognition method, the method comprising the following steps, such as... Figure 1 As shown:

[0022] S1. Preprocess the text image to obtain the normalized RGB tensor corresponding to the text image; the text image is the part of the target image that corresponds to the text region.

[0023] Specifically, the target image is an image containing text content to be recognized, such as an image containing vehicle identification number, license plate number, ticket text, or industrial nameplate. It is suitable for optical character recognition (OCR) scenarios, and will not be elaborated further here.

[0024] Specifically, preprocessing includes at least: rotation correction based on text orientation, color space conversion to RGB format, and pixel value normalization.

[0025] Furthermore, when the text image is in RAW format, preprocessing also includes white balance correction.

[0026] Specifically, the normalized RGB tensor is a three-dimensional real-valued array with a size of H×W×3, where H is the image height, W is the image width, and the third dimension corresponds to the three color channels: red (R), green (G), and blue (B). The image height and image width are measured in pixels. The specific values ​​of the image height and image width are preset by those skilled in the art according to actual needs, for example: H=64, W=256, which will not be elaborated here.

[0027] Furthermore, each element in the normalized RGB tensor represents the pixel value of the corresponding pixel in the corresponding color channel, and all pixel values ​​are linearly mapped from the interval [0, 255] to [0, 1].

[0028] Through the above steps, the text image is oriented, its color space is unified, and its pixel values ​​are normalized. When necessary, white balance correction is performed on the RAW format image. The text image is converted into a normalized RGB tensor with a unified structure and standardized values. This effectively eliminates interference from non-textual factors introduced by differences in shooting angle, lighting conditions, imaging equipment, or image format. It significantly improves the consistency and usability of the input data and provides high-quality, standardized input for subsequent text recognition.

[0029] Specifically, the process before step S1 also includes:

[0030] Text detection is performed on the target image to determine at least one text region and obtain the location information corresponding to each text region. Those skilled in the art will know that any text detection method in the prior art is within the protection scope of this invention. For example, the target image is processed by models such as DBNet, EAST, CRAFT or PSENet to locate at least one text region and output the location information corresponding to each text region. These methods will not be elaborated here.

[0031] For each text region, based on the location information corresponding to the text region, a portion of the image corresponding to that text region is cropped from the target image, and this portion of the image is used as the text image.

[0032] Through the above steps, one or more text regions are automatically located from the target image, and the corresponding part of the image is accurately cropped as the text image based on the position information of the text region. Irrelevant backgrounds, icons, table lines and other interferences are removed, and only the region of the character to be recognized is retained, which significantly reduces the difficulty of subsequent recognition.

[0033] S2. Based on the normalized RGB tensor and the image quality perception network, generate the image quality vector corresponding to the text image; each dimension of the image quality vector corresponds to a preset image degradation type in the preset image degradation type set, and the value range of each dimension is [0, 1].

[0034] Specifically, the preset image degradation type set includes several first-class preset image degradation types and several second-class preset image degradation types. The first-class preset image degradation types are specific physical degradation types, including at least: blur, Gaussian noise, JPEG compression, low illumination, low contrast, color cast, and rain / fog occlusion. The second-class preset image degradation types are degradation types that are not enabled.

[0035] Specifically, for each preset image degradation type, the degradation intensity corresponding to the preset image degradation type is represented by the image degradation type parameter corresponding to that preset image degradation type. The value range of the image degradation type parameter is [0, 1]. The larger the value of the image degradation type parameter, the greater the degradation intensity, the higher the degradation severity, and the lower the image quality level corresponding to that preset image degradation type. Among them, the degradation intensity corresponding to the preset image degradation type can be understood as: the quantitative intensity of the specific degradation phenomenon corresponding to the preset image degradation type in the image; the degradation severity corresponding to the preset image degradation type can be understood as: the negative impact of the specific degradation phenomenon corresponding to the preset image degradation type on the usability of the image; the image quality level corresponding to the preset image degradation type can be understood as: the quality of the image under the influence of the specific degradation phenomenon corresponding to the preset image degradation type, defined as 1 minus the image degradation type parameter corresponding to the preset image degradation type.

[0036] Specifically, the image quality vector has a dimension of 64.

[0037] Specifically, the sum of the number of the first type of preset image degradation type and the number of the second type of preset image degradation type in the preset image degradation type set is consistent with the dimension of the image quality vector; for example, the preset image degradation type set includes: Gaussian blur, Gaussian noise, JPEG compression, low illumination, low contrast, color cast, rain and fog occlusion, degradation type 1 not enabled, degradation type 2 not enabled, ..., degradation type 57 not enabled.

[0038] Specifically, the values ​​of each dimension in the image quality vector represent the image quality level of the text image under the preset image degradation type corresponding to each dimension. A larger dimension value indicates a higher image quality level for the text image under that preset image degradation type, and a more subtle degradation phenomenon in the text image corresponding to that dimension, resulting in a smaller adverse impact on text recognition. In other words, a higher image quality level for the text image under that preset image degradation type indicates a lower degree of degradation severity. Conversely, a smaller dimension value indicates a lower image quality level for the text image under that preset image degradation type, and a more pronounced degradation phenomenon in the text image corresponding to that dimension, resulting in a greater adverse impact on text recognition. In other words, a lower image quality level for the text image under that preset image degradation type indicates a higher degree of degradation severity.

[0039] Furthermore, the dimension in the image quality vector corresponding to the second preset image degradation type is always 1.

[0040] Specifically, the image quality perception network is constructed from a lightweight convolutional neural network, including:

[0041] Remove the pre-trained classification head of the lightweight convolutional neural network, connect three fully connected layers sequentially after the global average pooling layer of the lightweight convolutional neural network, and apply the Sigmoid activation function after the last fully connected layer to output an image quality vector with values ​​ranging from [0,1] for each dimension.

[0042] Specifically, the output dimensions of the three fully connected layers are 256, 128, and 64, respectively.

[0043] Specifically, the image quality perception network is trained based on a training sample set, which includes several training samples. Each training sample includes a normalized RGB tensor corresponding to a preset low-quality image and a true quality vector corresponding to the preset low-quality image. The preset low-quality image is obtained by randomly performing several (e.g., 1 to 3) degradation operations corresponding to the first type of preset image degradation on a preset high-quality image. The true quality vector is a vector used to represent the distribution of image quality levels of the preset low-quality image under each preset image degradation type. Each dimension of the true quality vector corresponds to a preset image degradation type in the set of preset image degradation types.

[0044] Specifically, for each dimension of the true quality vector corresponding to the preset low-quality image, the dimension value ranges from [0, 1], representing the image quality level of the preset low-quality image under the preset image degradation type corresponding to that dimension. The larger the value of the dimension, the higher the image quality level of the preset low-quality image under the preset image degradation type corresponding to that dimension; the smaller the value of the dimension, the lower the image quality level of the preset low-quality image under the preset image degradation type corresponding to that dimension. Among these, the lower the image quality level of the preset low-quality image under the preset image degradation type corresponding to that dimension, the higher the degradation severity of the preset low-quality image under that dimension; and the higher the image quality level of the preset low-quality image under the preset image degradation type corresponding to that dimension, the lower the degradation severity of the preset low-quality image under that dimension.

[0045] Specifically, the values ​​of each dimension in the true quality vector corresponding to the preset low-quality image are obtained in the following way:

[0046] For each preset image degradation type, if a degradation operation corresponding to the preset image degradation type is performed on the preset high-quality image during the process of obtaining the preset low-quality image from the preset high-quality image, then the difference obtained by subtracting the image degradation parameter corresponding to the degradation operation from 1 is used as the specific value of the dimension corresponding to the preset image degradation type in the true quality vector; the image degradation parameter corresponding to the degradation operation is the value obtained by normalizing the intensity parameter of the degradation operation to the [0, 1] interval according to the preset maximum intensity parameter corresponding to the degradation operation; it can be understood that the image degradation parameter corresponding to the degradation operation is equal to the intensity parameter of the degradation operation divided by the preset maximum intensity parameter corresponding to the degradation operation; the preset maximum intensity parameter corresponding to the degradation operation is preset by those skilled in the art according to actual needs.

[0047] For each preset image degradation type, if no degradation operation corresponding to the preset image degradation type is performed on the preset high-quality image during the process of obtaining the preset low-quality image from the preset high-quality image, then the specific value of the dimension corresponding to the preset image degradation type in the real quality vector is set to 1; for example: if the preset low-quality image is obtained by performing "Gaussian blur (intensity parameter 2.0)" and "JPEG compression (intensity parameter 40)" degradation operations on the preset high-quality image; then it can be seen that during the process of obtaining the preset low-quality image from the preset high-quality image, the two degradation operations corresponding to the preset image degradation type, Gaussian blur and JPEG compression, were performed on the preset high-quality image; where, if the preset low-quality image is obtained by performing "Gaussian blur (intensity parameter 2.0)" and "JPEG compression (intensity parameter 40)" degradation operations on the preset high-quality image, then the specific value of the dimension corresponding to the preset image degradation type in the real quality vector is set to 1; If the maximum intensity parameter is 5.0, then the image degradation parameter corresponding to the Gaussian blur degradation operation is 2.0 / 5.0 = 0.4. If the preset maximum intensity parameter corresponding to the JPEG compression degradation operation is 64, then the image degradation parameter corresponding to the JPEG compression degradation operation is 40 / 64 = 0.625. Therefore, in the true quality vector corresponding to this preset low-quality image, the specific value of the dimension corresponding to the preset image degradation type Gaussian blur is 1 - 0.4 = 0.6, the specific value of the dimension corresponding to the preset image degradation type JPEG compression is 1 - 0.625 = 0.375, and the specific value of the dimension corresponding to any other preset image degradation type besides Gaussian blur and JPEG is 1.

[0048] Specifically, the preset high-quality image is a clear text image that is predetermined by those skilled in the art based on actual needs, and will not be elaborated further here.

[0049] Specifically, the normalized RGB tensor corresponding to the preset low-quality image is obtained in the same way as in step S1, where the normalized RGB tensor corresponding to the text image is obtained.

[0050] Specifically, the loss function L used to train the image quality perception network satisfies the following conditions:

[0051] L=∑ n i=1 Smooth L1 (q i -q 1 i );q i q represents the value of the i-th dimension in the image quality vector obtained from the normalized RGB tensor corresponding to the predefined low-quality image and the image quality perception network; 1 i Let q be the value of the i-th dimension in the true quality vector corresponding to the preset low-quality image; where q i The corresponding preset image degradation type and q 1 i The corresponding preset image degradation types are consistent, all being the i-th preset image degradation type; 1≤i≤n, where n is the dimension of the image quality vector.

[0052] Specifically, the dimension of the true quality vector is the same as the dimension of the image quality vector.

[0053] Specifically, Smooth L1 () represents Smooth L1 Loss.

[0054] Furthermore, when |q i -q 1 i When | < 1, Smooth L1 (q i -q 1 i )=0.5×|q i -q 1 i | 2 ,|q i -q 1 i |for q i -q 1 i The absolute value of.

[0055] Furthermore, when |q i -q 1 i When |≥1, Smooth L1 (q i -q 1 i )=|q i -q 1 i -0.5.

[0056] Through the above steps, the image quality perception network is trained using the multidimensional SmoothL1 loss function. This loss function uses a squared term when the prediction error is small to ensure smooth gradients and stable convergence; when the error is large, it turns into a linear term, which effectively reduces the negative impact of outliers or annotation noise. It can improve the robustness of the training process while ensuring regression accuracy, thereby more accurately estimating the intensity of each preset image degradation type.

[0057] S3. Based on the normalized RGB tensor and the image visual feature extraction network, obtain the initial feature map corresponding to the text image.

[0058] Specifically, after step S2 and before step S3, the following steps are also included:

[0059] Based on the image quality vector corresponding to the text image, obtain the image quality score corresponding to the text image.

[0060] Specifically, the image quality score is positively correlated with image quality; the lower the image quality score, the worse the image quality; conversely, the higher the image quality score, the better the image quality.

[0061] Specifically, the image quality score F corresponding to the text image meets the following conditions:

[0062] F=∑ n i=1 (V i ×(1-X i )) / ∑ n i=1 V i V i For X i The corresponding preset importance weight for the preset image degradation type, X i , where is the value of the i-th dimension in the image quality vector corresponding to the text image; wherein, the preset importance weight is preset by those skilled in the art based on the degree of negative impact of the preset image degradation type on the text recognition performance: the smaller the negative impact, the higher the preset importance weight. For example, rain and fog occlusion are usually difficult to completely eliminate through subsequent image enhancement, causing significant interference to character recognition, so they are given a lower weight; while low-light degradation can be partially compensated by brightness correction or quality modulation, and the negative impact is relatively controllable, so it is given a higher weight; further details are omitted here.

[0063] When the image quality score corresponding to the text image is not less than the preset quality score, proceed to step S3; when the image quality score corresponding to the text image is less than the preset quality score, generate a prompt message and send the prompt message to the user, and terminate the process at the same time; the prompt message is used to inform the user that the image quality is low and text recognition cannot be performed.

[0064] Specifically, the preset quality score range is [0.6, 0.9], and preferably, the preset quality score is 0.75.

[0065] In another specific embodiment, X1, X2, ..., X i ..., X n The minimum value in the range is used as the image quality score corresponding to the text image.

[0066] Through the above steps, an image quality vector is obtained by explicitly modeling 64-dimensional degradation intensity using an image quality perception network. An image quality score is then obtained based on the values ​​of each dimension in the image quality vector. If the image quality score is not less than a preset quality score, the process proceeds to the next step. If the image quality score is less than the preset quality score, a prompt message is sent. This approach enables proactive interception of low-quality input during the inference stage, avoiding invalid recognition and error propagation. It overcomes the shortcomings of existing OCR technologies that do not discriminate between input image quality and blindly perform recognition, significantly improving the user experience.

[0067] In one specific embodiment, before proceeding to step S3, the normalized RGB tensor is preprocessed to eliminate several preset image degradation types that can be effectively compensated for by conventional image enhancement techniques.

[0068] Specifically, the preset image degradation types that can be effectively compensated by conventional image enhancement techniques include, but are not limited to, low illumination, color cast, and low contrast. The corresponding enhancement methods are, for example,: adaptive brightness correction or Retinex enhancement for low illumination images; white balance adjustment or gray-world hypothesis correction for color cast images; and histogram equalization or contrast stretching for low contrast images.

[0069] Furthermore, the preprocessed normalized RGB tensor is used as the input to the image visual feature extraction network in step S3.

[0070] Specifically, the above preprocessing is only for preset image degradation types that are easy to recover through pixel-level operations; for structural degradations that are difficult to fully recover, such as blurring and rain / fog occlusion, the original state is preserved.

[0071] Through the above steps, before visual feature extraction, lightweight image enhancement operations are selectively applied to the normalized RGB tensor. Only for degradation types that can be effectively compensated by conventional pixel-level methods (such as low illumination, color cast, and low contrast), mature techniques such as adaptive brightness correction, white balance, and histogram equalization are used for pre-repair. For structural or irreversible degradations such as blurring and rain / fog occlusion, the original state is preserved, and semantic-level compensation is performed by subsequent FiLM modulation and attention mechanisms based on image quality vectors. This avoids artifacts or information distortion that may be introduced by "one-size-fits-all" enhancement, while reducing the burden on the backbone network. While maintaining computational efficiency, it takes into account both explicit optimization for reparable degradations and implicit robust modeling for difficult-to-repair degradations, thereby improving the overall accuracy and stability of text recognition.

[0072] Specifically, the image visual feature extraction network is a deep convolutional neural network or a visual Transformer architecture, used to extract high-level semantic features from the normalized RGB tensor, and configured to output an initial feature map with 512 channels and a spatial downsampling factor of 8; for example: ResNet45 with channel attention mechanism, TinySVTR which adapts the output specification by adjusting the downsampling stride and channel projection layer, or PP-LCNetV2 which controls the downsampling rate by modifying the convolution stride in the final stage.

[0073] S4. Based on the image quality vector corresponding to the text image, the FiLM mechanism is used to modulate the initial feature map corresponding to the text image to obtain the intermediate feature map corresponding to the text image.

[0074] Specifically, the size of the intermediate feature map corresponding to the text image is h×w×512, where h is the length of the intermediate feature map corresponding to the text image, w is the width of the intermediate feature map corresponding to the text image, and the third dimension is the number of channels of the intermediate feature map corresponding to the text image; where h=H / 8 and w=W / 8.

[0075] Furthermore, the feature map T of the c-th channel of the intermediate feature map T2 corresponding to the text image... 2c Meets the following conditions:

[0076] T 2c =γ c ×Norm(T 1c )+β c , 1≤c≤512; where, T 1c This represents the feature map of the c-th channel of the initial feature map T1 corresponding to the text image; Norm() represents the normalization operation (e.g., LayerNorm, InstanceNorm); γ c The c-th component of the scaling factor γ corresponds to the c-th channel; β cIt is the c-th component of the offset factor β, corresponding to the c-th channel.

[0077] Furthermore, γ and β are both real-valued vectors of length 512, and γ and β satisfy the following conditions respectively:

[0078] γ=U γ ×X+b γ ;β=U β ×X+b β U γ and U β The weight matrix is ​​a learnable matrix with a size of 512×n; b γ and b β is a learnable bias vector, with a length of 512; X is the image quality vector corresponding to the text image.

[0079] Through the above steps, dynamic channel-level modulation is performed on the initial feature map based on the image quality vector, without relying on explicit pixel-level image restoration operations. This effectively avoids the introduction of enhancement artifacts and error propagation, and significantly improves the overall robustness of text recognition under degradation conditions.

[0080] S5. Multiply the spatial attention mask element-wise with the intermediate feature map corresponding to the text image to obtain the target feature map corresponding to the text image; the spatial attention mask is obtained based on the image quality vector corresponding to the text image.

[0081] Specifically, the image quality vector corresponding to the text image is broadcast to the same spatial dimension as the intermediate feature map corresponding to the text image. After 1×1 convolution and Sigmoid activation, a spatial attention mask with values ​​in the range of [0, 1] is obtained.

[0082] Through the above steps, channel-level scaling factors and offset factors are generated based on the image quality vector, and channel-dimensional modulation is applied to the intermediate feature map. At the same time, a spatial attention mask is generated in parallel, and spatial-dimensional weighting is applied to the modulated feature map, thereby achieving dual adaptive weighting of channel and space, enabling the network to focus on effective character regions and suppress background or noise regions affected by degradation.

[0083] S6. Perform sequence modeling and decoding on the target feature map corresponding to the text image to obtain the text recognition result corresponding to the text image.

[0084] In one specific embodiment, while obtaining the text recognition result corresponding to the text image, the confidence level of the text recognition result is also obtained. If the confidence level of the text recognition result is lower than a preset confidence threshold, a user-understandable quality feedback prompt is generated and sent based on the dimension with the smallest value in the image quality vector corresponding to the text image. For example, if the value of the dimension corresponding to Gaussian blur in the image quality vector is the smallest, the quality feedback prompt is output: "It is recommended to retake the picture (the current image is blurry)"; if the value of the dimension corresponding to low light is the smallest, the quality feedback prompt is output: "Please improve the ambient light and retake the picture".

[0085] Specifically, the default reliability threshold is set to 0.5.

[0086] Through the above steps, while outputting the text recognition result, the confidence level of the text recognition result is also obtained. When the confidence level of the text recognition result is lower than the preset confidence threshold, a quality feedback prompt that can be understood by the user is generated and sent based on the dimension with the smallest value in the image quality vector corresponding to the text image. This allows the user to clearly know the specific reason for the recognition failure or error and take effective improvement measures accordingly (such as reshooting or adjusting the lighting), rather than blindly retrying. Moreover, the feedback comes directly from the structured quality perception result, which has strong interpretability and technical consistency, avoids black box output, and is conducive to improving user experience and user satisfaction.

[0087] Specifically, the target feature map corresponding to the text image is expanded into a feature sequence along the width dimension, input into a two-layer bidirectional GRU (Bi-GRU) for context modeling, and then output as a character probability matrix through a CTC decoder; the size of the character probability matrix is ​​Y×Z, where Y is the time step (equal to the width of the target feature map); and Z is the character set size.

[0088] Furthermore, the character probability matrix is ​​decoded by CTC or autoregressive sampling to obtain the text recognition result.

[0089] In another specific embodiment, the target feature map corresponding to the text image is flattened and positional encoding is added. This is then used as input to the Transformer Decoder to generate a text sequence or output a character probability matrix through an autoregressive approach.

[0090] Specifically, after step S6, the following is also included:

[0091] Based on the text recognition results of all text images corresponding to the target image, obtain the text recognition results corresponding to the target image.

[0092] Through the above steps, a structured image quality vector is generated using an image quality-aware network to explicitly characterize the severity of degradation of the input text image across various preset image degradation types. Subsequently, based on the image quality vector, the initial feature map extracted by the image visual feature extraction network is subjected to channel-level adaptive modulation using the FiLM mechanism to obtain an intermediate feature map. Furthermore, based on a spatial attention mask, the intermediate feature map is weighted by region to obtain the target feature map, thereby effectively suppressing severely degradation-affected regions and enhancing the representation ability of reliable character regions. Further, the target feature map corresponding to the text image is sequence-modeled and decoded to obtain the text recognition result corresponding to the text image. This effectively alleviates the negative impact of image degradation on text recognition performance, significantly reduces the character error rate, and improves the accuracy of the text recognition result.

[0093] Specifically, the image-based text recognition method is implemented by a neural network containing learnable parameters, which is optimized during the training phase by minimizing a composite loss function.

[0094] Furthermore, the composite loss function composites L total Meets the following conditions:

[0095] L total =L rec +α1×L quality +α2×L grad L rec For the CTC loss of text recognition tasks, L quality L is the regression loss for the image quality vector. grad This is the gradient scaling penalty term, where α1 is L quality The corresponding preset adjustment weight, α2 is L grad The corresponding preset adjustment weight.

[0096] Specifically, α1=0.2, α2=0.1.

[0097] Specifically, L quality =∑ n i=1 Smooth L1 (q i -q 1 i ).

[0098] Specifically, L grad = e is the scaling factor in the FiLM mechanism, and f is the offset factor in the FiLM mechanism. Let e ​​and f represent the gradients of the image quality vector Q.

[0099] Specifically, during the training process, an online hard example mining (OHEM) strategy is introduced: in each batch, based on L... rec Sort the samples, select the top 30% of the most difficult samples, and calculate L only for these samples. quality .

[0100] Furthermore, the training samples used to train the neural network include a preset low-quality image and a text string corresponding to the preset low-quality image; the text string is obtained through manual annotation.

[0101] Through the above steps, by constructing an end-to-end neural network with learnable parameters and optimizing it with a composite loss function during the training phase, the text recognition accuracy and image quality perception capability are effectively improved in synergy. The composite loss function includes CTC loss, regression loss of image quality vector, and gradient scale penalty term, which can ensure accurate character sequence output, drive the quality perception network to accurately estimate the intensity of various degradations, and prevent FiLM parameter explosion. Furthermore, by adopting an online hard example mining strategy, the quality regression loss is calculated only for the top 30% of hard samples with the highest recognition loss in each batch, which significantly improves prediction accuracy and practicality.

[0102] The present invention also provides a specific embodiment, which differs from the above embodiment in that the text image is input into the target text recognition model to obtain the text recognition result corresponding to the text image.

[0103] Specifically, the target text recognition model includes: an image input module, a quality perception module, a visual feature extraction module, a quality perception fusion module, and a sequence recognition module.

[0104] Specifically, the image input module includes a preprocessing unit and a normalization unit, which are used to generate a normalized RGB tensor corresponding to the input text image.

[0105] Furthermore, the image input module hardware interface supports industrial cameras such as USB3.0, MIPI-CSI, and GigE, with an input resolution range of 640×480 to 3264×2448, and supports 8 / 12 bit RAW or YUV422 formats; the software interface is compatible with OpenCV VideoCapture, V4L2 (Linux), and DirectShow (Windows).

[0106] Specifically, the quality perception module includes an image quality perception network, which generates an image quality vector corresponding to the text image based on the input normalized RGB tensor.

[0107] Specifically, the visual feature extraction module includes an image visual feature extraction network, which is used to generate an initial feature map corresponding to the text image based on the input normalized RGB tensor.

[0108] Specifically, the quality-aware fusion module includes a FiLM modulation unit and a spatial attention modulation unit, which are used to generate a target feature map based on the input image quality vector and the initial feature map.

[0109] Specifically, the sequence recognition module includes a sequence modeling unit and a decoding unit, which are used to generate text recognition results based on the input target feature map.

[0110] Specifically, the target text recognition model in this embodiment is the corresponding implementation device of the aforementioned image-based text recognition method. The functions of each module of the target text recognition model correspond to the steps in the method, and the same training data, loss function and optimization strategy are used. For details, please refer to the description of the training process above.

[0111] Through the above steps, the aforementioned method is specifically implemented as a modular and deployable target text recognition model. It not only achieves quality awareness and adaptive feature modulation at the algorithm level, but also fully considers the needs of edge deployment in engineering implementation. Through ONNX model fusion and TensorRT INT8 quantization, the target text recognition model size is compressed to less than 6MB, which can run efficiently on resource-constrained edge devices (such as NVIDIA Jetson Nano). The end-to-end inference time is only 28ms / frame, the accuracy loss is less than 0.8%, and the real-time performance of 30 FPS is stably achieved.

[0112] Embodiments of the present invention also provide a non-transitory computer-readable storage medium that can be disposed in an electronic device to store a computer program related to implementing a method in the method embodiments, the computer program being loaded and executed by the processor to implement the method provided in the above embodiments.

[0113] Embodiments of the present invention also provide an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method provided in the above embodiments.

[0114] Embodiments of the present invention also provide a computer program product including program code, which, when the program product is run on an electronic device, causes the electronic device to perform the steps of the methods described above in various exemplary embodiments of the present invention.

[0115] While specific embodiments of the invention have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention.

Claims

1. An image-based text recognition method, characterized in that, The method includes the following steps: S1. Preprocess the text image to obtain the normalized RGB tensor corresponding to the text image; the text image is the part of the target image that corresponds to the text region. S2. Based on the normalized RGB tensor and the image quality perception network, generate the image quality vector corresponding to the text image; each dimension of the image quality vector corresponds to a preset image degradation type in the preset image degradation type set, and the value range of each dimension is [0, 1], which is used to represent the image quality level of the text image under the preset image degradation type corresponding to each dimension; The larger the dimension value, the higher the image quality level of the text image under the preset image degradation type corresponding to that dimension; in particular, the higher the image quality level of the text image under the preset image degradation type corresponding to that dimension, the lower the degradation severity of the text image under that preset image degradation type. S3. Based on the normalized RGB tensor and the image visual feature extraction network, obtain the initial feature map corresponding to the text image; S4. Based on the image quality vector corresponding to the text image, the FiLM mechanism is used to modulate the initial feature map corresponding to the text image to obtain the intermediate feature map corresponding to the text image; the feature map T of the c-th channel of the intermediate feature map T2 corresponding to the text image. 2c Meets the following conditions: T 2c =γ c ×Norm(T 1c )+β c , 1≤c≤512; where, T 1c This is the feature map of the c-th channel of the initial feature map T1 corresponding to the text image; Norm() represents the normalization operation; γ c The c-th component of the scaling factor γ corresponds to the c-th channel; β c γ is the c-th component of the offset factor β, corresponding to the c-th channel; γ and β are both real-valued vectors of length 512, and γ and β satisfy the following conditions respectively: γ=U γ ×X+b γ ;β=U β ×X+b β U γ and U β The weight matrix is ​​a learnable matrix with a size of 512×n; b γ and b β is a learnable bias vector, with a length of 512; X is the image quality vector corresponding to the text image; n is the dimension of the image quality vector; S5. Multiply the spatial attention mask element-wise with the intermediate feature map corresponding to the text image to obtain the target feature map corresponding to the text image. The spatial attention mask is obtained based on the image quality vector corresponding to the text image. Broadcast the image quality vector corresponding to the text image to the same spatial dimension as the intermediate feature map corresponding to the text image, and after 1×1 convolution and Sigmoid activation, obtain the spatial attention mask. S6. Perform sequence modeling and decoding on the target feature map corresponding to the text image to obtain the text recognition result corresponding to the text image.

2. The image-based text recognition method according to claim 1, characterized in that, After step S2 and before step S3, the following steps are also included: Based on the image quality vector corresponding to the text image, obtain the image quality score corresponding to the text image; the image quality score F corresponding to the text image meets the following conditions: F=∑ n i=1 (V i ×(1-X i )) / ∑ n i=1 V i V i For X i The corresponding preset importance weight for the preset image degradation type, X i This represents the value of the i-th dimension in the image quality vector corresponding to the text image; 1 ≤ i ≤ n; When the image quality score corresponding to the text image is not less than the preset quality score, proceed to step S3; When the image quality score corresponding to the text image is lower than the preset quality score, a prompt message is generated and sent to the user; the prompt message is used to inform the user that the image quality is low and text recognition is not possible.

3. The image-based text recognition method according to claim 1, characterized in that, The image quality perception network is constructed from a lightweight convolutional neural network, including: Remove the pre-trained classification head of the lightweight convolutional neural network, connect three fully connected layers sequentially after the global average pooling layer of the lightweight convolutional neural network, and apply the Sigmoid activation function after the last fully connected layer to output an image quality vector with values ​​ranging from [0,1] for each dimension.

4. The image-based text recognition method according to claim 1, characterized in that, The preset image degradation type set includes several first-class preset image degradation types and several second-class preset image degradation types. The first-class preset image degradation types are physical degradation types, including at least: blur, Gaussian noise, low illumination, low contrast, color cast, and rain / fog occlusion. The second-class preset image degradation types are degradation types that are not enabled.

5. The image-based text recognition method according to claim 4, characterized in that, The image quality perception network is trained based on a training sample set, which includes several training samples. Each training sample includes a normalized RGB tensor corresponding to a preset low-quality image and a true quality vector corresponding to the preset low-quality image. The preset low-quality image is obtained by randomly performing several degradation operations corresponding to a first type of preset image degradation on a preset high-quality image. The true quality vector is a vector used to represent the distribution of image quality levels of the preset low-quality image under each preset image degradation type. Each dimension of the true quality vector corresponds to a preset image degradation type in the set of preset image degradation types. For each dimension of the true quality vector corresponding to the preset low-quality image, the dimension value ranges from [0, 1], representing the image quality level of the preset low-quality image under the preset image degradation type corresponding to that dimension. The larger the value of the dimension, the higher the image quality level of the preset low-quality image under the preset image degradation type corresponding to that dimension; the smaller the value of the dimension, the lower the image quality level of the preset low-quality image under the preset image degradation type corresponding to that dimension. In particular, the lower the image quality level of the preset low-quality image under the preset image degradation type corresponding to that dimension, the higher the degradation severity of the preset low-quality image under that dimension; the higher the image quality level of the preset low-quality image under the preset image degradation type corresponding to that dimension, the lower the degradation severity of the preset low-quality image under that dimension. The specific values ​​of each dimension in the true quality vector corresponding to the preset low-quality image are obtained in the following way: For each preset image degradation type, if a degradation operation corresponding to the preset image degradation type is performed on the preset high-quality image during the process of obtaining the preset low-quality image from the preset high-quality image, then the difference obtained by subtracting the image degradation parameter corresponding to the degradation operation from 1 is used as the specific value of the dimension corresponding to the preset image degradation type in the true quality vector; the image degradation parameter corresponding to the degradation operation is the value obtained by normalizing the intensity parameter of the degradation operation to the [0, 1] interval according to the preset maximum intensity parameter corresponding to the degradation operation; For each preset image degradation type, if no degradation operation corresponding to the preset image degradation type is performed on the preset high-quality image during the process of obtaining the preset low-quality image from the preset high-quality image, then the specific value of the dimension corresponding to the preset image degradation type in the real quality vector is set to 1.

6. The image-based text recognition method according to claim 5, characterized in that, The loss function L used to train the image quality sensing network satisfies the following condition: L=∑ n i=1 Smooth L1 (q i -q 1 i );q i q represents the value of the i-th dimension in the image quality vector obtained from the normalized RGB tensor corresponding to the predefined low-quality image and the image quality perception network; 1 i Let q be the value of the i-th dimension in the true quality vector corresponding to the preset low-quality image; where q i The corresponding preset image degradation type and q 1 i The corresponding preset image degradation type is consistent; 1≤i≤n, where n is the dimension of the image quality vector; the dimension of the real quality vector is consistent with the dimension of the image quality vector.

7. The image-based text recognition method according to claim 1, characterized in that, The steps preceding step S1 also include: Perform text detection on the target image, identify at least one text region, and obtain the location information corresponding to each text region; For each text region, based on the location information corresponding to the text region, a portion of the image corresponding to that text region is cropped from the target image, and this portion of the image is used as the text image.

8. The image-based text recognition method according to claim 1, characterized in that, The process after step S6 also includes: Based on the text recognition results of all text images corresponding to the target image, obtain the text recognition results corresponding to the target image.

9. A non-transitory computer-readable storage medium, characterized in that, The storage medium stores a computer program, which is loaded and executed by a processor to implement the image-based text recognition method as described in any one of claims 1-8.

10. An electronic device, comprising: A processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the image-based text recognition method as described in any one of claims 1-8.

Images