Optical character recognition method, device and equipment

Abstract

The invention relates to the technical field of image processing, in particular to an optical character recognition method, device and equipment, and the method comprises the steps: obtaining a to-be-recognized target character image containing multiple languages, and carrying out the preprocessing of the target character image, and obtaining a pure character image; wherein the preprocessing comprises the step of carrying out noise reduction processing on the target character image by adopting a self-adaptive noise suppression module; obtaining a voice description corresponding to the target character image to obtain voice modal data, and obtaining a text label pre-labeled for the target character image to obtain label modal data; and inputting the voice modal data and / or the label modal data and the pure character image into a trained multi-modal character recognition model for recognition to obtain target character characters contained in the target character image. Therefore, the problem that an existing OCR recognition technology is low in recognition precision in complex scenes such as multiple languages can be effectively solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an optical character recognition method, device and equipment. Background Art

[0002] Currently, OCR (Optical Character Recognition) recognition scenes (such as ID cards, faces, bills, etc.) are one of the important components of financial industry business. In the prior art, the following methods are generally used for OCR recognition.

[0003] 1. Machine Learning Methods:

[0004] It mainly relies on manually designed feature extraction algorithms (such as edge detection, connected component analysis, etc.) and classifiers (such as support vector machines, decision trees, etc.) to recognize text images. Feature engineering is time-consuming and relies on domain expertise. It has poor adaptability to complex backgrounds, font changes, uneven lighting, etc., and the recognition accuracy is limited by the quality of the features. Multi-language support requires training multiple models separately, and has poor scalability.

[0005] 2. Deep Learning Methods:

[0006] It mainly uses a single deep neural network structure (such as fully connected network, simple CNN) for text recognition, and then cooperates with preprocessing steps (such as binarization, tilt correction). It is insufficient in capturing the spatial structure and sequence information of text images, and is sensitive to interference factors such as complex background, noise, and blur. It lacks an effective cross-language learning mechanism. Every time a new language recognition requirement is added, a large amount of labeled data needs to be collected again to train a separate model, which is costly and inefficient.

[0007] In short, OCR recognition in complex situations such as multiple languages ​​currently has the problem of low character target recognition accuracy. Summary of the invention

[0008] In view of this, the embodiments of the present application provide an optical character recognition method, device and equipment, which can effectively solve the problem of low recognition accuracy of existing OCR recognition technology in complex scenarios such as multiple languages.

[0009] In a first aspect, an embodiment of the present application provides an optical character recognition method, comprising:

[0010] Acquire a target text image containing multiple languages ​​to be recognized, and preprocess the target text image to obtain a pure text image; wherein the preprocessing includes using an adaptive noise suppression module to perform noise reduction processing on the target text image;

[0011] Acquire a voice description corresponding to the target text image to obtain voice modal data, and acquire a text label pre-labeled for the target text image to obtain label modal data;

[0012] The speech modal data and / or the label modal data, and the clean text image are input into a trained multimodal character recognition model for recognition to obtain target text characters contained in the target text image.

[0013] In some embodiments, the adaptive noise suppression module is a U-Net network; the U-Net network includes an encoder, a decoder and a local analysis function unit;

[0014] The adopting the adaptive noise suppression module to perform noise reduction processing on the target text image includes:

[0015] Downsampling the target text image through each downsampling layer in the encoder to obtain a corresponding first feature map;

[0016] The local analysis function unit is used to calculate the noise evaluation parameter of each pixel in each of the first feature maps, and the noise area is determined according to the noise evaluation parameter and the set noise threshold, and the noise area is marked to obtain each noise mask; wherein the noise evaluation parameter includes one or more combinations of mean, standard deviation and gradient information;

[0017] Through each upsampling layer in the decoder, the output feature of the current upsampling layer is determined according to the corresponding noise mask, the corresponding first feature and the image feature of the previous layer input of the current upsampling layer.

[0018] In some embodiments, before preprocessing the target text image, the method further includes:

[0019] The U-Net network is iteratively trained, wherein a cosine annealing strategy is used to adjust the learning rate during the iterative training process; and the termination condition of the iterative training is that a peak signal-to-noise ratio obtained by the iterative training is greater than a set signal-to-noise ratio threshold, and a structural similarity index obtained by the iterative training is greater than a set index threshold.

[0020] In some embodiments, the multimodal character recognition model includes a convolutional neural network module, a text embedding layer, an attention mechanism layer, and a recurrent neural network module connected in sequence;

[0021] The step of inputting the voice modal data and / or the label modal data, and the clean text image into a trained multimodal character recognition model for recognition to obtain target text characters includes:

[0022] The clean text image is input into the convolutional neural network module to obtain a visual feature map; the text data obtained by converting the speech modal data through speech recognition technology and the label modal data are respectively input into the text embedding layer to obtain a text feature map of the corresponding modality; the visual feature map and each of the text feature maps are input into the attention mechanism layer to calculate the attention weight of the feature map received by each channel; the feature maps are weighted and summed according to the attention weight of each channel to obtain a fused feature vector; the fused feature vector is input into the recurrent neural network module to extract context information and perform character recognition to obtain the target text character.

[0023] In some embodiments, the method further comprises:

[0024] The multimodal character recognition model is structurally optimized and adjusted using a preset adjustment strategy according to the iterative training result of the multimodal character recognition model.

[0025] In some embodiments, the recurrent neural network module includes a Bi-LSTM layer;

[0026] The method of optimizing the structure of the multimodal character recognition model by using a preset adjustment strategy according to the iterative training result of the multimodal character recognition model comprises:

[0027] If the iterative training result shows that the multimodal character recognition model is underfitting, a preset number of convolutional layers, a preset number of filters or a change in the filter size is added to the convolutional neural network module, a preset number of Bi-LSTM layers are stacked after the Bi-LSTM layer in the recurrent neural network module, or a temporal convolutional network layer is added before the Bi-LSTM layer;

[0028] Among them, if the difference between the set expected accuracy rate and the training accuracy rate obtained when training using multilingual training samples is greater than a set first difference value, and the absolute difference between the verification accuracy rate obtained when verifying using multilingual verification samples and the training accuracy rate is less than a set second difference value, it is determined that the multimodal character recognition model has an underfitting phenomenon.

[0029] In some embodiments, performing structural optimization adjustment on the multimodal character recognition model using a preset adjustment strategy according to the iterative training result of the multimodal character recognition model includes:

[0030] Unfreezing the output layer and the preset task layer of the multimodal character recognition model to update parameters;

[0031] Freezing the task layers other than the output layer and the specified task layer in the multimodal character recognition model;

[0032] If the difference between the character error rate obtained by the current iterative training and the character error rate obtained by the previous iterative training is less than the set error rate threshold and the word error rate obtained by the current iterative training is less than the word error rate obtained in the previous iterative cycle, or the gradient values ​​of the specified task layer within N iterative cycles are all less than the set gradient standard, then the next task layer is unfrozen.

[0033] In some embodiments, the step of optimizing the structure of the multimodal character recognition model using a preset adjustment strategy according to the iterative training result of the multimodal character recognition model includes:

[0034] If the character error rate and the word error rate in the iterative training result are both greater than the set maximum error rate threshold, or the training accuracy in the iterative training result is lower than the accuracy difference set by the expected accuracy, then the type of the erroneous character or grammatical structure is determined according to the character error rate and the word error rate, and a data sample corresponding to the type of the erroneous character or grammatical structure is added to the multilingual training sample;

[0035] If the state of the loss curve in the iterative training result changes, the learning rate is adjusted using a learning rate decay strategy.

[0036] In a second aspect, an embodiment of the present application provides an optical character recognition device, comprising:

[0037] A preprocessing module, used to obtain a target text image to be recognized, and preprocess the target text image to obtain a pure text image; wherein the preprocessing includes using an adaptive noise suppression module to perform noise reduction processing on the target text image;

[0038] A multimodal data acquisition module, used to acquire the voice description corresponding to the target text image to obtain voice modal data, and to acquire the text label annotated on the target text image to obtain label modal data;

[0039] The target text character recognition module is used to input the voice modal data and / or the label modal data, and the clean text image into a trained multimodal character recognition model for recognition to obtain the target text characters.

[0040] In a third aspect, an embodiment of the present application provides an optical character recognition device, the optical character recognition device comprising a processor and a memory, the memory storing a computer program, and the processor being used to execute the computer program to implement an optical character recognition method provided in the first aspect of the present application.

[0041] The embodiments of the present application have the following beneficial effects:

[0042] The present application uses an adaptive noise suppression module to perform noise reduction processing on target text images in multiple languages. Then, based on the speech description corresponding to the target text image, speech modal data is obtained, and the text label pre-labeled for the target text image is obtained to obtain label modal data; the speech modal data and / or the label modal data, as well as the pure text image are input into a trained multimodal character recognition model for recognition to obtain the target text characters contained in the target text image. The present application first removes the noise in the target text image, and considers multiple features, thereby effectively solving the problem of low recognition accuracy of existing OCR recognition technology in complex scenarios such as multiple languages. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A flow chart of an optical character recognition method according to an embodiment of the present application is shown;

[0045] Figure 2 A diagram showing a process of performing noise reduction processing on the target text image by using an adaptive noise suppression module in the optical character recognition method of an embodiment of the present application;

[0046] Figure 3 A structural block diagram of a multimodal character recognition model in an optical character recognition method according to an embodiment of the present application is shown;

[0047] Figure 4 A processing diagram of a multimodal character recognition model in an optical character recognition method according to an embodiment of the present application is shown;

[0048] Figure 5 A flowchart of constructing a multimodal character recognition model in an optical character recognition method according to an embodiment of the present application is shown;

[0049] Figure 6 The first structural optimization adjustment method in the optical character recognition method of the embodiment of the present application is shown;

[0050] Figure 7 The second structural optimization adjustment method in the optical character recognition method of the embodiment of the present application is shown;

[0051] Figure 8 The third structural optimization adjustment method in the optical character recognition method of the embodiment of the present application is shown;

[0052] Fig. 9 A schematic structural diagram of an optical character recognition device according to an embodiment of the present application is shown.

[0053] Description of main component symbols:

[0054] 100-convolutional neural network module; 200-text embedding layer; 300-attention mechanism layer; 400-recurrent neural network module; 910-preprocessing module; 920-multimodal data acquisition module; 930-target text character recognition module. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0056] The components of the embodiments of the present application generally described and shown in the drawings herein may be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0057] Hereinafter, the terms "including", "having" and their cognates used in various embodiments of the present application are intended only to indicate specific features, numbers, steps, operations, elements, components or a combination of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or a combination of the foregoing items or increasing the possibility of one or more features, numbers, steps, operations, elements, components or a combination of the foregoing items. In addition, the terms "first", "second", "third" and the like are only used to distinguish descriptions and cannot be understood as indicating or implying relative importance.

[0058] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present application.

[0059] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0060] The present application provides an optical character recognition method, device and equipment, which can effectively solve the problem of low recognition accuracy of existing OCR recognition technology in complex scenarios such as multiple languages.

[0061] The optical character recognition method is described below in conjunction with some specific embodiments.

[0062] Figure 1 A flow chart of an optical character recognition method according to an embodiment of the present application is shown. Exemplarily, the optical character recognition method comprises the following steps:

[0063] S110, obtaining a target text image to be recognized that contains multiple languages, and preprocessing the target text image to obtain a pure text image; wherein the preprocessing includes using an adaptive noise suppression module to perform noise reduction processing on the target text image.

[0064] It can be understood that before obtaining the target text image containing multiple languages ​​to be recognized, the process also includes: preparing a multi-language sample set, and training the adaptive noise suppression module.

[0065] Specifically, making a multilingual sample set includes the following steps:

[0066] (1) Collect diverse text images. Specifically, we collect a wide range of text image data in various fonts and styles (handwritten and printed) of each target language to ensure that the multilingual sample set covers all characters and languages.

[0067] (2) Perform data enhancement on text images to obtain more diverse training data. For example, use TensorFlow’s data enhancement API to rotate, flip, scale, add noise, and other operations on the original image to increase the diversity of training data and prevent overfitting.

[0068] (3) Setting labels for text images. Setting labels for text images includes pairing each text image with a corresponding text label (a text label that generally describes the text image) and establishing a mapping between the text image and its corresponding character sequence.

[0069] Batching and slicing text images. Divide the labeled text images (multilingual sample set) into a multilingual training sample set, a multilingual validation set, and a multilingual test set, usually in a ratio of 8:1:1, and divide each data subset into small batches for training.

[0070] When using a multilingual sample set to train an adaptive noise suppression module, and in the optical character recognition process, the target text image or the training subset needs to be preprocessed, specifically including:

[0071] The target text image or the training subset is subjected to at least one of grayscale conversion, binarization, size standardization, image cleaning, etc.

[0072] Exemplarily, the image cleaning process includes denoising, cropping, and boundary detection using the OpenCV library to remove impurities in the image, such as spots, borders, etc. Grayscale processing includes converting a color image into a grayscale image to reduce computational complexity. For example, the cvtColor function of OpenCV is used to convert a color image into a grayscale image to reduce computational complexity. Binarization processing includes enhancing the contrast between text and background, for example, using the OTSU algorithm to automatically determine the threshold, and achieving contrast adjustment through the threshold function of OpenCV. Size standardization processing includes adjusting the image to a standard size, such as uniformly adjusting it to 32×128 pixels. Among them, the aspect ratio is kept unchanged, and high-quality scaling methods such as bicubic interpolation are used.

[0073] Furthermore, preprocessing the target text image or the training subset also includes: using an adaptive noise suppression module to perform noise reduction processing on the target text image or the training subset.

[0074] Exemplarily, the adaptive noise suppression module is a U-Net network. The U-Net network is used to identify and reduce noise in the target text image to improve the purity of the text area. The U-Net network includes an encoder, a decoder, a bottleneck layer and a local analysis function unit.

[0075] Specifically, each layer in the encoder of the U-Net network consists of two consecutive 3×3 convolution kernels, and each convolution layer is followed by BatchNorm to accelerate convergence and improve stability, as well as ReLU activation function. For example, the first layer can have 64 filters, and the number of filters is doubled in each downsampling layer.

[0076] Among them, the downsampling layer uses the maximum pooling unit for downsampling, and the pooling kernel size is usually 2×2 with a step size of 2 to reduce the spatial dimension and extract higher-level features.

[0077] Bottleneck layer: Design one or several convolutional layers with fewer filters, such as 128 filters, to maintain the depth of the feature map but reduce the computational burden.

[0078] In the decoder, transposed convolution is used for upsampling to restore the resolution of the feature map. In each upsampling layer, the number of filters is halved and skip connections are made with the corresponding layer features of the encoder.

[0079] The skip connection is used to concatenate or weighted sum the output of the corresponding layer of the encoder with the upsampled features of the decoder to retain more detailed information.

[0080] Furthermore, before preprocessing the target text image, the method further includes:

[0081] (i) The U-Net network is iteratively trained, wherein the loss function selects MSE (mean square error) as the basic loss to measure the pixel difference between the predicted image and the real image, and then combines SSIM (structural similarity index) loss to consider the structural information of the image to avoid blurry reconstruction. In addition, edge preservation loss can be introduced to encourage the model to keep the image edges clear.

[0082] (ii) Perform data augmentation on the training subset, that is, add various types of noise to the training subset. For example, add different types of noise to the original clean text image data to generate noisy text images as additional training samples to simulate various image degradation conditions that may be encountered in the real world, such as Gaussian noise, salt and pepper noise, periodic noise and different intensities, so as to improve the generalization ability of the model.

[0083] (iii) Perform random rotation, scaling, shearing and other transformations on the text images in the training subset to increase data diversity.

[0084] During the iterative training process, the initial learning rate can be set to 1e-4, and the cosine annealing strategy is used to adjust the learning rate, periodically reducing it from the initial value to the minimum value to promote convergence. The termination condition of the iterative training is: the peak signal-to-noise ratio obtained by the iterative training is greater than the set signal-to-noise ratio threshold, and the structural similarity index obtained by the iterative training is greater than the set index threshold.

[0085] Furthermore, after the U-Net network is trained, the method further includes: evaluating the U-Net network. Using PSNR (peak signal-to-noise ratio) and SSIM (structural similarity index) to evaluate the denoising effect and reflect the image fidelity. Comparing the recognition accuracy of the multimodal character recognition model (OCR), especially the improvement of the recognition rate in a noisy environment.

[0086] Among them, a higher PSNR value (greater than 30dB) generally means better denoising effect, high image quality, visually close to the original image, and relatively complete details. A medium PSNR value (about 20-30dB) indicates that there is a certain degree of image quality loss, and denoising may be accompanied by some blurred details or slight distortion. A lower PSNR value (below 20dB) indicates that the denoising effect is poor, and there may be obvious distortion or noise residue in the image.

[0087] An SSIM value close to 1 indicates that the denoised image is highly similar in structure to the original image, and can well preserve the image's structure and texture information even when the pixel values ​​change. A medium SSIM value (about 0.7-0.9) indicates that the images are somewhat similar, but there may be a loss of some structure or details. An SSIM close to 0 or a negative value means that the structure between the images is very different, the denoising effect is poor, or the image is severely distorted.

[0088] like Figure 2 As shown, the adaptive noise suppression module is used to perform noise reduction processing on the target text image, including:

[0089] S111, downsampling the target text image through each downsampling layer in the encoder to obtain a corresponding first feature map.

[0090] S112, using the local analysis function unit to calculate the noise evaluation parameters of each pixel point in each of the first feature maps, determining the noise area according to the noise evaluation parameters and the set noise threshold, marking the noise area, and obtaining each noise mask; wherein the noise evaluation parameters include one or more combinations of mean, standard deviation and gradient information.

[0091] S113, through each upsampling layer in the decoder, according to the corresponding noise mask, the corresponding first feature and the image feature of the previous layer input of the current upsampling layer, determine the output feature of the current upsampling layer. Specifically, the first feature obtained by each downsampling layer in the jump connection is multiplied by the corresponding noise mask as an input of the current upsampling layer, and then combined with the image feature of the previous layer input of the current upsampling layer, the output feature of the current upsampling layer is determined, or the noise mask is directly applied on the decoder specific layer to adjust the feature response, focusing on correcting the noise area. Among them, the decoder specific layer can be a normalization layer (Normalization Layer) or a convolution layer, which is used to accelerate training and improve the stability of the model. When the noise mask is applied, the decoder specific layer will adjust its feature response according to the mask to ensure that the clarity and details of the text area are maintained while denoising. Specifically, if a region is marked as noise, then on the corresponding decoder layer, the feature response of the region may be suppressed, while the feature response of the text area will be strengthened or remain unchanged, so as to improve the purity and readability of the final output image.

[0092] S120, obtaining a voice description corresponding to the target text image to obtain voice modal data, and obtaining a text label pre-labeled for the target text image to obtain label modal data.

[0093] In the embodiment of the present application, voice modal data and label modal data are also considered to improve the accuracy of optical character recognition. In some scenarios, the voice description corresponding to the target text image is obtained, and the correlation between the voice description and the target text image is extracted. For example, in the guided tour application of the museum, while the user takes a photo of a painting, the application also records the description of the painting by the nearby interpreter. This voice description may include information such as the name of the painting, the author, the background of creation, and the details of the story in the painting. In the optical character recognition method of multiple languages, after the voice description is converted into text by voice recognition technology, it can be combined with the visual features extracted from the target text image to improve the recognition accuracy. For example, the text in the target text image is difficult to recognize due to its age, ambiguity or artistic fonts. The features extracted from the voice description can be used for fusion recognition, so as to more accurately recognize and understand the text content in the target text image. For example, when confirming the name of a painting or an illegible inscription. Voice description can be of great help.

[0094] In the scenario of social media or e-commerce platforms, the correlation between text tags and target text images is used to further improve recognition accuracy. For example, when a user uploads a product picture, he or she often enters some text tags or descriptions. The text tags directly reflect the subject content of the product picture. Although the text tags do not appear directly in the text area of ​​the product picture, they are closely related to the visual features in the product picture. In the optical character recognition method of multiple languages, the text tag can be regarded as a kind of prior knowledge to guide the multimodal character recognition model of the present application to find and confirm specific text patterns in the target text image. For example, if there is "retro" in the text tag, the embodiment of the present application may pay more attention to the old fonts or symbols of a specific era that may appear in the target text image, thereby improving the accuracy of identifying such texts.

[0095] S130, inputting the speech modal data, the label modal data, and the clean text image into a trained multimodal character recognition model for recognition, and obtaining the target text characters contained in the target text image. In other embodiments, the speech modal data or the label modal data, and the clean text image are input into a trained multimodal character recognition model for recognition, and obtaining the target text characters contained in the target text image.

[0096] Furthermore, if Figure 3 As shown, the multimodal character recognition model includes a convolutional neural network module 100, a text embedding layer 200, an attention mechanism layer 300 and a recurrent neural network module 400 which are connected in sequence.

[0097] In step S130, the speech modality data, the label modality data, and the clean text image are input into a trained multimodal character recognition model for recognition to obtain target text characters, such as Figure 4 As shown, the following steps are included:

[0098] S131, input the clean text image into the convolutional neural network module 100 to obtain a visual feature map.

[0099] S132, the text data obtained by converting the voice modal data through voice recognition technology and the label modal data are respectively input into the text embedding layer 200 to obtain a text feature map of the corresponding modality.

[0100] S133, input the visual feature map and each of the text feature maps into the attention mechanism layer 300 to calculate the attention weight of the feature map received by each channel. In the embodiment of the present application, the attention mechanism layer 300 is used for feature fusion.

[0101] S134, performing weighted summation on each of the feature maps according to the attention weight of each channel to obtain a fused feature vector.

[0102] S135, inputting the fused feature vector into the recurrent neural network module 400 to extract context information and perform character recognition to obtain target text characters.

[0103] Before inputting the speech modal data, the label modal data, and the clean text image into a trained multimodal character recognition model for recognition in step S130 to obtain the target text characters contained in the target text image, the method of the embodiment of the present application further includes:

[0104] Build a multimodal character recognition model and train a multimodal character recognition model.

[0105] like Figure 5 As shown, the following steps S210 to S250 corresponding to the steps are used to construct a multimodal character recognition model:

[0106] S210, extracting a visual feature map from the clean text image using a convolutional neural network module 100 (CNN), for example, using a ResNet50 model to extract deep visual features, and the pre-trained weights of ResNet50 can effectively identify various image features.

[0107] Among them, the method of constructing the CNN module includes:

[0108] S211, import library, that is, install the TensorFlo library and import related modules.

[0109] S212, use tf.keras.applications.ResNet50 to load the pre-trained ResNet50 model and remove the top fully connected layer.

[0110] S213, adjust the output size of the last layer as needed to match the input dimension of the subsequent RNN. That is, the feature map size output by the ResNet50 model before the top fully connected layer (fc1000) will be converted into a 1×1×2048 tensor through global average pooling (Global Average Pooling, GAP), where 2048 refers to the dimension of the ResNet50 feature. It can be understood that the two-dimensional feature is flattened into a one-dimensional vector, and then adjusted to the expected input size of the RNN module through a fully connected layer (Dense).

[0111] S220, for the text data obtained by converting the voice modal data through the voice recognition technology, a text embedding model (text embedding layer 200) is used to extract features, and the label modal data is also extracted by a text embedding module.

[0112] S230, the visual features and the text features corresponding to the speech modal data and the text labels are fused through a feature fusion layer. For example, the visual features extracted by ResNet50 are combined with the text features of other modalities through a fully connected layer, an attention mechanism or other fusion strategies to form a richer and multi-dimensional feature representation.

[0113] In this embodiment of the present application, the attention mechanism layer 300 receives the CNN feature map as input, calculates the attention weight of each feature map channel through a series of fully connected layers and activation function ReLU, and then performs weighted summation on the attention weights of the feature map to obtain the fused feature vector. The implementation is constructed using the TensorFlow API.

[0114] For example, the last convolutional layer of the CNN model (based on ResNet) outputs a feature map of size (height, width, channels), where channels represents the number of channels in the feature map. Taking ResNet50 as an example, the feature map size output by its last convolutional layer is (7, 7, 2048), which means there are 2048 feature channels, each of which represents the response strength of the image in different feature dimensions. In the attention mechanism, attention weights are not directly "set", but learned through a neural network. The specific operation is as follows:

[0115] First, global average pooling (GAP) is applied to each channel to transform the (7, 7, 2048) feature map into (1, 1, 2048), so that each channel becomes a scalar representing the average response of the channel over the entire feature map.

[0116] Secondly, through one or more fully connected layers (FC), such as two fully connected layers, the first FC layer can be used to reduce the dimension, the second is used to restore to the original number of channels, and use an activation function (such as ReLU) to calculate the attention weight of each channel. For example, the first FC layer can reduce the 2048 dimension to a smaller dimension (such as 128), and the second FC restores it to 2048, and each channel outputs an attention weight.

[0117] Finally, to ensure that the sum of the weights is 1, the Softmax function can be used to normalize the weights of the outputs corresponding to each channel, so that each channel has an attention weight between 0 and 1, indicating its relative importance.

[0118] The feature map of each channel is multiplied by its corresponding attention weight, and then the results of all channels are summed to obtain a weighted feature map with the same number of channel features. In order to adapt the input of the recurrent neural network module 400 (RNN), it is usually necessary to flatten the weighted feature map into a one-dimensional vector, i.e. (channels*height*width), so as to adapt the fused weighted feature vector to the input format of the RNN.

[0119] S240, using a recurrent neural network module 400 (RNN) to perform text sequence recognition and extract context information of the text. For example, using a Bi-LSTM layer, the fused features are input into a bidirectional Bi-LSTM layer to capture the context information of the text sequence and enhance the ability to understand complex text structures. Finally, an output layer is added, that is, a fully connected layer through a Softmax activation function to generate a probability distribution of characters for character recognition.

[0120] Among them, the steps corresponding to S241 to S245 are used to construct the Bi-LSTM layer:

[0121] S241, use tf.keras.layers.Bidirectional to wrap tf.keras.layers.LSTM and set an appropriate number of units, such as 128.

[0122] S242, stack multiple LSTM layers as needed to increase the expressive power of the model.

[0123] S243, add a flattening layer. Add a flattening layer (Flatten) between the CNN module and the Bi-LSTM layer to adapt to the transformation of the data shape. For example, flatten the weighted feature map into a one-dimensional vector, that is, (channels*height*width).

[0124] S244, add a fully connected layer (Dense) for feature integration. For example, the input of Bi-LSTM requires a fixed-length vector, and a fully connected layer (Dense Layer) can be used to further adjust the size of the flattened vector. For example, if the expected input of Bi-LSTM is 512-dimensional, and the flattened input is 1568-dimensional (taking the feature map of (2048,7,7) as an example), a Dense layer is used to map the 1568-dimensional to 512-dimensional.

[0125] S245, the activation function uses softmax to output the probability distribution of characters.

[0126] S250, multimodal character recognition model assembly. Use tf.keras.Sequential or functional API to connect the above layers in sequence or function to form a complete end-to-end multimodal character recognition model.

[0127] In one embodiment, the optical character recognition method of the embodiment of the present application further includes:

[0128] The multimodal character recognition model is iteratively trained, and a preset adjustment strategy is used to perform structural optimization adjustment on the multimodal character recognition model according to the iterative training result of the multimodal character recognition model.

[0129] In the iterative training process, a multi-language sample set is used to train a multimodal character recognition model, with the goal of minimizing the cross entropy loss at the character level. Iterative training uses a smaller learning rate (such as 1e-5 at the beginning), and then adopts a learning rate decay strategy (such as exponential decay, cosine annealing, etc.), or tries an adaptive learning rate algorithm (such as AdamW, Nadam) to adjust the learning rate; adjust the batch size, larger batches can speed up training, but may require more memory; smaller batches help the model converge, but the training speed is slower; increase L1 / L2 regularization to reduce overfitting, or use dropout in certain layers to randomly discard some neurons; try different optimizers based on model performance, such as RMSprop, Adagrad, etc., to see if it can improve training efficiency or model performance; adjust the number of Bi-LSTM units to find the optimal hidden layer size, too much may lead to overfitting, and too little may not be enough to capture complex features.

[0130] In addition, you need to record the configuration and results of each training in detail, use version control tools to manage the code, and facilitate backtracking and comparison of the effects of different configurations. Use TensorBoard or other visualization tools to monitor the training process and observe changes in loss rate and accuracy.

[0131] Exemplarily, when the recurrent neural network module 400 includes a Bi-LSTM layer, the multimodal character recognition model is structurally optimized and adjusted using a preset adjustment strategy according to the iterative training result of the multimodal character recognition model, such as Figure 6 As shown, including:

[0132] S310, if the iterative training result shows that the multimodal character recognition model is under-fitting, then a preset number of convolutional layers, the number of filters, or the filter size is increased in the convolutional neural network module 100, a preset number of Bi-LSTM layers are stacked after the Bi-LSTM layer in the recurrent neural network module 400, or a temporal convolutional network layer is added before the Bi-LSTM layer to improve the expressive power of the model.

[0133] S320, if the difference between the set expected accuracy rate and the training accuracy rate obtained when training using the multilingual training sample set is greater than the set first difference value, and the absolute difference between the verification accuracy rate obtained when verifying using the multilingual verification sample set and the training accuracy rate is less than the set second difference value, it is determined that the multimodal character recognition model has an underfitting phenomenon.

[0134] The training accuracy is significantly lower than the expected accuracy. For example, the expected accuracy of a simple task is at least 80%, while the actual training accuracy is only 60%-70%, and the verification accuracy is observed to be close to the training accuracy. It can be considered that the multimodal character recognition model is underfitting. When the multimodal character recognition model is underfitting, that is, the training accuracy and verification accuracy are both low, and the difference between the two is not large, consider increasing the depth of the multimodal character recognition model. The specific number of layers to be added depends on the depth and complexity of the current multimodal character recognition model, as well as resource constraints. In general, you can first try to insert additional convolutional layers in the CNN part, increase the number of filters or change the filter size to capture more levels of features. In the RNN part, you can stack another layer of Bi-LSTM after the Bi-LSTM, or add a temporal convolutional network (TCN) before the LSTM to capture changes in the performance of the long-term dependent observation model.

[0135] Furthermore, the multimodal character recognition model is structurally optimized and adjusted using a preset adjustment strategy according to the iterative training result of the multimodal character recognition model, such as Figure 7 As shown, it also includes:

[0136] S330, if the character error rate and word error rate in the iterative training result are both greater than the set maximum error rate threshold, or the training accuracy in the iterative training result is lower than the accuracy difference set by the expected accuracy, then the type of the erroneous character or grammatical structure is determined according to the character error rate and word error rate, and the data sample corresponding to the type of the erroneous character or grammatical structure is added to the multilingual sample set. For example, if it is found that the multimodal character recognition model has a high error rate when processing characters or grammatical structures that are specific to certain languages ​​(CER, WER reaches 20% or the target accuracy is 90%, and the actual accuracy reaches 80%), this may mean that the model lacks the ability to adapt to these features, and the enhanced adaptation layer that needs to be enhanced can help the model learn the specific patterns of the new language more effectively, reduce specific types of errors, such as the misrecognition rate of specific characters, and thus improve the overall recognition performance.

[0137] Among them, the character error rate (CER) is the ratio of the number of incorrectly recognized characters in the recognized text to the total number of characters. The calculation formula is: [CER = frac{(S+D+I)}{N}], where frac represents a function that takes the decimal part of a real number, S is the number of substitution errors (Substitutions), D is the number of deletion errors (Deletions), I is the number of insertion errors (Insertions), and N is the total number of characters in the reference text. The value range of CER is [0,1], where 0 means perfect recognition without errors, and 1 means that all characters are incorrectly recognized or not recognized.

[0138] The word error rate (WER) is similar to CER, but the error is calculated in units of words. The formula is: [WER = frac{(S+D+I)}{W}], where S, D, and I are the same as above, and W is the total number of words in the reference text. The value range is also [0,1], and the smaller the value, the better the performance.

[0139] S340, if the state of the loss curve in the iterative training result changes, the learning rate is adjusted using a learning rate decay strategy. For example, the learning rate is dynamically adjusted during the fine-tuning process, and a larger learning rate (1e-3 to 1e-2 (i.e., 0.001 to 0.1) is used in the initial stage to help the model quickly explore the solution space in the early stages of training and approach a good solution as soon as possible. For complex models or large data sets) quickly adjust key parameters, and gradually reduce the learning rate as the training progresses to fine-tune the model parameters. However, by monitoring the training loss and performance changes on the validation set, when the loss curve tends to stabilize or begins to fluctuate, it indicates that the learning rate may need to be adjusted. Converge quickly to a better solution in the early stage, and adjust slowly in the later stage to avoid overfitting and achieve better generalization ability.

[0140] Furthermore, according to the iterative training results of the multimodal character recognition model, a preset adjustment strategy is adopted to perform structural optimization adjustment on the multimodal character recognition model, such as Figure 8 As shown, it also includes:

[0141] S350, unfreeze the output layer and the preset task layer of the multimodal character recognition model to update parameters.

[0142] S360, freezing the task layers other than the output layer and the specified task layer in the multimodal character recognition model.

[0143] S370, if the difference between the character error rate obtained by the current iterative training and the character error rate obtained by the previous iterative training is less than the set error rate threshold and the word error rate obtained by the current iterative training is less than the word error rate obtained in the previous iterative cycle, or the gradient values ​​of the specified task layer within N iterative cycles are all less than the set gradient standard, then unfreeze the next task layer.

[0144] Specifically, in the early stages of fine-tuning, only the top layers of the multimodal character recognition model (usually the output layer and possible task-specific layers) are allowed to update parameters, while the underlying feature extraction layers remain frozen. As training progresses, more network layers are gradually unfrozen to allow deeper layers of the model to participate in learning. It is observed that the performance improvement of the multimodal character recognition model on the validation set begins to slow down (the CER and WER decreases for 5-10 consecutive rounds of validation set performance improvement of less than 0.1%, and the next layer can be unfrozen), or the gradient of a specific layer changes significantly (the standard of the gradient for several consecutive epochs is less than 1e-4), indicating that the currently unfrozen layer has basically adapted to the new language dataset. In the early stages, the model quickly adapts to the surface features of the new language; in the later stages, as more layers are unfrozen, the model can gradually absorb the deep structural information of the new language, further improving the recognition accuracy.

[0145] The present application also includes a cross-language transfer learning module, which includes: a cross-modal knowledge distillation unit, which constructs a multimodal teacher model that can process speech, text and image data simultaneously. For example, the image data is a pure text image obtained by processing an adaptive noise suppression module. By training on a large-scale data set containing multiple languages, the multimodal teacher model can learn rich language features and visual features. The rich language features and visual features are passed to the student model, i.e., the OCR recognition model (multimodal character recognition model), through knowledge distillation. This process not only strengthens the OCR recognition model's understanding of character shape and structure, but also enables it to learn additional language features from speech and text data, laying a solid foundation for subsequent cross-language migration. The cross-language transfer learning module also includes a meta-learning accelerator. Once the OCR recognition model obtains preliminary multilingual recognition capabilities through the cross-modal knowledge distillation unit, the next step is to improve its adaptation speed in the new language environment through the meta-learning accelerator. Demonstratively, the gradient-based meta-learning algorithm MAML is used to train the OCR model. During the training process, the OCR recognition model will learn on multiple language data sets, so that when encountering a new language, even with only a small number of samples, it can quickly adjust its own parameters to achieve a higher recognition accuracy. This ensures that the OCR recognition model can quickly switch between different languages ​​and reduce dependence on large-scale annotated data. The multimodal character recognition model of this application combines the advantages of CNN and RNN, effectively capturing the complex features of text images and improving recognition accuracy.

[0146] The multimodal feature fusion module combines the attention mechanism to dynamically integrate multiple preprocessing results, enhancing the model's adaptability to complex environments.

[0147] The adaptive noise suppression module uses deep learning denoising methods and context-aware algorithms to effectively remove image noise and improve recognition robustness.

[0148] The cross-language transfer learning module uses shared embedding space and transfer learning technology to achieve rapid adaptation and knowledge sharing of the model between multiple languages, thereby improving multi-language recognition capabilities.

[0149] Fig. 9 A schematic diagram of the structure of an optical character recognition device according to an embodiment of the present application is shown. Exemplarily, the optical character recognition device includes: a preprocessing module 910 , a multimodal data acquisition module 920 and a target text character recognition module 930 .

[0150] The preprocessing module 910 is used to obtain a target text image to be recognized and preprocess the target text image to obtain a pure text image; wherein the preprocessing includes using an adaptive noise suppression module to perform noise reduction processing on the target text image;

[0151] The multimodal data acquisition module 920 is used to acquire the voice description corresponding to the target text image to obtain voice modal data, and to acquire the text label annotated on the target text image to obtain label modal data;

[0152] The target text character recognition module 930 is used to input the voice modal data, the label modal data, and the clean text image into a trained multi-modal character recognition model for recognition to obtain target text characters.

[0153] It can be understood that the device of this embodiment corresponds to the optical character recognition method of the above embodiment, and the options in the above embodiment are also applicable to this embodiment, so they will not be described repeatedly here.

[0154] The present application also provides an optical character recognition device. Exemplarily, the optical character recognition device includes a processor and a memory, wherein the memory stores a computer program, and the processor runs the computer program to enable the optical character recognition device to perform the functions of the various modules in the above-mentioned optical character recognition method or the above-mentioned optical character recognition device.

[0155] Among them, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU) and a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or at least one of other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application.

[0156] The memory may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory is used to store a computer program, and the processor may execute the computer program accordingly after receiving an execution instruction.

[0157] The present application also provides a computer-readable storage medium for storing the computer program used in the optical character recognition device. For example, the computer-readable storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0158] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or the flow diagram, and the combination of boxes in the structure diagram and / or the flow diagram, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0159] In addition, the functional modules or units in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0160] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application.

[0161] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. An optical character recognition method, characterized in that: include: Acquire a target text image containing multiple languages ​​to be recognized, and preprocess the target text image to obtain a pure text image; wherein the preprocessing includes using an adaptive noise suppression module to perform noise reduction processing on the target text image; Acquire a voice description corresponding to the target text image to obtain voice modal data, and acquire a text label pre-labeled for the target text image to obtain label modal data; The speech modal data and / or the label modal data, and the clean text image are input into a trained multimodal character recognition model for recognition to obtain target text characters contained in the target text image.

2. The optical character recognition method according to claim 1, characterized in that: The adaptive noise suppression module is a U-Net network; the U-Net network includes an encoder, a decoder and a local analysis function unit; The step of using an adaptive noise suppression module to perform noise reduction processing on the target text image includes: Downsampling the target text image through each downsampling layer in the encoder to obtain a corresponding first feature map; The local analysis function unit is used to calculate the noise evaluation parameter of each pixel in each of the first feature maps, and the noise area is determined according to the noise evaluation parameter and the set noise threshold, and the noise area is marked to obtain each noise mask; wherein the noise evaluation parameter includes one or more combinations of mean, standard deviation and gradient information; Through each upsampling layer in the decoder, the output feature of the current upsampling layer is determined according to the corresponding noise mask, the corresponding first feature and the image feature of the previous layer input of the current upsampling layer.

3. The optical character recognition method according to claim 2, characterized in that: Before preprocessing the target text image, the method further includes: The U-Net network is iteratively trained, wherein a cosine annealing strategy is used to adjust the learning rate during the iterative training process; and the termination condition of the iterative training is that a peak signal-to-noise ratio obtained by the iterative training is greater than a set signal-to-noise ratio threshold, and a structural similarity index obtained by the iterative training is greater than a set index threshold.

4. The optical character recognition method according to claim 1, characterized in that: The multimodal character recognition model includes a convolutional neural network module, a text embedding layer, an attention mechanism layer and a recurrent neural network module connected in sequence; The step of inputting the voice modal data and / or the label modal data, and the clean text image into a trained multimodal character recognition model for recognition to obtain target text characters includes: The clean text image is input into the convolutional neural network module to obtain a visual feature map; the text data obtained by converting the speech modal data through speech recognition technology and the label modal data are respectively input into the text embedding layer to obtain a text feature map of the corresponding modality; the visual feature map and each of the text feature maps are input into the attention mechanism layer to calculate the attention weight of the feature map received by each channel; the feature maps are weighted and summed according to the attention weight of each channel to obtain a fused feature vector; the fused feature vector is input into the recurrent neural network module to extract context information and perform character recognition to obtain the target text character.

5. The optical character recognition method according to claim 4, characterized in that: The method further comprises: The multimodal character recognition model is structurally optimized and adjusted using a preset adjustment strategy according to the iterative training result of the multimodal character recognition model.

6. The optical character recognition method according to claim 5, characterized in that: The recurrent neural network module includes a Bi-LSTM layer; The method of optimizing the structure of the multimodal character recognition model by using a preset adjustment strategy according to the iterative training result of the multimodal character recognition model comprises: If the iterative training result shows that the multimodal character recognition model is underfitting, a preset number of convolutional layers, a preset number of filters or a change in the filter size is added to the convolutional neural network module, a preset number of Bi-LSTM layers are stacked after the Bi-LSTM layer in the recurrent neural network module, or a temporal convolutional network layer is added before the Bi-LSTM layer; Among them, if the difference between the set expected accuracy rate and the training accuracy rate obtained when training using multilingual training samples is greater than a set first difference value, and the absolute difference between the verification accuracy rate obtained when verifying using multilingual verification samples and the training accuracy rate is less than a set second difference value, it is determined that the multimodal character recognition model has an underfitting phenomenon.

7. The optical character recognition method according to claim 5, characterized in that: According to the iterative training result of the multimodal character recognition model, a preset adjustment strategy is adopted to perform structural optimization adjustment on the multimodal character recognition model, including: Unfreezing the output layer and the preset task layer of the multimodal character recognition model to update parameters; Freezing the task layers other than the output layer and the specified task layer in the multimodal character recognition model; If the difference between the character error rate obtained by the current iterative training and the character error rate obtained by the previous iterative training is less than the set error rate threshold and the word error rate obtained by the current iterative training is less than the word error rate obtained in the previous iterative cycle, or the gradient values ​​of the specified task layer within N iterative cycles are all less than the set gradient standard, then the next task layer is unfrozen.

8. The optical character recognition method according to claim 5, characterized in that: The step of optimizing the structure of the multimodal character recognition model by using a preset adjustment strategy according to the iterative training result of the multimodal character recognition model includes: If the character error rate and the word error rate in the iterative training result are both greater than the set maximum error rate threshold, or the training accuracy in the iterative training result is lower than the accuracy difference set by the expected accuracy, then the type of the erroneous character or grammatical structure is determined according to the character error rate and the word error rate, and a data sample corresponding to the type of the erroneous character or grammatical structure is added to the multilingual training sample; If the state of the loss curve in the iterative training result changes, the learning rate is adjusted using a learning rate decay strategy.

9. An optical character recognition device, characterized in that: include: A preprocessing module, used to obtain a target text image to be recognized, and preprocess the target text image to obtain a pure text image; wherein the preprocessing includes using an adaptive noise suppression module to perform noise reduction processing on the target text image; A multimodal data acquisition module, used to acquire the voice description corresponding to the target text image to obtain voice modal data, and to acquire the text label annotated on the target text image to obtain label modal data; The target text character recognition module is used to input the voice modal data and / or the label modal data, and the clean text image into a trained multimodal character recognition model for recognition to obtain the target text characters.

10. An optical character recognition device, characterized in that: The optical character recognition device comprises a processor and a memory, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the optical character recognition method according to any one of claims 1 to 8.

Images