Text recognition method, device, computer equipment and storage medium

Abstract

The present application relates to a method, device, computer equipment, storage medium and computer program product as described above for text recognition, wherein the method comprises: obtaining a text image; extracting character features of the text image to generate a text image feature sequence; obtaining neighborhood information of the text image feature sequence, and fusing the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence; extracting and combining the multi-temporal contextual relationships of the fused text image feature sequence to obtain a target text image feature sequence; and obtaining a text recognition result based on the target text image feature sequence. In the whole process, domain information and multi-temporal contextual relationships are obtained based on the image features of the text image, and then text recognition is performed based on the domain information and multi-temporal contextual relationships, which enhances the semantic features, significantly improves the accuracy of text recognition, and can achieve accurate text recognition.

Description

Technical Field

[0001] The present application relates to the technical field of text recognition, and in particular to a text recognition method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] Text recognition is an important task in the field of artificial intelligence, among which the recognition of handwritten text is the most challenging. Although technology is developing rapidly, handwritten activities are still easier, more efficient and cheaper. Many important documents still need to be filled out and digitized manually, including office handwritten document processing, mailing address recognition and precious historical manuscript recognition. Automatic recognition of handwritten Chinese text images is difficult because it involves multiple writing styles, complex character structures and a large number of character categories. Therefore, the research on recognition algorithms for handwritten Chinese text is of great significance and has great application prospects.

[0003] With the rise and popularity of deep learning technology, the use of neural networks to solve the problem of handwritten Chinese text has received more and more attention and has achieved great success. The most popular technical solution recently is to regard text recognition as an image sequence modeling task. Convolutional recurrent neural network is the most widely used method, in which convolutional neural network is used as feature extractor and recurrent neural network learns long-term dependencies.

[0004] Although the traditional text recognition solution based on convolutional recurrent neural network can realize text recognition, it also has the defect of low recognition accuracy. Summary of the invention

[0005] Based on this, it is necessary to provide an accurate text recognition method, device, computer equipment, computer-readable storage medium and computer program product to address the above technical problems.

[0006] In a first aspect, the present application provides a text recognition method. The method comprises:

[0007] Get text image;

[0008] Extract character features of text images and generate text image feature sequences;

[0009] Obtaining neighborhood information of the text image feature sequence, and fusing the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence;

[0010] Extract and combine the multi-temporal contextual relationship of the fused text image feature sequence to obtain the target text image feature sequence;

[0011] According to the feature sequence of the target text image, the text recognition result is obtained.

[0012] In one embodiment, obtaining a text recognition result according to a target text image feature sequence includes:

[0013] Input the target text image feature sequence into the convolutional neural network for text classification prediction to obtain the characters and the corresponding prediction probabilities of the characters;

[0014] According to the characters and the predicted probabilities corresponding to the characters, the probability distribution of different character sequences is obtained;

[0015] Select the character sequence with the highest probability to get the text recognition result.

[0016] In one embodiment, obtaining probability distributions of different character sequences based on characters and predicted probabilities corresponding to the characters includes:

[0017] Based on the characters and their corresponding predicted probabilities, the probability distribution of different character sequences is obtained through CTC.

[0018] In one embodiment, before extracting the character features of the text image and generating the text image feature sequence, the method further includes:

[0019] The text image is resized to update the text image.

[0020] In one embodiment, extracting character features of a text image and generating a text image feature sequence includes:

[0021] The character features of the text image are extracted through the residual convolutional network to generate a text image feature sequence. Each residual module in the residual convolutional network is connected to a convolutional layer.

[0022] In one embodiment, obtaining neighborhood information of a text image feature sequence, and fusing the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence includes:

[0023] Obtain the neighborhood information of the text image feature sequence through the Inception residual module;

[0024] The residual skip connection is used to fuse the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence.

[0025] In one embodiment, extracting and combining the multi-temporal contextual relationship of the fused text image feature sequence to obtain the target text image feature sequence includes:

[0026] Through the TCN bidirectional expansion network, the multi-temporal contextual relationship of the fused text image feature sequence is extracted and combined to obtain the target text image feature sequence.

[0027] In a second aspect, the present application also provides a text recognition device. The device comprises:

[0028] Image module, used to obtain text images;

[0029] A feature extraction module is used to extract character features of text images and generate text image feature sequences;

[0030] A fusion module is used to obtain the neighborhood information of the text image feature sequence, and fuse the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence;

[0031] A context extraction module is used to extract and combine the multi-temporal contextual relationship of the fused text image feature sequence to obtain a target text image feature sequence;

[0032] The text recognition module is used to obtain text recognition results based on the target text image feature sequence.

[0033] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0034] Get text image;

[0035] Extract character features of text images and generate text image feature sequences;

[0036] Obtaining neighborhood information of the text image feature sequence, and fusing the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence;

[0037] Extract and combine the multi-temporal contextual relationship of the fused text image feature sequence to obtain the target text image feature sequence;

[0038] According to the feature sequence of the target text image, the text recognition result is obtained.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0040] Get text image;

[0041] Extract character features of text images and generate text image feature sequences;

[0042] Obtaining neighborhood information of the text image feature sequence, and fusing the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence;

[0043] Extract and combine the multi-temporal contextual relationship of the fused text image feature sequence to obtain the target text image feature sequence;

[0044] According to the feature sequence of the target text image, the text recognition result is obtained.

[0045] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0046] Get text image;

[0047] Extract character features of text images and generate text image feature sequences;

[0048] Obtaining neighborhood information of the text image feature sequence, and fusing the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence;

[0049] Extract and combine the multi-temporal contextual relationship of the fused text image feature sequence to obtain the target text image feature sequence;

[0050] According to the feature sequence of the target text image, the text recognition result is obtained.

[0051] The above-mentioned text recognition method, device, computer equipment, storage medium and computer program product obtain a text image; extract character features of the text image to generate a text image feature sequence; obtain neighborhood information of the text image feature sequence, and fuse the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence; extract and combine the multi-temporal contextual relationship of the fused text image feature sequence to obtain a target text image feature sequence; obtain a text recognition result based on the target text image feature sequence. In the whole process, domain information and multi-temporal contextual relationship are obtained based on the image features of the text image, and text recognition is performed based on the domain information and multi-temporal contextual relationship, which enhances the semantic features, significantly improves the accuracy of text recognition, and can achieve accurate text recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A diagram showing an application environment of a text recognition method in an embodiment;

[0053] Figure 2 A flowchart of a text recognition method in one embodiment;

[0054] Figure 3 is a flowchart of a text recognition method in another embodiment;

[0055] Figure 4 This is a schematic diagram of the architecture of the residual convolutional network;

[0056] Figure 5 This is a schematic diagram of the Inception residual module architecture;

[0057] Figure 6 Schematic diagram of the TCN network residual block processing flow;

[0058] Figure 7 is a flowchart of a text recognition method in a specific application example;

[0059] Figure 8 is a structural block diagram of a text recognition device in one embodiment;

[0060] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] The text recognition method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 collects text images, sends the text images to the server 104, and the server 104 obtains the text images; extracts the character features of the text images, generates a text image feature sequence; obtains the neighborhood information of the text image feature sequence, and fuses the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence; extracts and combines the multi-temporal contextual relationship of the fused text image feature sequence to obtain a target text image feature sequence; obtains the text recognition result according to the target text image feature sequence. Further, the server 104 can send the text recognition result to the terminal 102. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0063] In one embodiment, Figure 2 As shown, a text recognition method is provided, which is applied to Figure 1Taking the server 104 in the example as an example, the following steps are included:

[0064] S100: Acquire a text image.

[0065] A text image refers to an image that carries characters, such as a handwritten Chinese text image.

[0066] S200: extracting character features of a text image and generating a text image feature sequence.

[0067] After receiving the text image, the server extracts the character features of the text image and arranges the extracted character features in order to obtain a text image feature sequence. Specifically, the character features of the text image can be extracted by constructing a feature extractor. The feature extractor can refer to the idea of ​​the residual network, and adjust some convolutional network parameters according to the long strip shape characteristics of the data acquisition image, and finally construct a residual convolutional network.

[0068] S300: Obtaining neighborhood information of a text image feature sequence, and fusing the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence.

[0069] Neighborhood information refers to the character feature information of the adjacent domain, that is, the domain information of the current single system. If necessary, the domain information can also be obtained through the residual module. Preferably, the Inception residual module can be used to extract the domain information of a single character. After the domain information is extracted, the domain information is fused with the text image feature sequence to enhance the semantic features and obtain a fused text image feature sequence. Furthermore, convolution kernels of different sizes can be used simultaneously to extract features of different scales, and the features of different scales can be fused by stacking them through the channel dimension.

[0070] S400: extracting and combining the multi-temporal contextual relationship of the fused text image feature sequence to obtain a target text image feature sequence.

[0071] Contextual relations specifically refer to contextual dependencies in time scales. Extracting contextual relations is also to enhance semantic features. Specifically, multi-time-series contextual information can be extracted by constructing a TCN bidirectional expansion network. The TCN bidirectional expansion network uses a hierarchical structure to extract contextual dependencies in time scales. When extracting multi-time-series contextual relations, the TCN bidirectional expansion network combines the multi-time-series contextual relations with the fused text image feature sequence to obtain the final target text image feature sequence.

[0072] S500: Obtaining text recognition results according to the target text image feature sequence.

[0073] Initial text recognition is performed based on the target text image feature sequence to obtain a preliminary text recognition result, which includes the recognized characters and their corresponding probabilities. Further selection is performed based on the preliminary text recognition result and its corresponding probability to obtain the final text recognition result. If necessary, the character with the highest probability can be selected to obtain the final character sequence in chronological order, that is, the final text recognition result.

[0074] The above-mentioned text recognition method obtains a text image; extracts character features of the text image to generate a text image feature sequence; obtains neighborhood information of the text image feature sequence, and fuses the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence; extracts and combines the multi-temporal contextual relationship of the fused text image feature sequence to obtain a target text image feature sequence; obtains a text recognition result based on the target text image feature sequence. In the whole process, domain information and multi-temporal contextual relationship are obtained based on the image features of the text image, and then text recognition is performed based on the domain information and multi-temporal contextual relationship, which enhances the semantic features, significantly improves the accuracy of text recognition, and can achieve accurate text recognition.

[0075] like Figure 3 As shown, in one embodiment, S500 includes:

[0076] S520: Input the target text image feature sequence into the convolutional neural network for text classification prediction to obtain characters and prediction probabilities corresponding to the characters;

[0077] S540: Obtaining probability distribution of different character sequences according to the characters and the prediction probabilities corresponding to the characters;

[0078] S560: Select the character sequence corresponding to the maximum probability to obtain the text recognition result.

[0079] In this embodiment, a preliminary classification prediction is first performed on the text through a convolutional neural network to obtain characters and their corresponding prediction probabilities, and then this part of the data is statistically analyzed to obtain the probability selection corresponding to different character sequences. Each character sequence can be understood as a text path, and the character sequence corresponding to the maximum probability is selected to obtain the text recognition result.

[0080] Furthermore, the probability distribution of different character sequences can be obtained through CTC. CTC refers to the connection time classifier, which is used to deal with the situation where the length of the unknown input sequence is inconsistent with the known label sequence. It mainly introduces blank labels to solve the problem of data alignment. In practical applications, computer languages ​​are all written based on binary languages. There is a corresponding Chinese character label for each Chinese character. When the computer obtains a text image, it will convert the characters into corresponding Chinese character labels based on the corresponding label data. The working process of CTC will be introduced in detail below: Assume that the output probability distribution sequence is y = (y1, y2, ..., y w ), w represents each time step. The mapping relationship of the classifier CTC is: p(l|y) = ∑ β(π)＝1 p(π|y), where p(l|y) is the conditional probability of outputting label sequence l given input sequence feature y, p(π|y) is the conditional probability of outputting label sequence π given input sequence feature y, β(π)=1 is the mapping of sequence π to sequence l, β is a many-to-one mapping relationship, which transforms π into a shorter l sequence as the final prediction result. Assuming that the character set C contains all the characters in our task, 'blank' represents the blank character, and the input feature sequence is of length T, a large number of label sequences of length T can be obtained. Each time step t will output the predicted probability of all characters. Selecting a character for each time step can obtain the output sequence, that is, a mapping path. When CTC outputs the label sequence, it first removes duplicate characters, then removes blank characters, and obtains the final recognition result. A true label can correspond to multiple mapping paths. For example, "I wish you a happy day" and "I wish you a happy day" both correspond to the same sequence "I wish you a happy day". We use a greedy strategy to select the path with the maximum probability l = β (argmax π p(π|y)) is used as the final output, that is, the label with the largest predicted probability is selected at each time, that is, the character with the largest predicted probability is selected.

[0081] In one of the embodiments, before extracting the character features of the text image and generating the text image feature sequence, the method further includes: performing size normalization processing on the text image to update the text image.

[0082] Before extracting the character features of the text image, the text image is also preprocessed, including size normalization. Specifically, the height of the image can be fixed to 64, while maintaining the aspect ratio, and the pixel value can be standardized (data is standardized based on the mean and standard deviation of the original data). The processed data conforms to the standard normal distribution, that is, the mean is 0 and the standard deviation is 1. After standardization, data centralization is achieved, which conforms to the data distribution law and can increase the generalization ability of the model.

[0083] In one embodiment, extracting character features of a text image and generating a text image feature sequence includes: extracting character features of a text image through a residual convolutional network and generating a text image feature sequence, wherein each residual module in the residual convolutional network is connected to a convolutional layer.

[0084] In this embodiment, a residual convolutional network is constructed to extract character features and generate a text image feature sequence. Specifically, the entire residual convolutional network is as follows: Figure 4 As shown, the network is mainly composed of convolutional layers and residual modules. The number of convolutional kernels of the residual module from top to bottom is 64, 128, 256 and 512 respectively. The standard components of the residual module are two convolution operations. Batch normalization is connected after each convolutional layer, and ReLU is used as the activation function. The convolution kernel size is set to 3x3. The step size of the residual module is set to 2, the padding value is set to 1, and the lower layer features are passed to the upper layer through a jump connection or 1x1 convolution. After each residual module, an additional convolutional layer is used for effective feature aggregation. In the above processing, batch normalization is connected after each convolutional layer to regularize the network and accelerate network convergence; and using ReLU as the activation function introduces more nonlinear factors, so that the network has stronger expression ability. In this embodiment, a convolutional layer of the same size is added after each residual module to fuse the features learned by the residual with the input features of the residual block to extract more effective features.

[0085] In one embodiment, obtaining neighborhood information of a text image feature sequence, and fusing the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence includes:

[0086] The neighborhood information of the text image feature sequence is obtained through the Inception residual module; the neighborhood information is fused with the text image feature sequence using the residual jump connection to obtain a fused text image feature sequence.

[0087] In this embodiment, the neighborhood information of a single character is extracted through the Inception residual module. Inception uses a wider structure to build a network, and uses convolution kernels of different sizes to extract feature representations of different scales, and stacks them through the channel dimension to fuse features of different scales. We use Inception to fuse multiple neighborhood information in the task, such as Figure 5As shown, three convolution kernels of different sizes are used, namely 1x1, 3x1 and 5x1, and the residual jump connection is used to fuse the image sequence features with the domain information. Specifically, in this embodiment, the neighborhood information is used to supplement and enrich the feature sequence. Technically, convolution kernels of different widths are used for operation, and then the output feature maps are spliced ​​along the channel dimension to output multi-neighborhood information, which is summed with the feature sequence.

[0088] In one embodiment, extracting and combining the multi-temporal contextual relationship of the fused text image feature sequence to obtain a target text image feature sequence; obtaining a text recognition result according to the target text image feature sequence includes:

[0089] Through the TCN bidirectional expansion network, the multi-temporal contextual relationship of the fused text image feature sequence is extracted and combined to obtain the target text image feature sequence; based on the target text image feature sequence, the text recognition result is obtained.

[0090] The TCN bidirectional expansion network uses a hierarchical structure to extract long-term contextual dependencies and uses dilated convolutions. Dilated convolutions allow for interval sampling of inputs during convolution, and the sampling rate is controlled by the dilation rate, so that fewer layers can be stacked to obtain a larger receptive field. We use a doubling of the dilation factor for each layer and use skip connections and 1x1 convolution operations. The skip connection adds the lower-layer features to the higher-layer features. The 1x1 convolution is to prevent the lower-layer features from not matching the higher-layer dimensions to increase the dimensionality of the features. Figure 6 As shown in the figure, in the hierarchical structure of the TCN network, the output sequence of each hidden layer has the same length as the input sequence. Using a one-dimensional convolution with a convolution kernel of 3, and setting a doubled expansion rate such as 1, 2, 4, ..., only 4 layers are used to obtain a 32-bit receptive field, and the amount of calculation and parameters is reduced. TCN also introduces a residual module, such as Figure 6 As shown in the figure, the module contains two branches. The residual branch consists of dilated convolution, weight normalization and activation operations, and the other branch includes 1x1 convolution or jump connection. Due to the causal convolution used by the TCN network, the output of each time step is only related to the past time and the current time, and the information of the future time step cannot be used, that is, only a one-way context dependency can be obtained. Using ordinary convolution to replace causal convolution, TCN is expanded into a bidirectional structure, such as Figure 7 As shown, each neuron represents the data of one time step, and each layer has the same number of neurons, which can realize information exchange in two directions and obtain richer multi-time series context information.

[0091] In order to explain the technical solution of the text recognition method of the present application in detail, the following will take the application of the text recognition method of the present application to the handwritten Chinese character text recognition scenario as an example to explain the entire processing process in detail, which specifically includes the following steps:

[0092] 1. Obtain images of handwritten Chinese text and their corresponding label data;

[0093] 2. Preprocess the text image and normalize the size of the text image;

[0094] 3. Use residual convolutional network to extract character features of text images and generate image feature sequences;

[0095] 4. Extract domain information of sequence features through the Inception residual module;

[0096] 5. TCN bidirectional expansion network extracts and combines the multi-temporal contextual relationships of image sequence features;

[0097] 6. CTC outputs the probability distribution of sequence labels and obtains the recognition result.

[0098] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0099] Based on the same inventive concept, the embodiment of the present application also provides a text recognition device for implementing the above-mentioned text recognition method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more text recognition device embodiments provided below can refer to the limitations of the text recognition method above, and will not be repeated here.

[0100] In one embodiment, Figure 8 As shown, a text recognition device is provided, comprising:

[0101] An image module 100 is used to obtain a text image;

[0102] The feature extraction module 200 is used to extract the character features of the text image and generate a text image feature sequence;

[0103] A fusion module 300 is used to obtain neighborhood information of the text image feature sequence, and fuse the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence;

[0104] A context extraction module 400 is used to extract and combine the multi-temporal contextual relationship of the fused text image feature sequence to obtain a target text image feature sequence;

[0105] The text recognition module 500 is used to obtain a text recognition result according to a target text image feature sequence.

[0106] The above-mentioned text recognition device obtains a text image; extracts character features of the text image to generate a text image feature sequence; obtains neighborhood information of the text image feature sequence, and fuses the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence; extracts and combines the multi-temporal contextual relationship of the fused text image feature sequence to obtain a target text image feature sequence; obtains a text recognition result based on the target text image feature sequence. In the whole process, domain information and multi-temporal contextual relationships are obtained based on the image features of the text image, and then text recognition is performed based on the domain information and multi-temporal contextual relationships, which enhances the semantic features, significantly improves the accuracy of text recognition, and can achieve accurate text recognition.

[0107] In one embodiment, the text recognition module 500 is also used to input the target text image feature sequence into a convolutional neural network for text classification prediction to obtain characters and corresponding prediction probabilities of the characters; obtain the probability distribution of different character sequences based on the characters and the corresponding prediction probabilities of the characters; select the character sequence corresponding to the maximum probability to obtain the text recognition result.

[0108] In one embodiment, the text recognition module 500 is further configured to obtain probability distributions of different character sequences through CTC according to the characters and the predicted probabilities corresponding to the characters.

[0109] In one embodiment, the feature extraction module 200 is further used to perform size normalization processing on the text image to update the text image.

[0110] In one embodiment, the feature extraction module 200 is further used to extract character features of a text image through a residual convolutional network to generate a text image feature sequence, and each residual module in the residual convolutional network is connected to a convolutional layer.

[0111] In one of the embodiments, the fusion module 300 is further used to obtain neighborhood information of the text image feature sequence through the Inception residual module; and to fuse the neighborhood information with the text image feature sequence using a residual jump connection to obtain a fused text image feature sequence.

[0112] In one of the embodiments, the context extraction module 400 is further used to extract and combine the multi-temporal contextual relationships of the fused text image feature sequence through a TCN bidirectional expansion network to obtain a target text image feature sequence.

[0113] Each module in the above-mentioned text recognition device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0114] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as the correspondence between characters and character label data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a text recognition method is implemented.

[0115] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0116] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0117] Get text image;

[0118] Extract character features of text images and generate text image feature sequences;

[0119] Obtaining neighborhood information of the text image feature sequence, and fusing the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence;

[0120] Extract and combine the multi-temporal contextual relationship of the fused text image feature sequence to obtain the target text image feature sequence;

[0121] According to the feature sequence of the target text image, the text recognition result is obtained.

[0122] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0123] The target text image feature sequence is input into the convolutional neural network for text classification prediction to obtain the characters and the corresponding prediction probabilities of the characters; based on the characters and the corresponding prediction probabilities of the characters, the probability distribution of different character sequences is obtained; the character sequence with the largest probability is selected to obtain the text recognition result.

[0124] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0125] Based on the characters and their corresponding predicted probabilities, the probability distribution of different character sequences is obtained through CTC.

[0126] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0127] The text image is resized to update the text image.

[0128] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0129] The character features of the text image are extracted through the residual convolutional network to generate a text image feature sequence. Each residual module in the residual convolutional network is connected to a convolutional layer.

[0130] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0131] The neighborhood information of the text image feature sequence is obtained through the Inception residual module; the neighborhood information is fused with the text image feature sequence using the residual jump connection to obtain a fused text image feature sequence.

[0132] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0133] Through the TCN bidirectional expansion network, the multi-temporal contextual relationship of the fused text image feature sequence is extracted and combined to obtain the target text image feature sequence.

[0134] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0135] Get text image;

[0136] Extract character features of text images and generate text image feature sequences;

[0137] Obtaining neighborhood information of the text image feature sequence, and fusing the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence;

[0138] Extract and combine the multi-temporal contextual relationship of the fused text image feature sequence to obtain the target text image feature sequence;

[0139] According to the feature sequence of the target text image, the text recognition result is obtained.

[0140] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0141] The target text image feature sequence is input into the convolutional neural network for text classification prediction to obtain the characters and the corresponding prediction probabilities of the characters; based on the characters and the corresponding prediction probabilities of the characters, the probability distribution of different character sequences is obtained; the character sequence with the largest probability is selected to obtain the text recognition result.

[0142] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0143] Based on the characters and their corresponding predicted probabilities, the probability distribution of different character sequences is obtained through CTC.

[0144] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0145] The text image is resized to update the text image.

[0146] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0147] The character features of the text image are extracted through the residual convolutional network to generate a text image feature sequence. Each residual module in the residual convolutional network is connected to a convolutional layer.

[0148] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0149] The neighborhood information of the text image feature sequence is obtained through the Inception residual module; the neighborhood information is fused with the text image feature sequence using the residual jump connection to obtain a fused text image feature sequence.

[0150] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0151] Through the TCN bidirectional expansion network, the multi-temporal contextual relationship of the fused text image feature sequence is extracted and combined to obtain the target text image feature sequence.

[0152] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0153] Get text image;

[0154] Extract character features of text images and generate text image feature sequences;

[0155] Obtaining neighborhood information of the text image feature sequence, and fusing the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence;

[0156] Extract and combine the multi-temporal contextual relationship of the fused text image feature sequence to obtain the target text image feature sequence;

[0157] According to the feature sequence of the target text image, the text recognition result is obtained.

[0158] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0159] The target text image feature sequence is input into the convolutional neural network for text classification prediction to obtain the characters and the corresponding prediction probabilities of the characters; based on the characters and the corresponding prediction probabilities of the characters, the probability distribution of different character sequences is obtained; the character sequence with the largest probability is selected to obtain the text recognition result.

[0160] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0161] Based on the characters and their corresponding predicted probabilities, the probability distribution of different character sequences is obtained through CTC.

[0162] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0163] The text image is resized to update the text image.

[0164] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0165] The character features of the text image are extracted through the residual convolutional network to generate a text image feature sequence. Each residual module in the residual convolutional network is connected to a convolutional layer.

[0166] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0167] The neighborhood information of the text image feature sequence is obtained through the Inception residual module; the neighborhood information is fused with the text image feature sequence using the residual jump connection to obtain a fused text image feature sequence.

[0168] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0169] Through the TCN bidirectional expansion network, the multi-temporal contextual relationship of the fused text image feature sequence is extracted and combined to obtain the target text image feature sequence.

[0170] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0171] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0172] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0173] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A text recognition method, characterized in that: The method comprises: Get text image; Extracting character features of the text image to generate a text image feature sequence; Acquire neighborhood information of a text image feature sequence, and fuse the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence, wherein the neighborhood information refers to character feature information of an adjacent domain; Extracting and combining the multi-temporal contextual relationship of the fused text image feature sequence to obtain a target text image feature sequence; Obtaining a text recognition result according to the target text image feature sequence; The obtaining of neighborhood information of a text image feature sequence and fusing the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence comprises: obtaining the neighborhood information of the text image feature sequence through an Inception residual module; and fusing the neighborhood information with the text image feature sequence using a residual jump connection to obtain a fused text image feature sequence.

2. The method according to claim 1, characterized in that: The obtaining of the text recognition result according to the target text image feature sequence comprises: Inputting the target text image feature sequence into a convolutional neural network for text classification prediction to obtain characters and prediction probabilities corresponding to the characters; Obtaining probability distribution of different character sequences according to the characters and the predicted probabilities corresponding to the characters; Select the character sequence with the highest probability to get the text recognition result.

3. The method according to claim 2, characterized in that The obtaining of probability distribution of different character sequences according to the characters and the predicted probabilities corresponding to the characters includes: According to the characters and the predicted probabilities corresponding to the characters, probability distributions of different character sequences are obtained through CTC.

4. The method according to claim 1, characterized in that: Before extracting the character features of the text image and generating the text image feature sequence, the method further includes: The text image is size normalized to update the text image.

5. The method according to claim 1, characterized in that The step of extracting character features of the text image and generating a text image feature sequence comprises: The character features of the text image are extracted through a residual convolutional network to generate a text image feature sequence, and each residual module in the residual convolutional network is connected to a convolutional layer.

6. The method according to claim 1, characterized in that The extraction and combination of the multi-temporal contextual relationship of the fused text image feature sequence to obtain the target text image feature sequence includes: Through the TCN bidirectional expansion network, the multi-temporal contextual relationship of the fused text image feature sequence is extracted and combined to obtain the target text image feature sequence.

7. A text recognition device, characterized in that: The device comprises: Image module, used to obtain text images; A feature extraction module, used to extract character features of the text image and generate a text image feature sequence; A fusion module, used to obtain neighborhood information of a text image feature sequence, and fuse the neighborhood information with the text image feature sequence to obtain a fused text image feature sequence, wherein the neighborhood information refers to character feature information of an adjacent domain; A context extraction module, used to extract and combine the multi-temporal contextual relationship of the fused text image feature sequence to obtain a target text image feature sequence; A text recognition module, used to obtain a text recognition result according to the target text image feature sequence; The fusion module is also used to obtain neighborhood information of the text image feature sequence through the Inception residual module; and to fuse the neighborhood information with the text image feature sequence using a residual jump connection to obtain a fused text image feature sequence.

8. The device according to claim 7, characterized in that The text recognition module is also used to input the target text image feature sequence into the convolutional neural network for text classification prediction, and obtain the characters and the prediction probabilities corresponding to the characters; according to the characters and the prediction probabilities corresponding to the characters, the probability distribution of different character sequences is obtained; Select the character sequence with the highest probability to get the text recognition result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images