Method, device and computer equipment for text image recognition based on global semantics
Through global semantic features guidance visual feature decoding, the problem of high error rate of text recognition on low-quality images is solved, and the recognition accuracy is improved.
Patent Information
- Application Number
- CN202210086253.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-25
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-01-25
AI Technical Summary
In the prior art, the error rate of text recognition is high, especially in the case of low image quality, such as image defects, blur, uneven light, and text occlusion, resulting in difficulty in recognition.
Using a text image recognition method based on global semantics, visual feature extraction and timing information extraction are performed through a pre-trained visual feature extraction model, visual feature matrix is obtained, and global semantic extraction is performed. The attention mechanism is used to decode feature with global semantic features as the guide. , improve recognition accuracy.
Improve the accuracy of text recognition in low-quality image, make the text recognition more accurate and reduce the error rate.
Smart Images

Figure CN114445832B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of artificial intelligence technology and image recognition, and in particular to a text image recognition method, device, computer equipment and storage medium based on global semantics. Background Art
[0002] With the development of artificial intelligence, image recognition technology has been implemented in various fields. Among them, character recognition technology (OCR) is a technology that extracts text content from images. It is widely used in daily production and life, such as license plate number recognition, text table recognition, customer information recognition, etc. However, in some occasions, due to the difficulty in ensuring the image quality of text images, there may be low-quality images such as image defects, blurred images, uneven lighting, obscured text, and similar characters. Such low-quality text images are difficult to recognize, resulting in a high error rate in text recognition. Summary of the invention
[0003] The purpose of the embodiments of the present application is to propose a method, apparatus, computer device and storage medium for text image recognition based on global semantics to solve the problem of high error rate in text recognition.
[0004] In order to solve the above technical problems, the embodiment of the present application provides a text image recognition method based on global semantics, which adopts the following technical solution:
[0005] Obtain the text image to be recognized;
[0006] Extracting visual features of the to-be-recognized text image through the visual part of the pre-trained visual feature extraction model, and extracting temporal information of the extracted visual features through the temporal part of the pre-trained visual feature extraction model, to obtain a visual feature matrix of the to-be-recognized text image;
[0007] Performing global semantic extraction on the visual feature matrix to obtain global semantic features of the text image to be recognized;
[0008] The global semantic features are used as state parameters, and the visual feature matrix is feature decoded through the attention mechanism in the trained decoding model to obtain the recognition text of the text image to be recognized.
[0009] In order to solve the above technical problems, the embodiment of the present application further provides a text image recognition device based on global semantics, which adopts the following technical solution:
[0010] An acquisition module, used for acquiring a text image to be recognized;
[0011] A first extraction module is used to extract visual features of the to-be-recognized text image through the visual part of the pre-trained visual feature extraction model, and to extract temporal information of the extracted visual features through the temporal part of the pre-trained visual feature extraction model, so as to obtain a visual feature matrix of the to-be-recognized text image;
[0012] A second extraction module is used to perform global semantic extraction on the visual feature matrix to obtain global semantic features of the text image to be recognized;
[0013] The decoding module is used to use the global semantic features as state parameters and perform feature decoding on the visual feature matrix through the attention mechanism in the trained decoding model to obtain the recognition text of the text image to be recognized.
[0014] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned global semantics-based text image recognition method are implemented.
[0015] In order to solve the above technical problems, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned global semantics-based text image recognition method are implemented.
[0016] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: after acquiring a text image to be recognized, visual features of the text image to be recognized are extracted through the visual part of a pre-trained visual feature extraction model, and temporal information of the extracted visual features is extracted through the temporal part of the pre-trained visual feature extraction model, so that the visual feature matrix has a temporal attribute, and global semantic features of the text image to be recognized are extracted. Guided by the global semantic features, the visual features are feature decoded through the attention mechanism to obtain the recognized text of the text image to be recognized. Since the process of decoding the visual features through the attention mechanism is guided by the global semantic features, the global semantics is referenced on the basis of decoding the local vision, so that the recognized text is more accurate, and the text recognition accuracy for low-quality images is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0018] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0019] Figure 2 is a flow chart of an embodiment of a method for text image recognition based on global semantics according to the present application;
[0020] Figure 3 is a flow chart of an embodiment of visual feature extraction according to the present application;
[0021] Figure 3a is a schematic diagram of an embodiment of a visual feature extraction model according to the present application;
[0022] Figure 4 is a flow chart of an embodiment of pre-training a visual feature extraction network according to the present application;
[0023] Figure 5 is a flow chart of an embodiment of global semantic extraction according to the present application;
[0024] Figure 6 is a flowchart of an embodiment of pre-training a semantic enhancement model according to the present application;
[0025] Figure 7 is a flow chart of an embodiment of feature decoding according to the present application;
[0026] Figure 8 is a structural schematic diagram of an embodiment of a text image recognition device based on global semantics according to the present application;
[0027] Fig. 9 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0029] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0031] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0032] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0033] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.
[0034] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .
[0035] It should be noted that the global semantics-based text image recognition method provided in the embodiment of the present application is generally executed by a server, and accordingly, the global semantics-based text image recognition device is generally set in the server.
[0036] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0037] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for character image recognition based on global semantics according to the present application. The method for character image recognition based on global semantics comprises the following steps:
[0038] Step S201, obtaining a text image to be recognized.
[0039] In this embodiment, the text image recognition method based on global semantics is executed on the electronic device (eg Figure 1 The server shown in the figure) can communicate with the terminal through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.
[0040] Specifically, the server may obtain the text image to be recognized through user scanning, user uploading, or downloading from the Internet. The text image to be recognized may be understood as an image carrier recording text content.
[0041] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned text images to be identified, the above-mentioned text images to be identified can also be stored in a node of a blockchain.
[0042] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.
[0043] Step S202, extract visual features of the text image to be recognized through the visual part of the pre-trained visual feature extraction model, and extract temporal information of the extracted visual features through the temporal part of the pre-trained visual feature extraction model to obtain a visual feature matrix of the text image to be recognized.
[0044] Specifically, after obtaining the text image to be recognized, the server can perform visual feature extraction and timing information extraction on the text image to be recognized through a pre-trained visual feature extraction model. The pre-trained visual feature extraction model includes a visual part and a timing part. The visual part can be used to extract visual features of the text image to be recognized, and the timing information of the extracted visual features can be extracted through the timing part of the pre-trained visual feature extraction model, thereby obtaining a visual feature matrix of the text image to be recognized. Among them, visual feature extraction refers to extracting image information into abstract features that can be recognized by a computer. It should be noted that image information refers to information that can be recognized by humans, and abstract features refer to information that can be recognized by computers. The visual feature matrix in this application is a kind of information that can be recognized and processed by computers.
[0045] In this embodiment, the visual part can be a preset convolutional neural network, the timing part can be a preset timing information extraction network, and the visual features can be extracted by using a convolutional neural network. The server calls the preset convolutional neural network to process the text image to be recognized, thereby obtaining a preliminary visual feature matrix of the text image to be recognized. Furthermore, the visual feature matrix is also processed by timing. After extracting the preliminary visual feature matrix of the text image to be recognized, the server calls the preset timing information extraction network to perform timing processing on the preliminary visual feature matrix to obtain a visual feature matrix with added timing attributes.
[0046] In a possible embodiment, the server may perform 0-1 normalization processing on the acquired image to be recognized to obtain a normalized image to be recognized.
[0047] In this embodiment, the service obtains the image to be recognized and performs 0-1 normalization processing to obtain the normalized image to be recognized, and then performs visual feature processing on the normalized image to be recognized. The above normalization processing can be understood as mapping the pixel values in the image to between 0 and 1, which can make the image to be recognized more in line with the calculation form of the computer, thereby improving the calculation speed of the computer and further improving the recognition efficiency of text recognition.
[0048] Step S203, performing global semantic extraction on the visual feature matrix to obtain global semantic features of the text image to be recognized.
[0049] After the server obtains the visual feature matrix, it can extract the global semantics of the visual features to obtain the global semantics of the text image to be recognized. Furthermore, the visual feature matrix of the text image to be recognized contains visual feature information of multiple scales. Therefore, the semantic features of the text image to be recognized at different scales can be extracted from the visual features of different scales, thereby forming the global semantics.
[0050] In this embodiment, the global semantic feature is an enhanced semantic feature, which can be understood as enhancing the semantic feature through a language model to improve the accuracy of semantics. This application enhances the global semantic feature, and further improves the text recognition accuracy of low-quality images through the enhanced global semantic feature.
[0051] Step S204, taking the global semantic features as state parameters, the visual feature matrix is feature decoded by the attention mechanism in the trained decoding model to obtain the recognition text of the text image to be recognized.
[0052] After obtaining the global semantic features, the server can input the global semantic features and the visual feature matrix into the trained decoding model for decoding. In the trained decoding model, the visual feature matrix can be converted into corresponding text through a pre-constructed coding table to obtain the recognized text of the text image to be recognized, thereby completing the recognition process from image to text.
[0053] Furthermore, the global semantic features can be used to initialize the trained decoding model, and the global semantic features can be used as state parameters of the trained decoding model to avoid the trained decoding model from using zero state initialization or random initialization, so that the trained decoding model does not need an initialization process. While improving the decoding speed, the global semantic features can be used as a guide to not only use local visual information for decoding, but also use global semantic information to produce more accurate results, and can produce coherent and semantically valid recognition text.
[0054] In this embodiment, global semantic extraction is performed on the visual feature matrix to obtain the global semantic features of the text image to be recognized. Guided by the global semantic features, the visual features are feature decoded through the attention mechanism to obtain the recognized text of the text image to be recognized. Since the process of decoding the visual features through the attention mechanism is guided by the global semantic features, the global semantics is referenced on the basis of the decoding of the local vision, which makes the recognized text more accurate and improves the text recognition accuracy for low-quality images.
[0055] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned identification text, the above-mentioned identification text can also be stored in a node of a blockchain.
[0056] Furthermore, the visual part of the pre-trained visual feature extraction model is the jointly trained visual feature extraction network, and the temporal part of the pre-trained visual feature extraction model is the trained first temporal information extraction network, which is used to extract temporal information at the visual level. Figure 3 , Figure 3The flowchart of an embodiment of visual feature extraction in the present application is shown. Before step S202, it also includes:
[0057] Step S2021, constructing a visual feature extraction network based on a convolutional neural network.
[0058] Specifically, the server constructs a visual feature extraction network based on a convolutional neural network. The visual feature extraction network may include a preset number of first convolutional layers and second convolutional layers corresponding to the first convolutional layers, and the second convolutional layers are used to perform convolution calculations on the convolution results of the previous second convolutional layer and the convolution results of the corresponding first convolutional layer.
[0059] More specifically, the server can pre-acquire a visual feature extraction model based on a convolutional neural network, and extract part of the computing network from the pre-acquired visual feature extraction model to construct the visual feature extraction network of the present application. The server can also obtain a visual feature extraction network constructed by the staff themselves.
[0060] In this embodiment, the visual feature extraction network can be constructed as a V-shaped structure. The V structure may include an input side and an output side. The input side may include a preset number of first convolutional layers, and the output side may include second convolutional layers with the same number as the first convolutional layers. Each convolutional layer on the input side corresponds in parallel to a convolutional layer on the output side. While each first convolutional layer inputs the convolution result into the next first convolutional layer, it also inputs the convolution result into the parallel second convolutional layer, so that the global information of the input image can be retained.
[0061] Step S2022, constructing a first time series information extraction network based on a recursive neural network.
[0062] Specifically, the server constructs a first timing information extraction network based on a recursive neural network, wherein the first timing information extraction network includes a timing processing unit corresponding one-to-one to the second convolutional layer, and the timing processing unit is used to perform timing processing on the processing result of the previous timing processing unit and the convolution result of the corresponding second convolutional layer.
[0063] More specifically, the server can pre-acquire a time series processing model based on a recursive neural network, and use the pre-acquired time series processing model to construct the first time series information extraction network of the present application. The server can also obtain the first time series information extraction network constructed by the staff themselves.
[0064] The first time series information extraction network is constructed as a bidirectional long short-term memory network structure, so that the first time series information extraction network can extract forward time series information and reverse time series information, thereby extracting more accurate time series information.
[0065] For example, for "I love my family", "I", "love", and "my family" can be extracted into three vectors such as (h0, h1, h2,) through the forward long short-term memory network, and "my family", "love", and "I" can be extracted into three vectors such as (r2, r1, r0,) through the reverse long short-term memory network. Since the three vectors in (h0, h1, h2) have taken into account the temporal dependency, and the three vectors in (r0, r1, r2) have also taken into account the temporal dependency, (h2, r2) can be used to represent the feature concatenation of the forward temporal information and the reverse temporal information, thereby extracting more accurate temporal information.
[0066] In this application, the visual feature matrix with time sequence attributes is h = (h1, hL), h1 is the last feature matrix in the reverse time sequence information, hL is the last feature matrix in the forward time sequence information, and the shape is L×C, where L is the width of the last feature matrix in the visual feature extraction network, and C is the depth of the feature matrix.
[0067] Step S2023, pre-training the visual feature extraction network using the pre-constructed first data set to obtain a pre-trained visual feature extraction network.
[0068] Specifically, the server can pre-build a first data set for training the visual feature extraction network. The first data set includes sample text images and corresponding image labels. The sample text images are image carriers that record text content, and the image labels are visual content labels of the sample text images. The image labels can be obtained by staff annotation or by intelligent annotation using image technology.
[0069] After constructing the first data set, the server uses the first data set to train the constructed visual feature extraction network. During the training process, the parameters of the constructed visual feature extraction network are first initialized, and then the visual feature extraction network after the initialization is used to extract visual features of the sample image to obtain the training output. The error between the training output and the image label is calculated, and the error loss is calculated. With the goal of minimizing the error loss, the visual feature extraction network is iteratively trained until the visual feature extraction network converges or reaches a preset number of iterations, thereby completing the pre-training of the visual feature extraction network and obtaining a trained visual feature extraction network.
[0070] Step S2024, jointly training the pre-trained visual feature extraction network and the first temporal information extraction network using the pre-constructed second data set to obtain a jointly trained visual feature extraction network and a trained first temporal information extraction network.
[0071] Specifically, after the server completes pre-training of the visual feature extraction network, a pre-trained visual feature extraction network is obtained, and the pre-trained visual feature extraction network can be combined with the constructed first temporal information extraction network.
[0072] It should be noted that during the pre-training process of the constructed visual feature extraction network, the visual feature extraction network can extract multi-scale information from text images, and can effectively and sufficiently extract information about the text part of the image. The pre-trained visual feature extraction network is jointly trained with the constructed first temporal information extraction network, so that the visual feature extraction network and the first temporal information extraction network can be better fitted during the training process, thereby improving the accuracy of the visual feature matrix.
[0073] The second data set includes sample text images and text labels, and a preliminary first decoding network can be constructed for the first temporal information extraction network. After the visual feature extraction network extracts the visual features of the sample text images, the visual features are input into the temporal information extraction network to obtain visual features with temporal attributes. The visual features with temporal attributes are decoded through the first decoding network, and the corresponding first decoding results are output. The error between the first decoding result and the text label is calculated. With the goal of minimizing the error loss between the first decoding result and the text label, the pre-trained visual feature extraction network and the constructed first temporal information extraction network are jointly trained, and iterate until the first temporal information extraction network converges or reaches a preset number of iterations, to obtain the jointly trained visual feature extraction network and the trained first temporal information extraction network.
[0074] The server can construct a pre-trained visual feature extraction model based on the jointly trained visual feature extraction network and the trained first temporal information extraction network, and extract visual features of the text image to be recognized through the pre-trained visual feature extraction model to obtain a visual feature matrix of the text image to be recognized.
[0075] The jointly trained visual feature extraction network and the trained first temporal information extraction network are constructed into a pre-trained visual feature extraction model. It should be noted that the trained visual feature extraction model includes the jointly trained visual feature extraction network and the trained first temporal information extraction network, so that a visual feature matrix with temporal attributes can be extracted.
[0076] Visual feature extraction models such as Figure 3a As shown, Figure 3aThe figure is a schematic diagram of a visual feature extraction model in an embodiment. In a possible implementation, the first data set and the second data set can be merged into one data set, and the merged data includes sample text images and image labels and text labels corresponding to the sample text images. The visual feature extraction model corresponding to the merged data set is trained to obtain a trained visual feature extraction model, and the trained visual feature extraction model includes a jointly trained visual feature extraction network and a trained first temporal information extraction network.
[0077] In this embodiment, after constructing a visual feature extraction network based on a convolutional neural network and a first temporal information extraction network based on a recursive neural network, the visual feature extraction network is pre-trained using a pre-constructed first data set to obtain a pre-trained visual feature extraction network, and the pre-trained visual feature extraction network and the first temporal information extraction network are jointly trained using a pre-constructed second data set to obtain a jointly trained visual feature extraction network and a trained first temporal information extraction network, and based on the jointly trained visual feature extraction network and the trained first temporal information extraction network, a trained visual feature extraction model is constructed, so that the extracted visual feature matrix has features at multiple scales, thereby retaining the global information of the text image to be recognized, and at the same time extracting more accurate temporal information, thereby improving the accuracy of text recognition.
[0078] For further information, please refer to Figure 4 , Figure 4 The flowchart of an embodiment of pre-training a visual feature extraction network in the present application is shown. The steps of pre-training the visual feature extraction network using a pre-constructed first data set to obtain a pre-trained visual feature extraction network specifically include:
[0079] Step S20231, obtain a sample text image.
[0080] Specifically, the server may obtain sample text images in advance to construct the first data set. The sample text images may be obtained by the server from a network channel, or may be obtained by a user uploading them to the server.
[0081] Step S20232, outlining the text part in the sample text image word by word according to the text or sentence arrangement rules, and obtaining a text outlining image corresponding to each character in the sample text image.
[0082] Specifically, after the server obtains the sample text image, the sample text image can be distributed to the staff to outline the text in the sample text image. During the outlining process, the text or sentence in the sample text image is outlined word by word according to the arrangement rules of the text or sentence in the sample text image.
[0083] If the arrangement rule of the text or sentence in the sample text image is from top to bottom and from left to right, then the text or sentence is outlined word by word from top to bottom and from left to right to obtain the corresponding text outline image; if the arrangement rule of the text in the sample text image is from top to bottom and from right to left, then the text or sentence is outlined word by word from top to bottom and from right to left to obtain the corresponding text outline image.
[0084] After the server obtains the sample text image, the sample text image can also be used to outline the text through image segmentation technology. Specifically, the text in the sample text image is segmented through an image segmentation network to obtain the segmented area of each text, and then the segmented area of each text is automatically masked and outlined. The automatically masked and outlined text area forms a corresponding text outline image according to the arrangement rules of the sample text.
[0085] In one implementation method, the text sketch corresponding to a sample text image is a plurality of sequence images. A sequence image may include the sketches of all the text in a sentence, that is, there are as many sequence images in the text sketch as there are sentences in the sample text image, and the sequence images are sorted according to the arrangement rules of the sentences.
[0086] In another implementation method, the text sketch corresponding to a sample text image is a plurality of sequence images. A sequence image may include a sketch corresponding to a text, that is, there are as many sequence images in the text sketch as there are characters in the sample text image, and the sequence images are sorted according to the arrangement rules of the characters.
[0087] In another implementation method, for a sample text image, after associating the corresponding text sketch, the sample text image can be randomly masked to add noise to the sample text image to reduce the image quality of the sample text image. In this way, the robustness of the trained visual feature extraction network can be improved.
[0088] Furthermore, in the process of outlining the text, random strokes can be erased in the area corresponding to the text strokes in the sample image, so that the text strokes in the sample image are missing, but the corresponding strokes in the text outline image are complete strokes, which is equivalent to correctly masking the missing strokes. In this way, the robustness of the trained visual feature extraction network can be further improved.
[0089] Step S20233, using the corresponding text sketch as the image label of the sample text image, constructing a first data set, and dividing the first data set into a training set and a test set.
[0090] Specifically, after obtaining the text sketch, the server associates the text sketch with the corresponding sample text image, so that the text sketch forms an image label of the sample text image. After all sample text images are associated with the corresponding text sketch, a corresponding first data set is obtained. The first data set is divided into a training set and a test set. The training set may include a portion of sample text images and corresponding text sketches in the first data set, and the test set may include another portion of sample text images and corresponding text sketches in the first data set.
[0091] Step S20234, during the training process, for the training set, the mask output layer of the visual feature extraction network outputs the mask result of the sample text image.
[0092] Specifically, the visual feature extraction network constructed by the server includes a mask output layer, which outputs the mask result for the text area in the sample text image. The text area in the sample text image in the training set can be mapped to the corresponding local visual feature through the calculation of the first convolution layer and the second convolution layer in the visual feature extraction network, and the mask feature is linearly transformed through the mask output layer to output the corresponding mask result.
[0093] The mask output layer is a structure added during the pre-training process of the visual feature extraction network. After training, the mask output layer can be deleted to reduce the parameters of the model.
[0094] Step S20235, calculate the error loss between the mask result and the text outline image, and iteratively train the visual feature extraction network with the goal of minimizing the error loss until the visual feature extraction network converges in the test set or reaches a preset number of iterations, completing the pre-training of the visual feature extraction network and obtaining a pre-trained visual feature extraction network.
[0095] Specifically, after the server obtains the mask result of the sample text image, it can calculate the error between the mask result and the text outline. The smaller the error between the mask result and the text outline, the better the performance of the visual feature extraction network and the more accurate the extracted visual features. The visual feature extraction is iteratively trained through the minimum error loss until the visual feature extraction network converges in the test set or reaches a preset number of iterations, completing the pre-training of the visual feature extraction network and obtaining a trained visual feature extraction network.
[0096] The test set may include sample text images and corresponding text sketches. Specifically, the first data set may be divided into a training set and a test set, and the training set is used to train the visual feature extraction network, while the test set does not participate in the training of the visual feature extraction network by the first data set, but only participates in the effect test of the visual feature extraction network.
[0097] The error loss used in this application can be a dice loss function, which is as follows:
[0098]
[0099] Among them, A represents the mask result generated by the visual feature extraction network, which is the mask area corresponding to each character in the sample text image, and B represents the text outline area in the text outline image. After the pre-training of the visual feature extraction network is completed, because the visual feature extraction network can completely generate the mask of each text area in the sample text image, it can be considered that the visual feature extraction network can extract multi-scale information from the text image and can effectively and sufficiently extract the information of the text part in the image.
[0100] In this embodiment, the text part in the sample text image is outlined word by word according to the text or sentence arrangement rules to obtain a text outline image corresponding to each character in the sample text image, and the corresponding text outline image is used as the image label of the sample text image to construct the first data set. During the training process, the mask result of the sample text image is output through the mask output layer of the visual feature extraction network, and the error loss between the mask result and the text outline image is calculated. The visual feature extraction network is iteratively trained with the goal of minimizing the error loss, which can improve the accuracy and robustness of the visual feature extraction network, thereby improving the accuracy of text recognition.
[0101] For further information, please refer to Figure 5 , Figure 5 The flowchart of an embodiment of global semantic extraction in the present application is shown. The steps of performing global semantic extraction on the visual feature matrix to obtain the global semantic features of the text image to be recognized specifically include:
[0102] Step S2031, constructing a semantic model and a semantic enhancement model.
[0103] Specifically, the server can construct a semantic model and a semantic enhancement model, the semantic model is used to extract the semantic information contained in the visual feature extraction matrix, and the semantic enhancement model is used to correct the semantic information extracted by the semantic model to improve the accuracy of the semantic information. The semantic model can include an implicit feature extraction network and a second temporal information extraction network.
[0104] More specifically, the server can pre-acquire a classification feature model based on a convolutional neural network, and extract part of the computing network from the pre-acquired classification model to construct the implicit feature extraction network of the present application. The server can also obtain the implicit feature extraction network constructed by the staff themselves.
[0105] For example, the latent feature extraction network can be based on a neural network with a ResNet structure, specifically a neural network based on a 45-layer ResNet structure. The latent feature extraction network is used to extract the global semantics in the visual features into the latent space, thereby obtaining high-level latent features of the global semantics.
[0106] The server can pre-acquire a time series processing model based on a recursive neural network, and use the pre-acquired time series processing model to construct the second time series information extraction network of the present application. The server can also obtain the second time series information extraction network constructed by the staff themselves. The second time series information extraction network is constructed as a long short-term memory network structure, so that the second time series information extraction network can extract more accurate time series information at the semantic level.
[0107] The server can build a semantic enhancement model based on any language model, for example, a semantic enhancement model can be built based on a skip-gram model.
[0108] Step S2032: pre-train the semantic model using a preset third data set to obtain a pre-trained semantic model.
[0109] Specifically, after the server builds the semantic model, the semantic model can be pre-trained through a third data set, and the third data set includes a visual feature matrix extracted by the trained visual feature model and corresponding semantic labels. Furthermore, the visual feature matrix extracted by the trained visual feature model in the third data set can be extracted based on sample text images in the first data set or the second data set, or can be extracted based on text images from other data sources.
[0110] Alternatively, since the input of the semantic model is a visual feature matrix, and the output of the first temporal information extraction network is a visual feature matrix, the server can jointly train the constructed semantic model with the first temporal information extraction network, and jointly train the semantic model with the sample text images and text labels in the second data set. At this time, the semantic model includes a second decoding network, so that the semantic model can replace the first decoding network to decode and output the first temporal information extraction network.
[0111] Alternatively, the server visual feature extraction model and the semantic model are jointly trained, and the data set used for training is a data set obtained by merging the first data set and the second data set.
[0112] It should be noted that since the visual feature extraction network, the first temporal information extraction network, and the semantic model have a series property, that is, the output of the visual feature extraction network is the input of the first temporal information extraction network, and the output of the first temporal information extraction network is the input of the semantic model, the visual feature extraction network, the first temporal information extraction network, and the semantic model can be flexibly jointly trained, such as jointly training the visual feature extraction network with the first temporal information extraction network, or jointly training the first temporal information extraction network with the semantic model, or jointly training the visual feature extraction network, the first temporal information extraction network, and the semantic model.
[0113] Before the visual feature matrix is input into the semantic model, the visual feature matrix can be bilinearly sampled so that the semantic model can focus more on higher-level semantic features rather than details, thereby improving the robustness of the semantic model.
[0114] Bilinear sampling can also be called bilinear sampling or bilinear interpolation. It is a method of determining the sampled points in the graph, using the four known values around the sampled points to calculate the new values as the values of the sampling points, and inserting the values of the sampled points into the original graph. This is bilinear upsampling, and using the values of all sampled points as the values of the new graph is bilinear downsampling.
[0115] The visual feature matrix will be expanded into a one-dimensional feature vector I in the semantic model. The semantic information is obtained by linear processing of the implicit feature extraction network and the second temporal information extraction network in the semantic model. For details, please refer to the following formula:
[0116] S=W2σ(W1I+b1)+b2
[0117] Where W1, W2, b1, b2 are trainable weights in the linear function, and σ is the activation function.
[0118] Step S2033, pre-training the semantic enhancement model using a preset fourth data set to obtain a pre-trained semantic enhancement model.
[0119] Specifically, the server may construct a fourth data set based on the sample corpus acquired in advance to train the semantic enhancement model. The sample corpus may be acquired from a network channel or uploaded by a staff member.
[0120] The semantic enhancement model can be pre-trained separately using the fourth data set, and finally the pre-trained semantic enhancement model can be combined with the semantic model. The semantic enhancement model is used to perform semantic correction on the semantic information in the semantic model, so that the semantic model obtains more accurate semantic information.
[0121] Step S2034, using the semantic enhancement model to guide the semantic model to perform global semantic extraction on the visual feature matrix, so as to obtain the global semantic features of the text image to be recognized.
[0122] Specifically, the server uses the semantic enhancement model as a guide for the semantic model, making the global semantic features extracted by the semantic model more accurate. Since the semantic enhancement model can handle "out-of-vocabulary" problems in the semantic model, using the semantic enhancement model is equivalent to embedding a vector of certain visual features of the input. Therefore, the embedding vector in the semantic enhancement model is actually equivalent to a correction of the semantic model features.
[0123] For example, a vocabulary is a conversion tool that maps one type of state information to another type of state information, such as converting feature vector H1 into the word vector V1 of "I", converting feature vector H2 into the word vector V2 of "love", converting feature vector H3 into the word vector V3 of "of", converting feature vector H4 into the word vector V4 of "ancestor", converting feature vector H5 into the word vector V5 of "country", and converting feature vector H6 into the word vector V6 of "suffer". In general, the vocabulary in the semantic model can convert (H1, H2, H1, H3, H4, H5) into (V1, V2, V1, V3, V4, V5), and then decode it to get "I love my motherland". However, due to image quality reasons, the semantic model may get (H1, H2, H6, H3, H4, H5), and the ideolog model will convert (H1, H6, H1, H3, H4, H5) into (V1, V6, V1, V3, V4, V5), and finally decode it to get "I love my motherland". The semantic enhancement model embeds the word vector (V1, V2) of "I love". Therefore, it will be decoded through the embedding vector (V1, V2) of "I love". ), the embedding vector of “love me” (V2, V1), the embedding vector of “I love me” (V1, V2, V1), the embedding vector of “love mine” (V2, V1, V3), the embedding vector of “I love mine” (V1, V2, V1, V3), the embedding vector of “love my ancestor” (V2, V1, V3, V4), the embedding vector of “I love my ancestor” (V1, V2, V1, V3, V4), the embedding vector of “love my motherland” (V2, V1, V3, V4, V5) are used to guide the semantic model, so that the semantics are converted from (H1, H6, H1, H3, H4, H5) to (V1, V2, V1, V3, V4, V5).
[0124] In this embodiment, the semantic information in the semantic model can be semantically corrected using a pre-trained semantic enhancement model, so that the semantic model can obtain more accurate semantic information, thereby further improving the accuracy of text recognition.
[0125] For further information, please refer to Figure 6 , Figure 6 The flowchart of an embodiment of pre-training the semantic enhancement model in the present application is shown. The steps of pre-training the semantic enhancement model by using the preset fourth data set to obtain the pre-trained semantic enhancement model specifically include:
[0126] Step S20331, obtaining sample corpus.
[0127] Specifically, the service can obtain sample corpus from online channels or from staff uploads. The sample corpus includes sentence texts of a preset length. For a sentence, it can be represented by T = {wi-k, wi+k}, where wi is the word in the sentence and k represents the length of the sentence.
[0128] Step S20332, training a preset feedforward network through the sample corpus, and optimizing the embedding vector during the training process to obtain an embedding vector that is semantically close to the sample corpus.
[0129] Specifically, the semantic enhancement model constructed by the server includes a preset feedforward network. The word wi can be represented by an embedding vector vi and then input into the preset feedforward network. The preset feedforward network is a feedforward neural network. The purpose of the preset feedforward network is to predict Ci = {wi-k, wi-1, wi+1, wi+k}.
[0130] More specifically, after the server obtains the sample corpus, it will additionally embed sub-words of the words in the sample corpus through a preset feed-forward network, and use these sub-words to generate the final word embedding vector of word wi.
[0131] For example, given the hyperparameters lmin and lmax, which represent the minimum and maximum lengths of subwords, for example, given lmin=2, lmax=5, the word is "where", the set of subwords is {wh, he, er, re, whe, her, ere, wher, here}.
[0132] After the server obtains the sample corpus, it will use the preset feed-forward network to embed sub-sentences in the sentences in the sample corpus, and use these sub-sentences to generate the final sentence embedding vector of sentence wi.
[0133] For example, given lmin=2, lmax=4, the sentence is "I love my motherland", the set of subwords is {I love, love me, my, ancestor, motherland, I love me, love my, my ancestor, motherland, I love my, love my ancestor, my motherland, I love my ancestor, love my motherland}
[0134] The embedded sub-sentences and embedded sub-words can be nested. The sentences in a sample corpus can be embedded into sub-sentences, and then the words in the sub-sentences can be embedded into sub-words. This can further extract the global semantics from the sample semantics.
[0135] Step S20333, combining the embedding vector with the corresponding word or sentence itself to obtain a representation of the word or sentence, and obtaining a pre-trained semantic enhancement model through the representation of the word or sentence.
[0136] After the server obtains the representation of the word or sentence, the representation of the word or sentence can be used as a pre-trained semantic enhancement model. The pre-trained semantic enhancement model is used to perform semantic correction on the semantic information in the semantic model, so that the semantic model obtains more accurate semantic information.
[0137] For example, given lmin=2, lmax=4, the sentence is "I love my motherland", the set of sub-words is {I love, love me, my, ancestor, motherland, I love me, love my, my ancestor, motherland, I love my, love my ancestor, my motherland, I love my ancestor, love my motherland}.
[0138] In this embodiment, the pre-trained semantic enhancement model can perform semantic correction on the semantic information in the semantic model, so that the semantic model can obtain more accurate semantic information.
[0139] For further information, please refer to Figure 7 , Figure 7 The flowchart of an embodiment of feature decoding in the present application is shown. Before step S204, it also includes:
[0140] Step S2041, construct a decoding model.
[0141] Specifically, the server can pre-acquire a classification model based on a convolutional neural network, extract the classification part of the network from the pre-acquired classification model to construct the decoding model of the present application, and add an attention mechanism to the constructed decoding model. The server can also obtain a decoding model containing an attention mechanism that is constructed by the staff themselves.
[0142] More specifically, the decoding model can be corrected using the Bahdanau attention mechanism, consisting of a single-layer attention GRU network, including 512 hidden units and 512 attention units. Preferably, the present application can use a unidirectional decoder. In particular, the dimension can be transformed by a linear function, the state of the GRU network can be initialized using semantic information, and the global semantic features can be used as state parameters as the state parameters of the decoding model to avoid the decoding model from using zero state initialization or random initialization, so that the decoding model does not need an initialization process. While improving the decoding speed, the global semantic features can be used as a guide, not only using local visual information for decoding, but also using global semantic information to produce more accurate results, and coherent and semantically valid recognition text can be produced. Preferably, the structure of the decoding model can be composed of the classification part of the resnet50 structure and the GRU network.
[0143] Step S2042: Based on the preset fifth data set, the decoding model is trained using the recognition loss function and the embedding loss function to obtain a trained decoding model.
[0144] Specifically, after constructing the model, the server constructs a corresponding fifth data set, wherein the fifth data set may include the second data set and the fourth data set, the second data set is used to calculate the recognition loss function, and the fourth data set is used to calculate the embedding loss function.
[0145] The training of the decoding model can be combined with the above-mentioned visual feature extraction model, semantic model and semantic enhancement model for simultaneous training, and the training of all models can be completed by combining the first data set, the second data set and the fourth data set. Of course, the above-mentioned visual feature extraction model, semantic model and semantic enhancement model can be pre-trained. When training together with the decoding model, only the parameters in the decoding model can be adjusted, and the parameters of the visual feature extraction model, the semantic model and the semantic enhancement model can be kept unchanged.
[0146] Specifically, the loss function is shown in the following formula:
[0147] L=L rec +λL sem
[0148] Among them, Lrec is the recognition loss function, Lsem is the embedding loss function, and λ is the preset weight parameter, which is a priori value.
[0149] The recognition loss function can adopt the cross entropy loss function, and the embedding loss function Lsem can be shown as follows:
[0150] L sem =1-cos(S,em)
[0151] Where S is the text result predicted by the decoding model and em is the embedding vector in the pre-trained semantic enhancement model.
[0152] The trained decoding model is used to decode the visual feature matrix with global semantic features as state parameters through the attention mechanism to obtain the recognition text of the text image to be recognized.
[0153] In this embodiment, the recognition error and the embedding error can be considered in the training process to improve the decoding accuracy of the decoding model, thereby further improving the accuracy of text recognition. At the same time, the global semantic features are used as the state parameters of the decoding model to avoid the decoding model from using zero state initialization or random initialization, so that the decoding model does not need an initialization process. While improving the decoding speed, the global semantic features can be used as a guide, not only using local visual information for decoding, but also using global semantic information to produce more accurate results, and can produce coherent and semantically valid recognition text.
[0154] This application can be applied in the field of smart cities to promote the construction of smart cities. For example, this application can be applied in the field of smart cities, such as advertisement recognition, license plate recognition, entity text recognition and other application fields involving text images.
[0155] Further references Figure 8 , as a response to the above Figure 2 The present application provides an embodiment of a text image recognition device based on global semantics, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0156] like Figure 8 As shown, the text image recognition device 800 based on global semantics described in this embodiment includes: an acquisition module 801, a first extraction module 802, a second extraction module 803 and a decoding module 804, wherein:
[0157] The acquisition module 801 is used to acquire a text image to be recognized.
[0158] The first extraction module 802 is used to extract visual features of the text image to be recognized through the visual part of the pre-trained visual feature extraction model, and to extract temporal information of the extracted visual features through the temporal part of the pre-trained visual feature extraction model to obtain a visual feature matrix of the text image to be recognized.
[0159] The second extraction module 803 is used to perform global semantic extraction on the visual feature matrix to obtain global semantic features of the text image to be recognized.
[0160] The decoding module 804 is used to use the global semantic features as state parameters and perform feature decoding on the visual feature matrix through the attention mechanism in the trained decoding model to obtain the recognition text of the text image to be recognized.
[0161] In this embodiment, global semantic extraction is performed on the visual feature matrix to obtain the global semantic features of the text image to be recognized. Guided by the global semantic features, the visual features are feature decoded through the attention mechanism to obtain the recognized text of the text image to be recognized. Since the process of decoding the visual features through the attention mechanism is guided by the global semantic features, the global semantics is referenced on the basis of the decoding of the local vision, which makes the recognized text more accurate and improves the text recognition accuracy for low-quality images.
[0162] In some optional implementations of this embodiment, the visual part of the pre-trained visual feature extraction model is a visual feature extraction network after joint training, and the temporal part of the pre-trained visual feature extraction model is a trained first temporal information extraction network, and the first temporal information extraction network is used to extract temporal information at the visual level; the global semantics-based text image recognition device 800 described in this embodiment also includes: a first construction module, a second construction module, a first training module, and a second training module and a third construction module, wherein:
[0163] A first construction module is used to construct a visual feature extraction network based on a convolutional neural network, wherein the visual feature extraction network includes a preset number of first convolutional layers and second convolutional layers corresponding to the first convolutional layers one by one, and the second convolutional layer is used to perform convolution calculation on the convolution result of the previous second convolutional layer and the convolution result of the corresponding first convolutional layer;
[0164] A second construction module is used to construct a first time series information extraction network based on a recursive neural network, wherein the first time series information extraction network includes a time series processing unit corresponding to the second convolutional layer one by one, and the time series processing unit is used to perform time series processing on the processing result of the previous time series processing unit and the convolution result of the corresponding second convolutional layer;
[0165] A first training module is used to pre-train the visual feature extraction network using a pre-constructed first data set to obtain a pre-trained visual feature extraction network;
[0166] The second training module is used to jointly train the pre-trained visual feature extraction network and the first temporal information extraction network through a pre-constructed second data set to obtain a jointly trained visual feature extraction network and a trained first temporal information extraction network.
[0167] In some optional implementations of this embodiment, the first training module includes: a first acquisition unit, a delineation unit, a construction unit, an output unit, and a calculation unit, wherein:
[0168] A first acquisition unit, used to acquire a sample text image;
[0169] A delineation unit is used to delineate the text part in the sample text image word by word according to the text or sentence arrangement rules to obtain a text delineation image corresponding to each text in the sample text image;
[0170] A construction unit, configured to construct the first data set by using the corresponding text sketch as the image label of the sample text image, and divide the first data set into a training set and a test set;
[0171] An output unit, used for outputting the mask result of the sample text image through the mask output layer of the visual feature extraction network during the training process;
[0172] The calculation unit is used to calculate the error loss between the mask result and the text outline image, and iteratively train the visual feature extraction network with the goal of minimizing the error loss until the visual feature extraction network converges in the test set or reaches a preset number of iterations, thereby completing the pre-training of the visual feature extraction network and obtaining a pre-trained visual feature extraction network.
[0173] In some optional implementations of this embodiment, the second extraction module 803 includes: a construction submodule, a first training submodule, a second training submodule and an extraction submodule, wherein:
[0174] A construction submodule is used to construct a semantic model and a semantic enhancement model, wherein the semantic model includes an implicit feature extraction network and a second temporal information extraction network, and the second temporal information extraction network is used to extract temporal information at the semantic level;
[0175] A first training submodule is used to pre-train the semantic model using a preset third data set to obtain a pre-trained semantic model;
[0176] A second training submodule is used to pre-train the semantic enhancement model using a preset fourth data set to obtain a pre-trained semantic enhancement model;
[0177] The extraction submodule is used to guide the semantic model to perform global semantic extraction on the visual feature matrix through the semantic enhancement model to obtain the global semantic features of the text image to be recognized.
[0178] In some optional implementations of this embodiment, the second training submodule includes: a second acquisition unit, a training unit, and a combining unit, wherein:
[0179] A second acquisition unit, used for acquiring sample corpus;
[0180] A training unit, used to train a preset feedforward network through a sample corpus, and optimize the embedding vector during the training process to obtain an embedding vector that is semantically close to the sample corpus;
[0181] The combining unit is used to combine the embedding vector with the corresponding word or sentence itself to obtain the representation of the word or sentence, and obtain a pre-trained semantic enhancement model through the representation of the word or sentence.
[0182] In some optional implementations of this embodiment, the text image recognition device 800 based on global semantics further includes: a third construction module and a third training module, wherein:
[0183] The third construction submodule is used to construct a decoding model;
[0184] The third training submodule is used to train the decoding model based on a preset fifth data set by using a recognition loss function and an embedding loss function to obtain a trained decoding model, and the fifth data set includes the second data set and the third data set.
[0185] In some optional implementations of this embodiment, the text image recognition device 800 based on global semantics further includes: a normalization module, wherein:
[0186] The normalization module is used to perform 0-1 normalization processing on the image to be identified to obtain a normalized image to be identified.
[0187] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.
[0188] The computer device 9 includes a memory 91, a processor 92, and a network interface 93 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 9 with components 91-93, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.
[0189] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0190] The memory 91 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 91 can be an internal storage unit of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 91 can also be an external storage device of the computer device 9, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 9. Of course, the memory 91 can also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 91 is generally used to store the operating system and various application software installed on the computer device 9, such as computer-readable instructions of a text image recognition method based on global semantics, etc. In addition, the memory 91 can also be used to temporarily store various types of data that have been output or are to be output.
[0191] The processor 92 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 92 is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 92 is used to run computer-readable instructions or process data stored in the memory 91, such as computer-readable instructions for running the global semantics-based text image recognition method.
[0192] The network interface 93 may include a wireless network interface or a wired network interface. The network interface 93 is generally used to establish a communication connection between the computer device 9 and other electronic devices.
[0193] The computer device provided in this embodiment can execute the steps of the above-mentioned method for character image recognition based on global semantics. The steps of the method for character image recognition based on global semantics here can be the steps of the method for character image recognition based on global semantics in the above-mentioned various embodiments.
[0194] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the global semantics-based text image recognition method as described above.
[0195] In this embodiment, after obtaining the text image to be recognized, the visual feature matrix of the text image to be recognized is extracted. The visual feature matrix is time-series processed and has time-series properties. Global semantic extraction is performed on the visual feature matrix to obtain the global semantic features of the text image to be recognized. Guided by the global semantic features, the visual features are feature decoded through the attention mechanism to obtain the recognized text of the text image to be recognized. Since the process of decoding the visual features through the attention mechanism is guided by the global semantic features, the global semantics is referenced on the basis of decoding the local vision, which makes the recognized text more accurate and improves the text recognition accuracy for low-quality images.
[0196] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0197] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. A text image recognition method based on global semantics, characterized in that: The steps include: Obtain the text image to be recognized; The visual features of the to-be-recognized text image are extracted by using the visual part of the pre-trained visual feature extraction model, and the temporal information of the extracted visual features is extracted by using the temporal part of the pre-trained visual feature extraction model, so as to obtain a visual feature matrix of the to-be-recognized text image, wherein the visual part of the pre-trained visual feature extraction model is a visual feature extraction network after joint training, and the temporal part of the pre-trained visual feature extraction model is a trained first temporal information extraction network, and the trained first temporal information extraction network is used to extract temporal information at the visual level; Performing global semantic extraction on the visual feature matrix to obtain global semantic features of the text image to be recognized; Using the global semantic features as state parameters, the visual feature matrix is feature decoded by the attention mechanism in the trained decoding model to obtain the recognition text of the text image to be recognized; Before the step of extracting visual features of the to-be-recognized text image by using the visual part of the pre-trained visual feature extraction model, and extracting temporal information of the extracted visual features by using the temporal part of the pre-trained visual feature extraction model, to obtain the visual feature matrix of the to-be-recognized text image, the method further includes: Constructing a visual feature extraction network based on a convolutional neural network, wherein the visual feature extraction network includes a preset number of first convolutional layers and second convolutional layers corresponding to the first convolutional layers one by one, wherein the second convolutional layers are used to perform convolution calculation on the convolution result of the previous second convolutional layer and the convolution result of the corresponding first convolutional layer; Constructing a first time series information extraction network based on a recursive neural network, wherein the first time series information extraction network includes a time series processing unit corresponding to the second convolution layer one by one, and the time series processing unit is used to perform time series processing on the processing result of the previous time series processing unit and the convolution result of the corresponding second convolution layer; Pre-training the visual feature extraction network using a pre-constructed first data set to obtain a pre-trained visual feature extraction network; The pre-trained visual feature extraction network and the first temporal information extraction network are jointly trained by using a pre-constructed second data set to obtain a jointly trained visual feature extraction network and a trained first temporal information extraction network; The step of pre-training the visual feature extraction network by using the pre-constructed first data set to obtain a pre-trained visual feature extraction network comprises: Get sample text image; Outlining the text part in the sample text image word by word according to the text or sentence arrangement rules to obtain a text outline image corresponding to each text in the sample text image; Using the corresponding text sketch as the image label of the sample text image, constructing the first data set, and dividing the first data set into a training set and a test set; During the training process, for the training set, the mask result of the sample text image is output through the mask output layer of the visual feature extraction network; The error loss between the mask result and the text outline is calculated, and the visual feature extraction network is iteratively trained with the goal of minimizing the error loss until the visual feature extraction network converges in a test set or reaches a preset number of iterations, thereby completing the pre-training of the visual feature extraction network and obtaining a pre-trained visual feature extraction network.
2. The method for character image recognition based on global semantics according to claim 1, characterized in that: The step of performing global semantic extraction on the visual feature matrix to obtain the global semantic features of the text image to be recognized comprises: Constructing a semantic model and a semantic enhancement model, wherein the semantic model includes a latent feature extraction network and a second temporal information extraction network, and the second temporal information extraction network is used to extract temporal information at a semantic level; Pre-training the semantic model using a preset third data set to obtain a pre-trained semantic model; Pre-training the semantic enhancement model by using a preset fourth data set to obtain a pre-trained semantic enhancement model; The semantic enhancement model guides the semantic model to perform global semantic extraction on the visual feature matrix to obtain the global semantic features of the text image to be recognized.
3. The method for character image recognition based on global semantics according to claim 2, characterized in that: The step of pre-training the semantic enhancement model by using the preset fourth data set to obtain the pre-trained semantic enhancement model comprises: Get sample corpus; Training a preset feedforward network with the sample corpus, optimizing the embedding vector during the training process, and obtaining an embedding vector that is semantically close to the sample corpus; The embedding vector is combined with the corresponding word or sentence itself to obtain the representation of the word or sentence, and a pre-trained semantic enhancement model is obtained through the representation of the word or sentence.
4. The method for character image recognition based on global semantics according to claim 2, characterized in that: Before the step of taking the global semantic features as state parameters and decoding the visual feature matrix by the attention mechanism in the trained decoding model to obtain the recognition text of the text image to be recognized, the step further includes: Build a decoding model; Based on a preset fifth data set, the decoding model is trained using a recognition loss function and an embedding loss function to obtain a trained decoding model, and the fifth data set includes the second data set and the third data set.
5. The method for character image recognition based on global semantics according to any one of claims 1 to 4, characterized in that: Before the step of taking the global semantic features as state parameters and decoding the visual feature matrix by the attention mechanism in the trained decoding model to obtain the recognized text of the text image to be recognized, the method further includes: The to-be-recognized text image is subjected to 0-1 normalization processing to obtain a normalized to-be-recognized text image.
6. A text image recognition device based on global semantics, characterized in that: include: An acquisition module, used for acquiring a text image to be recognized; A first extraction module is used to extract visual features of the to-be-recognized text image through the visual part of the pre-trained visual feature extraction model, and to extract temporal information of the extracted visual features through the temporal part of the pre-trained visual feature extraction model, so as to obtain a visual feature matrix of the to-be-recognized text image, wherein the visual part of the pre-trained visual feature extraction model is a jointly trained visual feature extraction network, and the temporal part of the pre-trained visual feature extraction model is a trained first temporal information extraction network, and the trained first temporal information extraction network is used to extract temporal information at the visual level; A second extraction module is used to perform global semantic extraction on the visual feature matrix to obtain global semantic features of the text image to be recognized; A decoding module, used to use the global semantic features as state parameters and perform feature decoding on the visual feature matrix through an attention mechanism in a trained decoding model to obtain a recognition text of the text image to be recognized; The device further comprises: a first construction module, a second construction module, a first training module and a second training module, wherein: The first construction module is used to construct a visual feature extraction network based on a convolutional neural network, wherein the visual feature extraction network includes a preset number of first convolutional layers and second convolutional layers corresponding to the first convolutional layers one by one, and the second convolutional layers are used to perform convolution calculation on the convolution result of the previous second convolutional layer and the convolution result of the corresponding first convolutional layer; The second construction module is used to construct a first time series information extraction network based on a recursive neural network, wherein the first time series information extraction network includes a time series processing unit corresponding to each of the second convolutional layers, and the time series processing unit is used to perform time series processing on a processing result of a previous time series processing unit and a convolution result of the corresponding second convolutional layer; The first training module is used to pre-train the visual feature extraction network using a pre-constructed first data set to obtain a pre-trained visual feature extraction network; A second training module is used to jointly train the pre-trained visual feature extraction network and the first temporal information extraction network through a pre-constructed second data set to obtain a jointly trained visual feature extraction network and a trained first temporal information extraction network; The first training module includes: a first acquisition unit, a delineation unit, a construction unit, an output unit and a calculation unit, wherein: The first acquisition unit is used to acquire a sample text image; The outlining unit is used to outline the text part in the sample text image word by word according to the text or sentence arrangement rules to obtain a text outline image corresponding to each text in the sample text image; The construction unit is used to construct the first data set by using the corresponding text sketch as the image label of the sample text image, and divide the first data set into a training set and a test set; The output unit is used to output the mask result of the sample text image through the mask output layer of the visual feature extraction network for the training set during the training process; The calculation unit is used to calculate the error loss between the mask result and the text outline image, and iteratively train the visual feature extraction network with the goal of minimizing the error loss until the visual feature extraction network converges in a test set or reaches a preset number of iterations, thereby completing the pre-training of the visual feature extraction network and obtaining a pre-trained visual feature extraction network.
7. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the global semantics-based text image recognition method as described in any one of claims 1 to 5 are implemented.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the text image recognition method based on global semantics as described in any one of claims 1 to 5 are implemented.
