Text recognition method, device, equipment and storage medium based on feature fusion

Abstract

The present invention relates to the field of artificial intelligence, and discloses a text recognition method, device, equipment and storage medium based on feature fusion, which are used to improve the accuracy of text recognition. A preset neural network group is called to perform feature extraction on a medical image to be identified, generate a medical local feature image and a medical global feature image, and a preset weighted sum function is called to perform feature fusion on the medical local feature image and the medical global feature image to generate a target medical fusion feature image; a preset bidirectional two-layer recurrent neural network is called to perform character set extraction on the target medical fusion feature image to generate a medical classification character set, and alignment decoding is performed based on the medical classification character set and a preset time series classification algorithm to generate a target medical text. In addition, the present invention also relates to blockchain technology, and the medical image to be identified can be stored in the blockchain.

Description

Technical Field

[0001] The present invention relates to the field of neural networks, and in particular to a text recognition method, device, equipment and storage medium based on feature fusion. Background Art

[0002] With the development of science and technology, artificial intelligence is applied in more and more fields. Among them, text recognition is an important field in artificial intelligence, and has been successfully applied to many businesses such as ID card recognition, bank card recognition, medical diagnosis certificate recognition, and medical card recognition.

[0003] In the prior art, convolutional neural networks are used to perform text recognition on images. Convolutional neural networks mainly focus on the extraction of local features. Due to the complex background of the image, convolutional neural networks cannot combine global information to distinguish between text and background, resulting in inaccurate text recognition. Summary of the invention

[0004] The present invention provides a text recognition method, device, equipment and storage medium based on feature fusion, which are used to improve the accuracy of text recognition.

[0005] The first aspect of the present invention provides a text recognition method based on feature fusion, comprising: calling a preset neural network group to perform feature extraction on a medical image to be identified, generating a medical local feature image and a medical global feature image, and calling a preset weighted sum function to perform feature fusion on the medical local feature image and the medical global feature image, generating a target medical fusion feature image; calling a preset bidirectional two-layer recurrent neural network to perform character set extraction on the target medical fusion feature image, generating a medical classification character set, and performing alignment decoding based on the medical classification character set and a preset time series classification algorithm to generate a target medical text.

[0006] Optionally, in a first implementation method of the first aspect of the present invention, the calling of a preset neural network group to perform feature extraction on the medical image to be identified, generate a medical local feature image and a medical global feature image, and calling a preset weighted sum function to perform feature fusion on the medical local feature image and the medical global feature image to generate a target medical fusion feature image, including: obtaining an initial medical image, and preprocessing the initial medical image to generate a preprocessed medical image, wherein the initial medical image includes an initial medical text; normalizing the preprocessed medical image to generate a medical image to be identified, wherein the medical image to be identified includes a medical text to be identified; calling a convolutional neural network preset in the neural network group to perform local feature extraction on the medical image to be identified, and generate a medical local feature image; calling a preset attention neural network in the neural network group to perform global feature extraction on the medical image to be identified, and generate a medical global feature image; calling a preset weighted sum function to perform feature fusion on the medical local feature image and the medical global feature image to generate a target medical fusion feature image.

[0007] Optionally, in a second implementation method of the first aspect of the present invention, the acquiring of an initial medical image and preprocessing of the initial medical image to generate a preprocessed medical image, wherein the initial medical image includes an initial medical text, comprises: acquiring an initial medical image, and determining whether a direction of the initial medical text in the initial medical image is a preset standard direction, wherein the initial medical image includes an initial medical text; if the direction of the initial medical text in the initial medical image is not the preset standard direction, then adjusting the initial medical image according to the preset standard direction in combination with the initial medical text to generate a preprocessed medical image.

[0008] Optionally, in a third implementation method of the first aspect of the present invention, the normalizing the preprocessed medical image to generate a medical image to be identified, wherein the medical image to be identified includes medical text to be identified, including: scaling the preprocessed medical image according to a preset scaling rule to generate a scaled medical image; normalizing the scaled medical image to generate a medical image to be identified, wherein the medical image to be identified includes medical text to be identified.

[0009] Optionally, in a fourth implementation manner of the first aspect of the present invention, calling the convolutional neural network preset in the neural network group to perform local feature extraction on the medical image to be identified and generate a medical local feature image includes: calling the convolutional neural network preset in the neural network group to convolve the normalized medical image to be identified to generate a convolved medical feature image; performing batch normalization on the convolved medical feature image to generate a batch normalized medical feature image; calling an activation function to activate the batch normalized medical feature image to generate an activated medical feature image; inputting the activated medical feature image and the convolved medical feature image into the pooling layer in the convolutional neural network in the first group of neural networks for pooling to generate a first medical local feature image.

[0010] Optionally, in a fifth implementation manner of the first aspect of the present invention, calling the preset attention neural network in the neural network group to perform global feature extraction on the medical image to be identified and generating a medical global feature image includes: calling the preset attention neural network in the neural network group to perform sequence decomposition on the normalized medical image to be identified to generate a decomposed medical image to be identified; calling a preset linear feature extraction algorithm to encode the decomposed medical image to be identified to generate an encoded medical feature image; performing layer normalization on the encoded medical feature image to generate a first medical global feature image.

[0011] Optionally, in a sixth implementation manner of the first aspect of the present invention, the calling of a preset bidirectional two-layer recurrent neural network to perform character set extraction on the target medical fusion feature image to generate a medical classification character set, and aligning and decoding based on the medical classification character set and a preset time series classification algorithm to generate a target medical text includes: extracting a target medical fusion feature sequence from the target medical fusion feature image, and calling a preset bidirectional two-layer recurrent neural network to perform character set extraction on the target medical fusion feature sequence to generate a medical classification character set, wherein the target medical fusion feature sequence is the medical fusion feature sequence of the medical text to be identified; calling a preset probability function to perform probability calculation based on the medical classification character set to generate multiple medical character probabilities; calling a preset time series classification algorithm to align and decode the target medical fusion feature sequence in combination with the multiple medical character probabilities to generate a target medical text.

[0012] The second aspect of the present invention provides a text recognition device based on feature fusion, including: a feature extraction module, used to call a preset neural network group to perform feature extraction on a medical image to be identified, generate a medical local feature image and a medical global feature image, and call a preset weighted sum function to perform feature fusion on the medical local feature image and the medical global feature image to generate a target medical fusion feature image; a decoding module, used to call a preset bidirectional two-layer recurrent neural network to perform character set extraction on the target medical fusion feature image, generate a medical classification character set, and perform alignment decoding based on the medical classification character set and a preset time series classification algorithm to generate a target medical text.

[0013] Optionally, in a first implementation manner of the second aspect of the present invention, the feature extraction module includes: a preprocessing unit, used to acquire an initial medical image, and preprocess the initial medical image to generate a preprocessed medical image, wherein the initial medical image includes an initial medical text; a normalization processing unit, used to normalize the preprocessed medical image to generate a medical image to be identified, wherein the medical image to be identified includes a medical text to be identified; a local feature extraction unit, used to call a convolutional neural network preset in the neural network group to perform local feature extraction on the medical image to be identified, and generate a medical local feature image; a global feature extraction unit, used to call an attention neural network preset in the neural network group to perform global feature extraction on the medical image to be identified, and generate a medical global feature image; a feature fusion unit, used to call a preset weighted sum function to perform feature fusion on the medical local feature image and the medical global feature image, and generate a target medical fusion feature image.

[0014] Optionally, in a second implementation method of the second aspect of the present invention, the preprocessing unit can also be specifically used to: obtain an initial medical image, determine whether the direction of an initial medical text in the initial medical image is a preset standard direction, and the initial medical image includes an initial medical text; if the direction of the initial medical text in the initial medical image is not the preset standard direction, then adjust the initial medical image according to the preset standard direction in combination with the initial medical text to generate a preprocessed medical image.

[0015] Optionally, in a third implementation method of the second aspect of the present invention, the normalization processing unit can also be specifically used to: scale the preprocessed medical image according to a preset scaling rule to generate a scaled medical image; normalize the scaled medical image to generate a medical image to be identified, wherein the medical image to be identified includes medical text to be identified.

[0016] Optionally, in a fourth implementation manner of the second aspect of the present invention, the local feature extraction unit can also be specifically used to: call the convolutional neural network preset in the neural network group to convolve the normalized medical image to be identified to generate a convolved medical feature image; perform batch normalization on the convolved medical feature image to generate a batch normalized medical feature image; call an activation function to activate the batch normalized medical feature image to generate an activated medical feature image; input the activated medical feature image and the convolved medical feature image into the pooling layer in the convolutional neural network in the first group of neural networks for pooling to generate a first medical local feature image.

[0017] Optionally, in the fifth implementation manner of the second aspect of the present invention, the global feature extraction unit can also be specifically used to: call the preset attention neural network in the neural network group to perform sequence decomposition on the normalized medical image to be identified, and generate a decomposed medical image to be identified; call the preset linear feature extraction algorithm to encode the decomposed medical image to be identified, and generate an encoded medical feature image; perform layer normalization on the encoded medical feature image, and generate a first medical global feature image.

[0018] Optionally, in a sixth implementation manner of the second aspect of the present invention, the decoding module can also be specifically used to: extract a target medical fusion feature sequence from the target medical fusion feature image, and call a preset bidirectional two-layer recurrent neural network to perform character set extraction on the target medical fusion feature sequence to generate a medical classification character set, wherein the target medical fusion feature sequence is the medical fusion feature sequence of the medical text to be identified; call a preset probability function to perform probability calculation based on the medical classification character set to generate multiple medical character probabilities; call a preset time series classification algorithm in combination with the multiple medical character probabilities to align and decode the target medical fusion feature sequence to generate a target medical text.

[0019] The third aspect of the present invention provides a text recognition device based on feature fusion, comprising: a memory and at least one processor, wherein instructions are stored in the memory; the at least one processor calls the instructions in the memory so that the text recognition device based on feature fusion executes the above-mentioned text recognition method based on feature fusion.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned text recognition method based on feature fusion.

[0021] In the technical solution provided by the present invention, a preset neural network group is called to extract features of the medical image to be identified, generate a medical local feature image and a medical global feature image, and a preset weighted sum function is called to fuse the features of the medical local feature image and the medical global feature image to generate a target medical fusion feature image; a preset bidirectional two-layer recurrent neural network is called to extract a character set of the target medical fusion feature image to generate a medical classification character set, and alignment decoding is performed based on the medical classification character set and a preset temporal classification algorithm to generate a target medical text. In the embodiment of the present invention, a convolutional neural network and an attention neural network are called to extract local features and global features of the normalized medical image to be identified, and finally the extracted medical features are decoded to generate a target medical text; by using a convolutional neural network and an attention neural network to extract local features and global features of the normalized medical image to be identified, and fusing the two parts of features for text decoding, the influence of the image background on text recognition is eliminated, and both global features and local features can be considered in text recognition, thereby improving the accuracy of text recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of an embodiment of a text recognition method based on feature fusion in an embodiment of the present invention;

[0023] Figure 2 Schematic diagram of another embodiment of a text recognition method based on feature fusion in an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of an embodiment of a text recognition device based on feature fusion in an embodiment of the present invention;

[0025] Figure 4 It is a schematic diagram of another embodiment of a text recognition device based on feature fusion in an embodiment of the present invention;

[0026] Figure 5 The figure is a schematic diagram of an embodiment of a text recognition device based on feature fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The embodiments of the present invention provide a text recognition method, apparatus, device and storage medium based on feature fusion, which are used to improve the accuracy of text recognition.

[0028] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of a text recognition method based on feature fusion in an embodiment of the present invention includes:

[0030] 101. Calling a preset neural network group to extract features of the medical image to be identified, generating a medical local feature image and a medical global feature image, and calling a preset weighted summation function to perform feature fusion on the medical local feature image and the medical global feature image, generating a target medical fusion feature image;

[0031] The server calls a preset neural network group including a convolutional neural network and a self-attention neural network to perform global feature extraction on the medical image to be identified, generates a medical local feature image and a medical global feature image, and calls a preset weighted summation function to perform feature fusion on the medical local feature image and the medical global feature image to generate a target medical fusion feature image. It should be emphasized that in order to further ensure the privacy and security of the above-mentioned medical images to be identified, the above-mentioned medical images to be identified can also be stored in a node of a blockchain.

[0032] When performing text recognition on a medical image to be identified, the server calls a convolutional neural network to extract local features of the medical image to be identified, calls an attention neural network to extract global features of the medical image to be identified, and then calls a preset weighted sum function to fuse the extracted features to generate a more accurate target medical fusion feature image. The convolutional neural network is mainly used to extract shallow local features, while the attention neural network also pays attention to global information when extracting shallow information, thereby realizing the extraction of global features.

[0033] Since the semantic information of features extracted by the lower feature layer of the convolutional neural network is not rich enough, the server extracts global features through the high-level feature layer of the attention neural network, and then performs weighted fusion on these two features. The preset weighted sum function applied is:

[0034] g(x)=(1-α)f0(x)+αf1(x)

[0035] Among them, α represents the fusion ratio between fusion features, g(x) represents the target medical fusion feature image, f0(x) represents the local feature obtained by local feature extraction, and f1(x) represents the global feature obtained by global feature extraction.

[0036] It is understandable that the execution subject of the present invention can be a text recognition device based on feature fusion, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking the server as the execution subject as an example. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0037] 102. Call a preset bidirectional two-layer recurrent neural network to extract character sets from the target medical fusion feature image, generate a medical classification character set, and perform alignment decoding based on the medical classification character set and a preset time series classification algorithm to generate a target medical text.

[0038] After generating the target fused medical feature image, the server calls a bidirectional two-layer recurrent network and a temporal classification algorithm to perform character set extraction and alignment decoding on the target medical feature fused image, and generates a target medical text corresponding to the medical text to be identified in the medical image to be identified.

[0039] The server first uses a bidirectional double-layer recurrent network to perform sequence modeling on the target feature fusion image, and then generates a medical character set based on the model generated by the sequence modeling. The goal of sequence modeling is to strengthen the relationship between each node in the feature sequence. After the target medical feature fusion image passes through the model of sequence modeling, a medical model output feature map is generated. The height of the feature map is 1, which can be understood as a feature. The server then generates a medical character set based on the feature. The server then calls the preset time series classification algorithm to perform hit calculations on the target medical characters on the medical character set, and obtains a probability set. Finally, the server calls the preset time series classification algorithm to align and decode the probability set to generate the target medical text.

[0040] It should be noted that when performing probability calculation based on the medical classification character set, it is necessary to consider the length misalignment factor between the input and the output. Therefore, this embodiment introduces a time series classification algorithm for probability calculation. The time series classification algorithm is:

[0041]

[0042] Where y is any probability in the probability set, π is the character sequence corresponding to the probability, P(πx) is the character probability calculated based on the time series classification algorithm, t is the time, L T The server performs the above calculation based on each medical character in the medical character set, thereby obtaining multiple P(πx). After the multiple P(πx) are obtained, a one-to-one mapping is performed based on the maximum value of each P(πx) in the medical character set to determine multiple target medical characters, thereby obtaining a target medical text.

[0043] In an embodiment of the present invention, a convolutional neural network and an attention neural network are called to perform local feature extraction and global feature extraction on the normalized medical image to be identified, and finally the extracted medical features are decoded to generate target medical text; by using a convolutional neural network and an attention neural network to perform local feature extraction and global feature extraction on the normalized medical image to be identified, and fusing the two parts of features for text decoding, the influence of the image background on text recognition is eliminated, and both global features and local features can be taken into account in text recognition, thereby improving the accuracy of text recognition.

[0044] See also Figure 2 Another embodiment of the text recognition method based on feature fusion in the embodiment of the present invention includes:

[0045] 201. Acquire an initial medical image, and preprocess the initial medical image to generate a preprocessed medical image, wherein the initial medical image includes an initial medical text;

[0046] The server obtains an initial medical image including an initial medical text, and preprocesses the initial medical image to generate a preprocessed medical image.

[0047] In this embodiment, the server performs text recognition on the initial medical image including the initial medical text. Since only text in the horizontal direction can be recognized in this embodiment, the server needs to preprocess the initial medical image after acquiring the initial medical image to ensure that the direction of the initial medical text in the initial medical image is horizontal, and the server obtains the preprocessed medical image.

[0048] Specifically, the server obtains an initial medical image and determines whether the direction of the initial medical text in the initial medical image is a preset standard direction, and the initial medical image includes the initial medical text; if the direction of the initial medical text in the initial medical image is not the preset standard direction, the server adjusts the initial medical image according to the preset standard direction in combination with the initial medical text to generate a preprocessed medical image.

[0049] The server obtains an initial medical image including an initial text, and first determines whether the direction of the initial medical text in the initial medical image is a preset standard direction. In this embodiment, the preset standard direction is a horizontal direction. The server determines whether the direction of the initial medical text in the initial image is a horizontal direction. If it is determined that the direction of the initial medical text in the initial medical image is not a horizontal direction but a vertical direction or other directions, the direction of the initial medical image is adjusted until the direction of the initial medical text in the initial medical image is a horizontal direction. When the direction of the initial medical text in the initial medical image is a horizontal direction, the server obtains a preprocessed medical image.

[0050] 202. Performing normalization processing on the preprocessed medical image to generate a medical image to be identified, where the medical image to be identified includes a medical text to be identified;

[0051] The server normalizes the preprocessed medical image into a medical image to be recognized including the medical text to be recognized.

[0052] Specifically, the server scales the preprocessed medical image according to a preset scaling rule to generate a scaled medical image; the server normalizes the scaled medical image to generate a medical image to be identified, and the medical image to be identified includes medical text to be identified.

[0053] The server scales the preprocessed medical image according to the preset scaling rules to generate a scaled medical image. In this embodiment, the server scaling rules scale the height of the preprocessed medical image to 32 pixels, and then scale the width of the preprocessed medical image according to the scaling rules of equal proportion. If the scaled width is less than 320 pixels, the server supplements the pixels of the preprocessed medical image. In this embodiment, black pixels are used to supplement its width to 320 pixels, thereby generating a scaled medical image. After generating the scaled medical image, the server normalizes the scaled medical image, that is, divides the pixel value of the scaled medical image by 255 to obtain the medical image to be identified. The pixel values ​​of the medical image to be identified are all within the range of 0-1, so as to facilitate data processing.

[0054] 203. Calling a preset convolutional neural network in the neural network group to extract local features of the medical image to be identified, and generating a medical local feature image;

[0055] The server inputs the medical image to be identified into the convolutional neural network preset in the neural network group to extract local features and generate a medical local feature image.

[0056] Specifically, the server calls the preset convolutional neural network in the neural network group to convolve the normalized medical image to be identified to generate a convolved medical feature image; the server performs batch normalization on the convolved medical feature image to generate a batch normalized medical feature image; the server calls the activation function to activate the batch normalized medical feature image to generate an activated medical feature image; the server inputs the activated medical feature image and the convolved medical feature image into the pooling layer in the convolutional neural network in the first group of neural networks for pooling to generate a first medical local feature image.

[0057] When calling the convolutional neural network to extract local features of the medical image to be identified, the server first calls the convolutional neural network to convolve the medical image to be identified, and then generates a convolved medical feature image; the server then performs batch normalization on the convolved medical feature image, and batch normalization can be understood as normalizing the same data circle of the same batch of data in the convolved medical feature image, and the server obtains a batch normalized medical feature image; the server then calls the relu activation function to activate the batch normalized medical feature image to generate an activated medical feature image. After generating the activated medical feature image, the server then performs feature fusion on the activated medical feature image and the convolved medical feature image, and inputs the medical fusion result into the 2*2 pooling layer in the convolutional neural network for pooling to generate a first medical local feature image. In this embodiment, the width of the first medical local feature image is 80 pixels and the height is 8 pixels.

[0058] 204. Calling a preset attention neural network in the neural network group to extract global features of the medical image to be identified, and generating a medical global feature image;

[0059] The server inputs the medical image to be identified into the preset attention neural network in the neural network group to extract global features and generate a medical global feature image.

[0060] Specifically, the server calls the preset attention neural network in the neural network group to perform sequence decomposition on the normalized medical image to be identified, and generates a decomposed medical image to be identified; the server calls the preset linear feature extraction algorithm to encode the decomposed medical image to be identified, and generates an encoded medical feature image; the server performs layer normalization on the encoded medical feature image, and generates a first medical global feature image.

[0061] When calling the attention neural network to extract global features of the medical image to be identified, the server first calls the attention neural network to perform sequence decomposition of the medical image to be identified, decomposing it into a medical image sequence composed of 4*4 blocks, thereby generating a decomposed medical image to be identified; the server then encodes the decomposed medical image to be identified. In this embodiment, a preset linear feature extraction algorithm is mainly called to encode the decomposed medical image to be identified, thereby generating an encoded medical feature image; finally, the server performs layer normalization on the encoded medical feature image to generate a first medical global feature image. In this embodiment, the width of the first medical global feature image is 80 pixels and the height is 8 pixels.

[0062] It should be noted that in step 203 and step 204, in order to facilitate the description of feature extraction using one neural network group, in other embodiments, the number of neural network groups may be multiple groups.

[0063] 205. Calling a preset weighted sum function to perform feature fusion on the medical local feature image and the medical global feature image to generate a target medical fusion feature image;

[0064] The server calls a preset weighted sum function to fuse the medical local feature image and the medical global feature image to generate a target medical fusion feature image.

[0065] The preset weighted sum function is:

[0066] g(x)=(1-α)f0(x)+αf1(x)

[0067] Among them, α represents the fusion ratio between fusion features, g(x) represents the target medical fusion feature image, f0(x) represents the local feature obtained by local feature extraction, and f1(x) represents the global feature obtained by global feature extraction.

[0068] When performing text recognition on the medical image to be identified, the server calls the convolutional neural network to extract local features of the normalized medical image to be identified, calls the attention neural network to extract global features of the medical image to be identified, and then calls the preset weighted fusion function to fuse the extracted features to generate a more accurate target medical fusion feature image. The convolutional neural network is mainly used to extract shallow local features, while the attention neural network also pays attention to global information when extracting shallow information, thereby realizing the extraction of global features.

[0069] 206. Call a preset bidirectional two-layer recurrent neural network to extract character sets from the target medical fusion feature image, generate a medical classification character set, and perform alignment decoding based on the medical classification character set and a preset time series classification algorithm to generate a target medical text.

[0070] After generating the target fused medical feature image, the server calls a bidirectional two-layer recurrent network and a temporal classification algorithm to perform character set extraction and alignment decoding on the target medical feature fused image, and generates a target medical text corresponding to the medical text to be identified in the medical image to be identified.

[0071] The server first uses a bidirectional double-layer recurrent network to perform sequence modeling on the target feature fusion image, and then generates a medical character set based on the model generated by the sequence modeling. The goal of sequence modeling is to strengthen the relationship between each node in the feature sequence. After the target medical feature fusion image passes through the model of sequence modeling, a medical model output feature map is generated. The height of the feature map is 1, which can be understood as a feature. The server then generates a medical character set based on the feature. The server then calls the preset time series classification algorithm to perform hit calculations on the target medical characters on the medical character set, and obtains a probability set. Finally, the server calls the decoding function to perform character mapping on the probability set to generate the target medical text.

[0072] Specifically, the server extracts a target medical fusion feature sequence from a target medical fusion feature image, and calls a preset bidirectional two-layer recurrent neural network to perform character set extraction on the target medical fusion feature sequence to generate a medical classification character set. The target medical fusion feature sequence is a medical fusion feature sequence of a medical text to be identified; the server calls a preset probability function to perform probability calculation based on the medical classification character set to generate multiple medical character probabilities; the server calls a preset decoding function to decode the target medical fusion feature sequence in combination with multiple medical character probabilities to generate a target medical text.

[0073] The server first extracts the target medical fusion feature sequence from the target medical fusion feature image, and then calls a bidirectional two-layer recurrent neural network to extract the character set of the target medical fusion feature sequence to generate a medical classification character set, wherein the target medical fusion feature sequence is the medical fusion feature sequence of the medical text to be identified. In this embodiment, a general multi-eight-bit coded character set is mainly used. In other embodiments, the medical character set can also be a character set such as Chinese character encoding or five major codes. The server calls a bidirectional two-layer recurrent neural network to determine the characters in the general multi-eight-bit coded character set to generate a medical classification character set. Then the server calls a preset probability function softmax function to calculate the probability that the target medical fusion feature sequence matches each character in the medical classification character set, generates multiple medical character probabilities, and then the server decodes the target medical fusion feature sequence based on the preset temporal classification algorithm for each medical character probability and its corresponding target medical character, thereby generating a target medical text.

[0074] The time series classification algorithm is:

[0075]

[0076] Where y is any medical character probability among multiple medical character probabilities, π is the character sequence corresponding to the medical character probability, P(π|x) is the temporal character probability calculated based on the temporal classification algorithm, t is the time, L T The server performs the above calculation based on each medical character in the medical character set, thereby obtaining multiple P(π|x). After obtaining multiple P(π|x), the server performs one-to-one mapping based on the maximum value of each P(π|x) in the medical character set, determines multiple target medical characters, and obtains the target medical text.

[0077] In an embodiment of the present invention, a convolutional neural network and an attention neural network are called to perform local feature extraction and global feature extraction on the normalized medical image to be identified, and finally the extracted medical features are decoded to generate target medical text; by using a convolutional neural network and an attention neural network to perform local feature extraction and global feature extraction on the normalized medical image to be identified, and fusing the two parts of features for text decoding, the influence of the image background on text recognition is eliminated, and both global features and local features can be taken into account in text recognition, thereby improving the accuracy of text recognition.

[0078] The above describes the text recognition method based on feature fusion in the embodiment of the present invention. The following describes the text recognition device based on feature fusion in the embodiment of the present invention. Figure 3 In one embodiment of the present invention, a text recognition device based on feature fusion includes:

[0079] The feature extraction module 301 is used to call a preset neural network group to perform feature extraction on the medical image to be identified, generate a medical local feature image and a medical global feature image, and call a preset weighted sum function to perform feature fusion on the medical local feature image and the medical global feature image to generate a target medical fusion feature image;

[0080] The decoding module 302 is used to call a preset bidirectional two-layer recurrent neural network to extract the character set of the target medical fusion feature image, generate a medical classification character set, and perform alignment decoding based on the medical classification character set and a preset time series classification algorithm to generate a target medical text.

[0081] In an embodiment of the present invention, a convolutional neural network and an attention neural network are called to perform local feature extraction and global feature extraction on the normalized medical image to be identified, and finally the extracted medical features are decoded to generate target medical text; by using a convolutional neural network and an attention neural network to perform local feature extraction and global feature extraction on the normalized medical image to be identified, and fusing the two parts of features for text decoding, the influence of the image background on text recognition is eliminated, and both global features and local features can be taken into account in text recognition, thereby improving the accuracy of text recognition.

[0082] See also Figure 4 Another embodiment of the text recognition device based on feature fusion in the embodiment of the present invention includes:

[0083] The feature extraction module 301 is used to call a preset neural network group to perform feature extraction on the medical image to be identified, generate a medical local feature image and a medical global feature image, and call a preset weighted sum function to perform feature fusion on the medical local feature image and the medical global feature image to generate a target medical fusion feature image;

[0084] The decoding module 302 is used to call a preset bidirectional two-layer recurrent neural network to extract the character set of the target medical fusion feature image, generate a medical classification character set, and perform alignment decoding based on the medical classification character set and a preset time series classification algorithm to generate a target medical text.

[0085] Optionally, the feature extraction module 301 includes:

[0086] A preprocessing unit 3011 is used to obtain an initial medical image and preprocess the initial medical image to generate a preprocessed medical image, wherein the initial medical image includes an initial medical text;

[0087] A normalization processing unit 3012 is used to perform normalization processing on the preprocessed medical image to generate a medical image to be recognized, wherein the medical image to be recognized includes a medical text to be recognized;

[0088] A local feature extraction unit 3013 is used to call a convolutional neural network preset in the neural network group to extract local features of the medical image to be identified, and generate a medical local feature image;

[0089] A global feature extraction unit 3014 is used to call the attention neural network preset in the neural network group to extract global features of the medical image to be identified, and generate a medical global feature image;

[0090] The feature fusion unit 3015 is used to call a preset weighted sum function to perform feature fusion on the medical local feature image and the medical global feature image to generate a target medical fusion feature image.

[0091] Optionally, the preprocessing unit 3011 may also be specifically used for:

[0092] Acquire an initial medical image, and determine whether a direction of an initial medical text in the initial medical image is a preset standard direction, wherein the initial medical image includes the initial medical text;

[0093] If the direction of the initial medical text in the initial medical image is not a preset standard direction, the initial medical image is adjusted according to the preset standard direction in combination with the initial medical text to generate a preprocessed medical image.

[0094] Optionally, the normalization processing unit 3012 may also be specifically used for:

[0095] Scaling the preprocessed medical image according to a preset scaling rule to generate a scaled medical image;

[0096] The scaled medical image is normalized to generate a medical image to be identified, wherein the medical image to be identified includes medical text to be identified.

[0097] Optionally, the local feature extraction unit 3013 may also be specifically used for:

[0098] Calling a convolutional neural network preset in the neural network group to perform convolution on the normalized medical image to be identified, to generate a convolved medical feature image;

[0099] Performing batch normalization processing on the convolved medical feature image to generate a batch normalized medical feature image;

[0100] Calling an activation function to activate the batch normalized medical feature images to generate an activated medical feature image;

[0101] The activated medical feature image and the convolved medical feature image are input into a pooling layer in a convolutional neural network in the first group of neural networks for pooling to generate a first medical local feature image.

[0102] Optionally, the global feature extraction unit 3014 may also be specifically used for:

[0103] Calling the preset attention neural network in the neural network group to perform sequence decomposition on the normalized medical image to be identified, and generating a decomposition medical image to be identified;

[0104] Calling a preset linear feature extraction algorithm to encode the disassembled medical image to be identified to generate an encoded medical feature image;

[0105] The encoded medical feature image is subjected to layer normalization processing to generate a first medical global feature image.

[0106] Optionally, the decoding module 302 may also be specifically configured to:

[0107] Extracting a target medical fusion feature sequence from the target medical fusion feature image, and calling a preset bidirectional two-layer recurrent neural network to perform character set extraction on the target medical fusion feature sequence to generate a medical classification character set, wherein the target medical fusion feature sequence is the medical fusion feature sequence of the medical text to be identified;

[0108] Calling a preset probability function to perform probability calculation based on the medical classification character set to generate multiple medical character probabilities;

[0109] The preset time series classification algorithm is called to align and decode the target medical fusion feature sequence in combination with the multiple medical character probabilities to generate a target medical text.

[0110] In an embodiment of the present invention, a convolutional neural network and an attention neural network are called to perform local feature extraction and global feature extraction on the normalized medical image to be identified, and finally the extracted medical features are decoded to generate target medical text; by using a convolutional neural network and an attention neural network to perform local feature extraction and global feature extraction on the normalized medical image to be identified, and fusing the two parts of features for text decoding, the influence of the image background on text recognition is eliminated, and both global features and local features can be taken into account in text recognition, thereby improving the accuracy of text recognition.

[0111] above Figure 3 and Figure 4 The text recognition device based on feature fusion in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The text recognition device based on feature fusion in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0112] Figure 51 is a schematic diagram of the structure of a text recognition device based on feature fusion provided by an embodiment of the present invention. The text recognition device 500 based on feature fusion may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 may be temporary storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the text recognition device 500 based on feature fusion. Furthermore, the processor 510 may be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the text recognition device 500 based on feature fusion.

[0113] The feature fusion-based text recognition device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 5 The structure of the feature fusion-based text recognition device shown does not constitute a limitation on the feature fusion-based text recognition device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0114] The present invention also provides a text recognition device based on feature fusion, wherein the computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the text recognition method based on feature fusion in the above-mentioned embodiments.

[0115] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the feature fusion-based text recognition method.

[0116] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0117] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0118] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0119] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text recognition method based on feature fusion, characterized in that: The text recognition method based on feature fusion includes: Calling a preset neural network group to perform feature extraction on the medical image to be identified, generating a medical local feature image and a medical global feature image, and calling a preset weighted summation function to perform feature fusion on the medical local feature image and the medical global feature image, generating a target medical fusion feature image; Calling a preset bidirectional two-layer recurrent neural network to extract a character set from the target medical fusion feature image, generating a medical classification character set, and performing alignment decoding based on the medical classification character set and a preset time series classification algorithm to generate a target medical text; Among them, the calling of the preset neural network group to perform feature extraction on the medical image to be identified, generate a medical local feature image and a medical global feature image, and calling the preset weighted sum function to perform feature fusion on the medical local feature image and the medical global feature image to generate a target medical fusion feature image includes: obtaining an initial medical image, and preprocessing the initial medical image to generate a preprocessed medical image, wherein the initial medical image includes an initial medical text; normalizing the preprocessed medical image to generate a medical image to be identified, wherein the medical image to be identified includes a medical text to be identified; calling the preset convolutional neural network in the neural network group to convolve the normalized medical image to be identified to generate a convolved medical feature image; and performing batch normalization on the convolved medical feature image to generate a batch normalized medical feature image. image; calling an activation function to activate the batch normalized medical feature images to generate an activated medical feature image; inputting the activated medical feature image and the convolved medical feature image into the pooling layer of the convolutional neural network in the first group of neural networks for pooling to generate a first medical local feature image; calling the preset attention neural network in the neural network group to perform sequence decomposition on the normalized medical image to be identified to generate a decomposed medical image to be identified; calling a preset linear feature extraction algorithm to encode the decomposed medical image to be identified to generate an encoded medical feature image; performing layer normalization processing on the encoded medical feature image to generate a first medical global feature image; calling a preset weighted sum function to perform feature fusion on the medical local feature image and the medical global feature image to generate a target medical fusion feature image; The calling of a preset bidirectional two-layer recurrent neural network to perform character set extraction on the target medical fusion feature image to generate a medical classification character set, and performing alignment decoding based on the medical classification character set and a preset time series classification algorithm to generate a target medical text includes: extracting a target medical fusion feature sequence from the target medical fusion feature image, and calling a preset bidirectional two-layer recurrent neural network to perform character set extraction on the target medical fusion feature sequence to generate a medical classification character set, wherein the target medical fusion feature sequence is a medical fusion feature sequence of the medical text to be identified; calling a preset probability function to perform probability calculation based on the medical classification character set to generate multiple medical character probabilities; calling a preset time series classification algorithm to perform alignment decoding on the target medical fusion feature sequence in combination with the multiple medical character probabilities to generate a target medical text.

2. The text recognition method based on feature fusion according to claim 1 is characterized in that: The initial medical image is obtained, and the initial medical image is preprocessed to generate a preprocessed medical image, wherein the initial medical image includes an initial medical text including: Acquire an initial medical image, and determine whether a direction of an initial medical text in the initial medical image is a preset standard direction, wherein the initial medical image includes the initial medical text; If the direction of the initial medical text in the initial medical image is not a preset standard direction, the initial medical image is adjusted according to the preset standard direction in combination with the initial medical text to generate a preprocessed medical image.

3. The text recognition method based on feature fusion according to claim 1, characterized in that: The preprocessed medical image is normalized to generate a medical image to be identified, wherein the medical image to be identified includes the medical text to be identified: Scaling the preprocessed medical image according to a preset scaling rule to generate a scaled medical image; The scaled medical image is normalized to generate a medical image to be identified, wherein the medical image to be identified includes medical text to be identified.

4. A text recognition device based on feature fusion, characterized in that: The text recognition device based on feature fusion includes: A feature extraction module is used to call a preset neural network group to perform feature extraction on the medical image to be identified, generate a medical local feature image and a medical global feature image, and call a preset weighted summation function to perform feature fusion on the medical local feature image and the medical global feature image to generate a target medical fusion feature image; A decoding module, used for calling a preset bidirectional two-layer recurrent neural network to extract a character set from the target medical fusion feature image, generate a medical classification character set, and perform alignment decoding based on the medical classification character set and a preset time series classification algorithm to generate a target medical text; Among them, the calling of the preset neural network group to perform feature extraction on the medical image to be identified, generate a medical local feature image and a medical global feature image, and calling the preset weighted sum function to perform feature fusion on the medical local feature image and the medical global feature image to generate a target medical fusion feature image includes: obtaining an initial medical image, and preprocessing the initial medical image to generate a preprocessed medical image, wherein the initial medical image includes an initial medical text; normalizing the preprocessed medical image to generate a medical image to be identified, wherein the medical image to be identified includes a medical text to be identified; calling the preset convolutional neural network in the neural network group to convolve the normalized medical image to be identified to generate a convolved medical feature image; and performing batch normalization on the convolved medical feature image to generate a batch normalized medical feature image. image; calling an activation function to activate the batch normalized medical feature images to generate an activated medical feature image; inputting the activated medical feature image and the convolved medical feature image into the pooling layer of the convolutional neural network in the first group of neural networks for pooling to generate a first medical local feature image; calling the preset attention neural network in the neural network group to perform sequence decomposition on the normalized medical image to be identified to generate a decomposed medical image to be identified; calling a preset linear feature extraction algorithm to encode the decomposed medical image to be identified to generate an encoded medical feature image; performing layer normalization processing on the encoded medical feature image to generate a first medical global feature image; calling a preset weighted sum function to perform feature fusion on the medical local feature image and the medical global feature image to generate a target medical fusion feature image; The calling of a preset bidirectional two-layer recurrent neural network to perform character set extraction on the target medical fusion feature image to generate a medical classification character set, and performing alignment decoding based on the medical classification character set and a preset time series classification algorithm to generate a target medical text includes: extracting a target medical fusion feature sequence from the target medical fusion feature image, and calling a preset bidirectional two-layer recurrent neural network to perform character set extraction on the target medical fusion feature sequence to generate a medical classification character set, wherein the target medical fusion feature sequence is a medical fusion feature sequence of the medical text to be identified; calling a preset probability function to perform probability calculation based on the medical classification character set to generate multiple medical character probabilities; calling a preset time series classification algorithm to perform alignment decoding on the target medical fusion feature sequence in combination with the multiple medical character probabilities to generate a target medical text.

5. A text recognition device based on feature fusion, characterized in that: The feature fusion-based text recognition device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the feature fusion-based text recognition device executes the feature fusion-based text recognition method as described in any one of claims 1-3.

6. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the text recognition method based on feature fusion as described in any one of claims 1 to 3 is implemented.

Images