Medical image recognition method, device, equipment and storage medium

Abstract

The present invention relates to the field of artificial intelligence, and discloses a medical image recognition method, device, equipment and storage medium, which are applied in the field of smart medical care to improve the recognition efficiency of medical images. The method of the present invention includes: receiving an initial image sent by a user terminal; classifying the initial image to determine the image type of the initial image; when the initial image is a medical feature image, performing image segmentation on the medical feature image through a preset dual attention network model to obtain a medical foreground image; correcting the medical foreground image to obtain a corrected medical foreground image; performing text detection on the corrected medical foreground image to obtain a target text; performing text recognition on the target text to obtain an identified field; performing correction check on the indicator field in the identified field to obtain a corrected field; classifying the corrected field to obtain a classification result, and sending the classification result to the target terminal.

Description

Technical Field

[0001] The present invention relates to the field of medical image recognition, and in particular to a medical image recognition method, device, equipment and storage medium. Background Art

[0002] For a long time, the uneven distribution of medical resources in my country, the difficulty of seeing a doctor, the high cost of seeing a doctor and other problems have been plaguing the people. With the promotion of the national medical reform process, the mobile medical market has emerged. Thanks to the popularization of the concept of knowledge payment and the liberalization of medical e-commerce policies, the market size of the mobile medical industry has grown rapidly, and the number of users' online consultations has also increased sharply.

[0003] However, when users consult online, they usually need to take photos and upload historical clinical examination reports for online doctors to diagnose and treat. A large number of unclassified photos will greatly reduce the diagnosis and treatment efficiency of online doctors and cause diagnostic errors.

[0004] The existing solution uses a recognition model for a single image of a determined category, without considering the situation where users upload a large number of different test reports. The existing recognition model cannot accurately recognize the text data in the report. Summary of the invention

[0005] The present invention provides a medical image recognition method, device, equipment and storage medium for recognizing different categories of test report images, thereby improving the recognition efficiency of medical images and further improving the recognition efficiency of text information in medical images.

[0006] A first aspect of an embodiment of the present invention provides a medical image recognition method, comprising: receiving an initial image sent by a user terminal, the initial image comprising different types of report images, wherein the image specifications of different types of reports are different; classifying the initial image by a preset fine-grained classification model to determine the image type of the initial image, wherein the image type comprises a medical feature image and a non-medical feature image; when the initial image is a medical feature image, performing image segmentation on the medical feature image by a preset dual attention network model to obtain a medical foreground image; correcting the medical foreground image by perspective transformation to obtain a corrected medical foreground image; calling a preset differential binarization network model to perform text detection on the corrected medical foreground image to obtain a target text; calling a preset recognition model to perform text recognition on the target text to obtain an identified field; performing correction check on an indicator field in the identified field according to a preset dictionary set to obtain a corrected field; classifying the corrected field by a preset TextCNN model to obtain a classification result, and sending the classification result to a target terminal, wherein the classification result is used to indicate the risk of the indicator field.

[0007] Optionally, in a first implementation method of the first aspect of the embodiment of the present invention, the initial image is classified by a preset fine-grained classification model to determine the image type of the initial image, and the image type includes medical feature images and non-medical feature images, including: calling a preset discriminant filtering learning network model to identify the initial image to obtain a recognition result, and the recognition result is a historical medical imaging diagnosis report, a medical test report, a drug packaging or other; determining the image type of the initial image according to the recognition result, and the image type includes medical feature images and non-medical feature images.

[0008] Optionally, in a second implementation method of the first aspect of the embodiment of the present invention, the calling of a preset discriminant filter learning network model to identify the initial image to obtain a recognition result, wherein the recognition result is a historical medical imaging diagnosis report, a medical test report, a drug packaging or others, includes: inputting the initial image into a preset discriminant filter learning network DFL-CNN model; performing convolution and pooling operations through the convolution pooling layer of the dense convolution network DenseNet in the DFL-CNN model to obtain an initial feature map; inputting the initial feature map into the first dense block of DenseNet for dense connection operation to obtain a first intermediate feature map; inputting the first intermediate feature map into Dens The first transition layer of eNet performs a convolution pooling operation to obtain a first pooling feature map; the first pooling feature map is input into the second dense block of DenseNet for dense connection operation to obtain a second intermediate feature map; the second intermediate feature map is input into the second transition layer of DenseNet for convolution pooling operation to generate a second pooling feature map, and the second pooling feature map is continuously subjected to dense connection operation and convolution pooling operation until the final feature map obtained is input into the fully connected layer and the discriminant filter learning DFL network to generate a target feature map; the recognition result of the initial image is generated according to the target feature map, and the recognition result is a historical medical imaging diagnosis report, a medical test report, a drug packaging or other.

[0009] Optionally, in a third implementation method of the first aspect of the embodiment of the present invention, when the initial image is a medical feature image, the medical feature image is segmented by a preset dual attention network model to obtain a medical foreground image, including: when the initial image is a medical feature image, the medical feature image is identified by a preset residual network to generate an original feature matrix; the original feature matrix is ​​input into a position attention model in a preset dual attention network model to generate a spatial feature matrix; the original feature matrix is ​​input into a channel attention model in a preset dual attention network model to generate a channel feature matrix; the spatial feature matrix and the channel feature matrix are fused to obtain a fused feature matrix; and a medical foreground image is generated according to the fused feature matrix.

[0010] Optionally, in a fourth implementation method of the first aspect of the embodiment of the present invention, the inputting the original feature matrix into the position attention model in a preset dual attention network model to generate a spatial feature matrix includes: calling the position attention model in the preset dual attention model to identify the original feature matrix to obtain a spatial attention matrix, and the spatial attention matrix is ​​used to indicate the spatial dependency relationship between any two pixels in the medical feature image; multiplying the spatial attention matrix with the original feature matrix to generate a spatial product matrix; performing weighted summation of the spatial product matrix and the original feature matrix to generate a spatial feature matrix, and the spatial feature matrix is ​​used to indicate long-range context information.

[0011] Optionally, in a fifth implementation manner of the first aspect of the embodiment of the present invention, the inputting the original feature matrix into the channel attention model in the preset dual attention network model to generate a channel feature matrix includes: calling the channel attention model in the preset dual attention model to identify the original feature matrix to obtain a channel attention matrix, and the channel attention matrix is ​​used to indicate the global dependency relationship between any channel graphs; multiplying the channel attention matrix with the original feature matrix to generate a channel product matrix; performing weighted summation of the channel product matrix and the original feature matrix to generate a channel feature matrix, and the channel feature matrix is ​​used to indicate long-range dependencies.

[0012] Optionally, in a sixth implementation manner of the first aspect of the embodiments of the present invention, the use of perspective transformation to correct the medical foreground image to obtain a corrected medical foreground image includes: grayscale processing the medical foreground image to obtain a grayscale image; edge detection on the grayscale image to obtain initial edge information; Hough transform on the initial edge information to obtain target edge information of the medical foreground image; calculating the target edge information to obtain four corner point coordinates; and performing perspective transformation according to the four corner point coordinates to obtain a corrected medical foreground image.

[0013] The second aspect of the embodiment of the present invention provides a medical image recognition device, including: an image receiving module, used to receive an initial image sent by a user terminal, the initial image includes different types of report images, wherein the image specifications of different types of reports are different; an image classification module, used to classify the initial image through a preset fine-grained classification model to determine the image type of the initial image, the image type includes a medical feature image and a non-medical feature image; an image segmentation module, used to perform image segmentation on the medical feature image through a preset dual attention network model when the initial image is a medical feature image, to obtain a medical foreground image; an image correction module, used to adopt perspective distortion to obtain a medical foreground image; The medical prospect image is corrected to obtain a corrected medical prospect image; a text detection module is used to call a preset differential binarization network model to perform text detection on the corrected medical prospect image to obtain a target text; a text recognition module is used to call a preset recognition model to perform text recognition on the target text to obtain a recognized field; a text correction module is used to perform correction check on the indicator field in the recognized field according to a preset dictionary set to obtain a corrected field; a text classification module is used to classify the corrected field using a preset TextCNN model to obtain a classification result, and send the classification result to a target terminal, wherein the classification result is used to indicate the risk of the indicator field.

[0014] Optionally, in a first implementation method of the second aspect of the embodiment of the present invention, the image classification module includes: an image recognition unit, used to call a preset discriminant filtering learning network model to recognize the initial image to obtain a recognition result, wherein the recognition result is a historical medical imaging diagnosis report, a medical test report, a drug packaging or others; a type determination unit, used to determine the image type of the initial image according to the recognition result, wherein the image type includes a medical feature image and a non-medical feature image.

[0015] Optionally, in a second implementation of the second aspect of the embodiment of the present invention, the image recognition unit is specifically used to: input the initial image into a preset discriminant filter learning network DFL-CNN model; perform convolution and pooling operations through the convolution and pooling layer of the dense convolution network DenseNet in the DFL-CNN model to obtain an initial feature map; input the initial feature map into the first dense block of DenseNet to perform dense connection operations to obtain a first intermediate feature map; input the first intermediate feature map into the first transition layer of DenseNet to perform convolution and pooling operations to obtain a first pooled feature map; The first pooled feature map is input into the second dense block of DenseNet for dense connection operation to obtain a second intermediate feature map; the second intermediate feature map is input into the second transition layer of DenseNet for convolution pooling operation to generate a second pooled feature map, and the second pooled feature map is continuously subjected to dense connection operation and convolution pooling operation until the final feature map obtained is input into the fully connected layer and the discriminant filter learning DFL network to generate a target feature map; the recognition result of the initial image is generated according to the target feature map, and the recognition result is a historical medical imaging diagnosis report, a medical test report, a drug packaging or others.

[0016] Optionally, in a third implementation method of the second aspect of the embodiment of the present invention, the image segmentation module includes: a feature recognition unit, used to identify the medical feature image through a preset residual network when the initial image is a medical feature image, and generate an original feature matrix; a first generation unit, used to input the original feature matrix into a position attention model in a preset dual attention network model to generate a spatial feature matrix; a second generation unit, used to input the original feature matrix into a channel attention model in a preset dual attention network model to generate a channel feature matrix; a fusion unit, used to fuse the spatial feature matrix and the channel feature matrix to obtain a fused feature matrix; and a third generation unit, used to generate a medical foreground image based on the fused feature matrix.

[0017] Optionally, in a fourth implementation method of the second aspect of the embodiment of the present invention, the first generating unit is specifically used for: if the position attention model in the preset dual attention model is called to identify the original feature matrix, a spatial attention matrix is ​​obtained, and the spatial attention matrix is ​​used to indicate the spatial dependency relationship between any two pixels in the medical feature image; the spatial attention matrix is ​​multiplied by the original feature matrix to generate a spatial product matrix; the spatial product matrix and the original feature matrix are weightedly summed to generate a spatial feature matrix, and the spatial feature matrix is ​​used to indicate long-range context information.

[0018] Optionally, in a fifth implementation manner of the second aspect of the embodiment of the present invention, the second generation unit is specifically used to: call the channel attention model in the preset dual attention model to identify the original feature matrix to obtain a channel attention matrix, and the channel attention matrix is ​​used to indicate the global dependency relationship between any channel graphs; multiply the channel attention matrix with the original feature matrix to generate a channel product matrix; perform weighted summation on the channel product matrix and the original feature matrix to generate a channel feature matrix, and the channel feature matrix is ​​used to indicate long-range dependencies.

[0019] Optionally, in a sixth implementation manner of the second aspect of the embodiment of the present invention, the image correction module is specifically used to: grayscale the medical foreground image to obtain a grayscale image; perform edge detection on the grayscale image to obtain initial edge information; perform Hough transform on the initial edge information to obtain target edge information of the medical foreground image; calculate the target edge information to obtain four corner point coordinates; perform perspective transformation according to the four corner point coordinates to obtain a corrected medical foreground image.

[0020] A third aspect of an embodiment of the present invention provides a medical image recognition device, a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected via lines; the at least one processor calls the instructions in the memory so that the medical image recognition device executes the above-mentioned medical image recognition method.

[0021] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores instructions. When the instructions are executed by a processor, the steps of the medical image recognition method described in any of the above embodiments are implemented.

[0022] In the technical solution provided by the embodiment of the present invention, an initial image sent by a user terminal is received, the initial image includes different types of report images, wherein the image specifications of different types of report images are different; the initial image is classified by a preset fine-grained classification model to determine the image type of the initial image, and the image type includes a medical feature image and a non-medical feature image; when the initial image is a medical feature image, the medical feature image is segmented by a preset dual attention network model to obtain a medical foreground image; the medical foreground image is corrected by perspective transformation to obtain a corrected medical foreground image; a preset differential binarization network model is called to perform text detection on the corrected medical foreground image to obtain a target text; a preset recognition model is called to perform text recognition on the target text to obtain an identified field; a correction check is performed on the indicator field in the identified field according to a preset dictionary set to obtain a corrected field; a preset TextCNN model is used to classify the corrected field to obtain a classification result, and the classification result is sent to a target terminal, and the classification result is used to indicate the risk of the indicator field. The embodiment of the present invention distinguishes different categories of test report images through an improved fine-grained classification model, and uniformly processes test report images of different types of formats, thereby achieving accurate recognition of different categories of test report images, improving the recognition efficiency of medical images, and further improving the recognition efficiency of text information in medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of an embodiment of a method for recognizing a medical image in an embodiment of the present invention;

[0024] Figure 2 is a schematic diagram of another embodiment of the medical image recognition method in an embodiment of the present invention;

[0025] Figure 3 A schematic diagram of an embodiment of a medical image recognition device in an embodiment of the present invention;

[0026] Figure 4 is a schematic diagram of another embodiment of the medical image recognition device in an embodiment of the present invention;

[0027] Figure 5 It is a schematic diagram of an embodiment of a medical image recognition device in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention provides a medical image recognition method, device, equipment and storage medium for recognizing different categories of test report images, thereby improving the recognition efficiency of medical images and further improving the recognition efficiency of text information in medical images.

[0029] In order to enable those skilled in the art to better understand the solutions of the present invention, the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0030] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] See also Figure 1 , a flowchart of a medical image recognition method provided by an embodiment of the present invention specifically includes:

[0032] 101. Receive an initial image sent by a user terminal, where the initial image includes different types of report images, wherein different types of report images have different specifications.

[0033] The server receives an initial image sent by a user terminal, where the initial image includes different types of report images, wherein the image specifications of different types of reports are different.

[0034] It should be noted that the initial image uploaded by the user may include historical medical imaging diagnosis reports, medical examination reports, and drug packaging. For example, the historical medical imaging diagnosis report may be a B-ultrasound report, the medical examination report may be a routine blood test report, and the drug packaging may be an amoxicillin packaging box. The initial image may also be other medical-related images. The initial image may also be other images unrelated to medicine, for example, the user uploads an erroneous image (payment slip, invoice, etc.), a non-drug packaging image (oxygen tube packaging, gauze packaging, etc.), and the specifics are not limited here.

[0035] It is understandable that the execution subject of the present invention may be a medical image recognition device or a server, which is not limited here. The embodiment of the present invention is described by taking the server as the execution subject as an example.

[0036] 102. Classify the initial image through a preset fine-grained classification model to determine the image type of the initial image, where the image type includes a medical feature image and a non-medical feature image.

[0037] Specifically, the server calls a preset discriminant filter learning network model to recognize the initial image and obtains a recognition result, which is a historical medical imaging diagnosis report, a medical test report, a drug package or other; the server determines the image type of the initial image based on the recognition result, and the image type includes a medical feature image and a non-medical feature image. Among them, the discriminant filter learning network model is the above-mentioned fine-grained classification model.

[0038] Among them, historical medical imaging diagnosis reports include radiological diagnosis reports, ultrasound diagnosis reports, nuclear medicine diagnosis reports and interventional radiological diagnosis reports; among them, radiological diagnosis reports can include X-ray films, computed tomography (CT) images and magnetic resonance (MR) images; ultrasound diagnosis reports can include traditional surface ultrasound, endoscopic ultrasound and vaginal ultrasound; nuclear medicine diagnosis reports include emission computed tomography (ECT) images, positron emission computed tomography / computerized tomography (PET / CT) images, positron emission computed tomography / magnetic resonance imaging (PET / MR) images, etc. Medical examination reports can include blood routine test reports, urine routine blood sugar test reports, leucorrhea test reports, semen test reports, human papilloma virus (HPV) test reports, etc. Pharmaceutical packaging can include images of medicine bottles and medicine boxes.

[0039] 103. When the initial image is a medical feature image, the medical feature image is segmented by a preset dual attention network model to obtain a medical foreground image.

[0040] When the initial image is a medical feature image, the server performs image segmentation on the medical feature image through a preset dual attention network model to obtain a medical foreground image.

[0041] It should be noted that this embodiment is constructed using a dual attention network DANet model of a position attention mechanism and a channel attention mechanism, wherein the position attention mechanism uses a self-attention mechanism to capture the spatial dependency between feature maps at any two positions, and aggregates and updates the features of all positions by weighted summation, where the weights are determined by the feature similarity of the corresponding two positions. Among them, the channel attention mechanism uses a self-attention mechanism to capture the channel dependency between any two channel maps, and uses all channel maps for weighted summation to update each channel map. In this embodiment, similar features at any scale are adaptively integrated from a global perspective, and the relationship between space and channels is explicitly considered, so that scene understanding can utilize long-term dependencies.

[0042] 104. Use perspective transformation to correct the medical foreground image to obtain a corrected medical foreground image.

[0043] Specifically, the server grayscales the medical foreground image to obtain a grayscale image; the server performs edge detection on the grayscale image to obtain initial edge information; the server performs Hough transform on the initial edge information to obtain target edge information of the medical foreground image; the server calculates the target edge information to obtain coordinates of four corner points; the server performs perspective transform based on the coordinates of the four corner points to obtain a corrected medical foreground image.

[0044] Among them, given the four pairs of pixel point coordinates (four corner point coordinates) corresponding to the perspective transformation, the perspective transformation matrix can be obtained, and the image can be corrected based on the perspective transformation matrix.

[0045] 105. Call the preset differential binarization network model to perform text detection on the corrected medical foreground image to obtain the target text.

[0046] The server calls a preset differential binarization network model to perform text detection on the corrected medical foreground image to obtain the target text.

[0047] It should be noted that since there are a lot of diagnostic report contents in the imaging report, and the diagnostic report is usually a long text with multiple lines, compared with other models, the differential binarization network DB-Net uses deformation convolution to effectively detect long text and tilted text. In addition, each pixel is adaptively binarized, and the binarization threshold is learned by the network, so that the final output image will be more stable to the threshold.

[0048] 106. Call a preset recognition model to perform text recognition on the target text to obtain recognized fields.

[0049] The server calls the preset recognition model to perform text recognition on the target text and obtains the recognized fields.

[0050] Among them, the preset recognition model is built based on the dense convolutional network DenseNet and Connectionist Temporal Classification (CTC). Compared with the traditional CRNN+Attention or CRNN+CTC, the DenseNet model is lighter than the CRNN model (CRNN contains a RNN structure model with a slower processing speed), and the CTC model has a higher accuracy rate in recognizing long texts than Attention (Attention is suitable for phrase recognition but poor at recognizing long sentences).

[0051] 107. Perform a correction check on the indicator field in the identified field according to the preset dictionary set to obtain a correction field.

[0052] The server performs a correction check on the indicator field in the identified field according to the preset dictionary set to obtain a correction field. The preset dictionary set is constructed based on clinical test and inspection knowledge to correct the inspection indicator field, such as detailed name, unit, reference value, etc.

[0053] 108. Use the preset TextCNN model to classify the correction field, obtain the classification result, and send the classification result to the target terminal. The classification result is used to indicate the risk of the indicator field.

[0054] The server uses a preset TextCNN model to classify the correction field, obtains the classification result, and sends the classification result to the target terminal. The classification result is used to indicate the risk of the indicator field.

[0055] It should be noted that the preset TextCNN model is pre-trained. The training process is as follows: first, the Bert pre-trained model provided by Google is used to vectorize the training data. The word vector size of each field value or diagnostic sentence (recognition result) is 512*768 (the maximum length of the field value or diagnostic sentence is 512 characters, and each character is represented by a 1*768-dimensional vector), and the word vector of the training data is saved. Secondly, the word vector of the training data is used as the input of the TextCNN model for fine-tuning, and the training model results are saved. Finally, the trained model can be used for text category prediction. Among them, the Bert model consists of a 12-layer transformer encoder, which is pre-trained by a large Chinese text dataset to obtain all the parameters in the Bert model and save them.

[0056] The embodiment of the present invention distinguishes different types of test report images through an improved fine-grained classification model, uniformly processes test report images of different types, realizes accurate recognition of test report images of different types, improves the recognition efficiency of medical images, and further improves the recognition efficiency of text information in medical images. In addition, this solution can be applied to the field of smart medical care, thereby promoting the construction of smart cities.

[0057] See also Figure 2 , another flow chart of the medical image recognition method provided by an embodiment of the present invention, specifically comprising:

[0058] 201. Receive an initial image sent by a user terminal, where the initial image includes different types of report images, wherein different types of report images have different specifications.

[0059] The server receives an initial image sent by a user terminal, where the initial image includes different types of report images, wherein the image specifications of different types of reports are different.

[0060] It should be noted that the initial image uploaded by the user may include historical medical imaging diagnosis reports, medical examination reports, and drug packaging. For example, the historical medical imaging diagnosis report may be a B-ultrasound report, the medical examination report may be a routine blood test report, and the drug packaging may be an amoxicillin packaging box. The initial image may also be other medical-related images. The initial image may also be other images unrelated to medicine, for example, the user uploads an erroneous image (payment slip, invoice, etc.), a non-drug packaging image (oxygen tube packaging, gauze packaging, etc.), and the specifics are not limited here.

[0061] It is understandable that the execution subject of the present invention may be a medical image recognition device or a server, which is not limited here. The embodiment of the present invention is described by taking the server as the execution subject as an example.

[0062] 202. Call a preset discriminant filter learning network model to recognize the initial image and obtain a recognition result, where the recognition result is a historical medical imaging diagnosis report, a medical test report, a drug package or others.

[0063] Specifically, the server inputs the initial image into a preset discriminant filter learning network DFL-CNN model; the server performs convolution and pooling operations through the convolution pooling layer of the dense convolution network DenseNet in the DFL-CNN model to obtain an initial feature map; the server inputs the initial feature map into the first dense block of DenseNet for dense connection operation to obtain a first intermediate feature map; the server inputs the first intermediate feature map into the first transition layer of DenseNet for convolution pooling operation to obtain a first pooling feature map; the server inputs the first pooling feature map into the second dense block of DenseNet for dense connection operation to obtain a second intermediate feature map; the server inputs the second intermediate feature map into the second transition layer of DenseNet for convolution pooling operation to generate a second pooling feature map, and continues to perform dense connection operation and convolution pooling operation on the second pooling feature map until the final feature map obtained is input into the fully connected layer and the discriminant filter learning DFL network to generate a target feature map; the server generates a recognition result of the initial image according to the target feature map, and the recognition result is a historical medical imaging diagnosis report, a medical test report, a drug packaging or other.

[0064] 203. Determine the image type of the initial image according to the recognition result, where the image type includes a medical feature image and a non-medical feature image.

[0065] The server determines the image type of the initial image according to the recognition result, and the image type includes a medical feature image and a non-medical feature image. When the recognition result is a historical medical imaging diagnosis report, a medical test report, or a drug package, the corresponding initial image is classified as a medical feature image; when the recognition result is other, the corresponding initial image is classified as a non-medical feature image.

[0066] Among them, historical medical imaging diagnosis reports include radiological diagnosis reports, ultrasound diagnosis reports, nuclear medicine diagnosis reports and interventional radiological diagnosis reports; among them, radiological diagnosis reports can include X-ray films, CT images and MR images, etc.; ultrasound diagnosis reports can include traditional surface ultrasound, ultrasound endoscopy and vaginal ultrasound, etc.; nuclear medicine diagnosis reports include ECT images, PET / CT images, PET / MR images, etc. Medical examination reports can include blood routine test reports, urine routine blood sugar test reports, leucorrhea test reports, semen test reports, HPV test reports, etc. Pharmaceutical packaging can include images of medicine bottles and medicine boxes, etc.

[0067] 204. When the initial image is a medical feature image, the medical feature image is segmented by a preset dual attention network model to obtain a medical foreground image.

[0068] Specifically, when the initial image is a medical feature image, the server recognizes the medical feature image through a preset residual network to generate an original feature matrix; the server inputs the original feature matrix into the position attention model in the preset dual attention network model to generate a spatial feature matrix; the server inputs the original feature matrix into the channel attention model in the preset dual attention network model to generate a channel feature matrix; the server fuses the spatial feature matrix and the channel feature matrix to obtain a fused feature matrix; the server generates a medical prospect image based on the fused feature matrix.

[0069] It should be noted that this embodiment is constructed using a dual attention network DANet model of a position attention mechanism and a channel attention mechanism, wherein the position attention mechanism uses a self-attention mechanism to capture the spatial dependency between feature maps at any two positions, and aggregates and updates the features of all positions by weighted summation, where the weights are determined by the feature similarity of the corresponding two positions. Among them, the channel attention mechanism uses a self-attention mechanism to capture the channel dependency between any two channel maps, and uses all channel maps for weighted summation to update each channel map. In this embodiment, similar features at any scale are adaptively integrated from a global perspective, and the relationship between space and channels is explicitly considered, so that scene understanding can utilize long-term dependencies.

[0070] Optionally, the server inputs the original feature matrix into the position attention model in the preset dual attention network model to generate a spatial feature matrix, specifically including:

[0071] The server calls the position attention model in the preset dual attention model to identify the original feature matrix and obtains the spatial attention matrix, which is used to indicate the spatial dependency between any two pixels in the medical feature image; the server multiplies the spatial attention matrix with the original feature matrix to generate a spatial product matrix; the server performs a weighted summation of the spatial product matrix and the original feature matrix to generate a spatial feature matrix, which is used to indicate long-range context information.

[0072] Optionally, the server inputs the original feature matrix into a channel attention model in a preset dual attention network model to generate a channel feature matrix, specifically including:

[0073] The server calls the channel attention model in the preset dual attention model to identify the original feature matrix and obtains the channel attention matrix, which is used to indicate the global dependency between any channel graphs; the server multiplies the channel attention matrix with the original feature matrix to generate a channel product matrix; the server performs a weighted summation of the channel product matrix and the original feature matrix to generate a channel feature matrix, which is used to indicate long-range dependencies.

[0074] 205. Correct the medical foreground image using perspective transformation to obtain a corrected medical foreground image.

[0075] Specifically, the server grayscales the medical foreground image to obtain a grayscale image; the server performs edge detection on the grayscale image to obtain initial edge information; the server performs Hough transform on the initial edge information to obtain target edge information of the medical foreground image; the server calculates the target edge information to obtain coordinates of four corner points; the server performs perspective transform based on the coordinates of the four corner points to obtain a corrected medical foreground image.

[0076] Among them, given the four pairs of pixel point coordinates (four corner point coordinates) corresponding to the perspective transformation, the perspective transformation matrix can be obtained, and the image can be corrected based on the perspective transformation matrix.

[0077] 206. Call a preset differential binarization network model to perform text detection on the corrected medical foreground image to obtain the target text.

[0078] The server calls a preset differential binarization network model to perform text detection on the corrected medical foreground image to obtain the target text.

[0079] It should be noted that since there are a lot of diagnostic report contents in the imaging report, and the diagnostic report is usually a long text with multiple lines, compared with other models, the differential binarization network DB-Net uses deformation convolution to effectively detect long text and tilted text. In addition, each pixel is adaptively binarized, and the binarization threshold is learned by the network, so that the final output image will be more stable to the threshold.

[0080] 207. Call a preset recognition model to perform text recognition on the target text to obtain recognized fields.

[0081] The server calls the preset recognition model to perform text recognition on the target text and obtains the recognized fields.

[0082] Among them, the preset recognition model is built based on the dense convolutional network DenseNet and Connectionist Temporal Classification (CTC). Compared with the traditional CRNN+Attention or CRNN+CTC, the DenseNet model is lighter than the CRNN model (CRNN contains a RNN structure model with a slower processing speed), and the CTC model has a higher accuracy rate in recognizing long texts than Attention (Attention is suitable for phrase recognition but poor at recognizing long sentences).

[0083] 208. Perform a correction check on the indicator field in the identified field according to the preset dictionary set to obtain a correction field.

[0084] The server performs a correction check on the indicator field in the identified field according to the preset dictionary set to obtain a correction field. The preset dictionary set is constructed based on clinical test and inspection knowledge to correct the inspection indicator field, such as detailed name, unit, reference value, etc.

[0085] 209. Use the preset TextCNN model to classify the correction field, obtain the classification result, and send the classification result to the target terminal. The classification result is used to indicate the risk of the indicator field.

[0086] The server uses a preset TextCNN model to classify the correction field, obtains the classification result, and sends the classification result to the target terminal. The classification result is used to indicate the risk of the indicator field.

[0087] It should be noted that the preset TextCNN model is pre-trained. The training process is as follows: first, the Bert pre-trained model provided by Google is used to vectorize the training data. The word vector size of each field value or diagnostic sentence (recognition result) is 512*768 (the maximum length of the field value or diagnostic sentence is 512 characters, and each character is represented by a 1*768-dimensional vector), and the word vector of the training data is saved. Secondly, the word vector of the training data is used as the input of the TextCNN model for fine-tuning, and the training model results are saved. Finally, the trained model can be used for text category prediction. Among them, the Bert model consists of a 12-layer transformer encoder, which is pre-trained by a large Chinese text dataset to obtain all the parameters in the Bert model and save them.

[0088] It is understandable that the classification result includes multiple categories, each category is distinguished according to attributes, and the classification result compares and marks the inspection result of the indicator field with the reference value, and the mark is used to remind the doctor whether the indicator is rising, falling or within the normal range, to assist the doctor in diagnosis. In addition, in this embodiment, the target terminal can be a terminal used by the doctor, and the user terminal is a terminal used by the patient.

[0089] The embodiment of the present invention distinguishes different types of test report images through an improved fine-grained classification model, uniformly processes test report images of different types, realizes accurate recognition of test report images of different types, improves the recognition efficiency of medical images, and further improves the recognition efficiency of text information in medical images. In addition, this solution can be applied to the field of smart medical care, thereby promoting the construction of smart cities.

[0090] The above describes the medical image recognition method in the embodiment of the present invention. The following describes the medical image recognition device in the embodiment of the present invention. Figure 3 , an embodiment of the medical image recognition device in the embodiment of the present invention includes:

[0091] The image receiving module 301 is used to receive an initial image sent by a user terminal, wherein the initial image includes different types of report images, wherein the image specifications of different types of report images are different;

[0092] An image classification module 302 is used to classify the initial image by using a preset fine-grained classification model to determine the image type of the initial image, wherein the image type includes a medical feature image and a non-medical feature image;

[0093] An image segmentation module 303 is used for, when the initial image is a medical feature image, performing image segmentation on the medical feature image by using a preset dual attention network model to obtain a medical foreground image;

[0094] An image correction module 304 is used to correct the medical foreground image by using perspective transformation to obtain a corrected medical foreground image;

[0095] A text detection module 305 is used to call a preset differential binarization network model to perform text detection on the corrected medical prospect image to obtain a target text;

[0096] The text recognition module 306 is used to call a preset recognition model to perform text recognition on the target text to obtain recognized fields;

[0097] A text correction module 307, configured to perform a correction check on the indicator field in the identified field according to a preset dictionary set to obtain a correction field;

[0098] The text classification module 308 is used to classify the correction field using a preset TextCNN model to obtain a classification result, and send the classification result to the target terminal. The classification result is used to indicate the risk of the indicator field.

[0099] The embodiment of the present invention distinguishes different types of test report images through an improved fine-grained classification model, uniformly processes test report images of different types, realizes accurate recognition of test report images of different types, improves the recognition efficiency of medical images, and further improves the recognition efficiency of text information in medical images. In addition, this solution can be applied to the field of smart medical care, thereby promoting the construction of smart cities.

[0100] See also Figure 4 Another embodiment of the medical image recognition device in the embodiment of the present invention includes:

[0101] The image receiving module 301 is used to receive an initial image sent by a user terminal, wherein the initial image includes different types of report images, wherein the image specifications of different types of report images are different;

[0102] An image classification module 302 is used to classify the initial image by using a preset fine-grained classification model to determine the image type of the initial image, wherein the image type includes a medical feature image and a non-medical feature image;

[0103] An image segmentation module 303 is used for, when the initial image is a medical feature image, performing image segmentation on the medical feature image by using a preset dual attention network model to obtain a medical foreground image;

[0104] An image correction module 304 is used to correct the medical foreground image by using perspective transformation to obtain a corrected medical foreground image;

[0105] A text detection module 305 is used to call a preset differential binarization network model to perform text detection on the corrected medical prospect image to obtain a target text;

[0106] The text recognition module 306 is used to call a preset recognition model to perform text recognition on the target text to obtain recognized fields;

[0107] A text correction module 307, configured to perform a correction check on the indicator field in the identified field according to a preset dictionary set to obtain a correction field;

[0108] The text classification module 308 is used to classify the correction field using a preset TextCNN model to obtain a classification result, and send the classification result to the target terminal. The classification result is used to indicate the risk of the indicator field.

[0109] Optionally, the image classification module 302 includes:

[0110] The image recognition unit 3021 is used to call a preset discriminant filter learning network model to recognize the initial image and obtain a recognition result, wherein the recognition result is a historical medical imaging diagnosis report, a medical test report, a drug package or others;

[0111] The type determination unit 3022 is used to determine the image type of the initial image according to the recognition result, and the image type includes a medical feature image and a non-medical feature image.

[0112] Optionally, the image recognition unit 3021 is further used for:

[0113] The initial image is input into a preset discriminant filter learning network DFL-CNN model; convolution and pooling operations are performed through the convolution pooling layer of the dense convolution network DenseNet in the DFL-CNN model to obtain an initial feature map; the initial feature map is input into the first dense block of DenseNet for dense connection operation to obtain a first intermediate feature map; the first intermediate feature map is input into the first transition layer of DenseNet for convolution pooling operation to obtain a first pooling feature map; the first pooling feature map is input into the second dense block of DenseNet for dense connection operation to obtain a second intermediate feature map; the second intermediate feature map is input into the second transition layer of DenseNet for convolution pooling operation to generate a second pooling feature map, and the second pooling feature map is continuously subjected to dense connection operation and convolution pooling operation until the final feature map obtained is input into the fully connected layer and the discriminant filter learning DFL network to generate a target feature map; the recognition result of the initial image is generated according to the target feature map, and the recognition result is a historical medical imaging diagnosis report, a medical test report, a drug package or other.

[0114] Optionally, the image segmentation module 303 includes:

[0115] A feature recognition unit 3031 is used for, when the initial image is a medical feature image, recognizing the medical feature image through a preset residual network to generate an original feature matrix;

[0116] A first generating unit 3032 is used to input the original feature matrix into a position attention model in a preset dual attention network model to generate a spatial feature matrix;

[0117] The second generating unit 3033 is used to input the original feature matrix into the channel attention model in the preset dual attention network model to generate a channel feature matrix;

[0118] A fusion unit 3034 is used to fuse the spatial feature matrix and the channel feature matrix to obtain a fused feature matrix;

[0119] The third generating unit 3035 is used to generate a medical foreground image according to the fused feature matrix.

[0120] Optionally, the first generating unit 3032 is specifically configured to:

[0121] If the position attention model in the preset dual attention model is called to identify the original feature matrix, a spatial attention matrix is ​​obtained, and the spatial attention matrix is ​​used to indicate the spatial dependency relationship between any two pixels in the medical feature image; the spatial attention matrix is ​​multiplied by the original feature matrix to generate a spatial product matrix; the spatial product matrix and the original feature matrix are weightedly summed to generate a spatial feature matrix, and the spatial feature matrix is ​​used to indicate long-range context information.

[0122] Optionally, the second generating unit 3033 is specifically used for:

[0123] The channel attention model in the preset dual attention model is called to identify the original feature matrix to obtain a channel attention matrix, where the channel attention matrix is ​​used to indicate the global dependency between any channel graphs; the channel attention matrix is ​​multiplied by the original feature matrix to generate a channel product matrix; the channel product matrix and the original feature matrix are weightedly summed to generate a channel feature matrix, where the channel feature matrix is ​​used to indicate the long-range dependency.

[0124] Optionally, the image correction module 304 is specifically used for:

[0125] The medical foreground image is gray-scaled to obtain a gray-scale image; edge detection is performed on the gray-scale image to obtain initial edge information; Hough transform is performed on the initial edge information to obtain target edge information of the medical foreground image; the target edge information is calculated to obtain four corner point coordinates; perspective transformation is performed according to the four corner point coordinates to obtain a corrected medical foreground image.

[0126] The embodiment of the present invention distinguishes different types of test report images through an improved fine-grained classification model, uniformly processes test report images of different types, realizes accurate recognition of test report images of different types, improves the recognition efficiency of medical images, and further improves the recognition efficiency of text information in medical images. In addition, this solution can be applied to the field of smart medical care, thereby promoting the construction of smart cities.

[0127] above Figure 3 to Figure 4 The medical image recognition device in the embodiment of the present invention is described in detail from the perspective of modular functional entities, and the medical image recognition device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0128] Figure 5: is a schematic diagram of the structure of a medical image recognition device provided by an embodiment of the present invention. The medical image recognition device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 can be temporary storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the medical image recognition device 500. Furthermore, the processor 510 can be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the medical image recognition device 500.

[0129] The medical image recognition device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 5 The structure of the medical image recognition device shown does not constitute a limitation on the medical image recognition device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0130] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the medical image recognition method.

[0131] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0132] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0133] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A medical image recognition method, characterized in that: include: Receiving an initial image sent by a user terminal, wherein the initial image includes different types of report images, wherein the image specifications of different types of report images are different; Classifying the initial image by using a preset fine-grained classification model to determine an image type of the initial image, wherein the image type includes a medical feature image and a non-medical feature image; When the initial image is a medical feature image, performing image segmentation on the medical feature image through a preset dual attention network model to obtain a medical foreground image; Correcting the medical foreground image using perspective transformation to obtain a corrected medical foreground image; Calling a preset differential binarization network model to perform text detection on the corrected medical foreground image to obtain target text; Calling a preset recognition model to perform text recognition on the target text to obtain a recognized field, wherein the preset recognition model is a model constructed based on a dense convolutional network DenseNet and connectionist temporal classification; Performing a correction check on the indicator field in the identified field according to a preset dictionary set to obtain a correction field; The correction field is classified using a preset TextCNN model to obtain a classification result, and the classification result is sent to a target terminal. The classification result is used to indicate the risk of the indicator field.

2. The medical image recognition method according to claim 1, characterized in that: The initial image is classified by a preset fine-grained classification model to determine the image type of the initial image, wherein the image type includes a medical feature image and a non-medical feature image, including: Calling a preset discriminant filter learning network model to recognize the initial image and obtain a recognition result, wherein the recognition result is a historical medical imaging diagnosis report, a medical test report, a drug package or others; The image type of the initial image is determined according to the recognition result, and the image type includes a medical feature image and a non-medical feature image.

3. The medical image recognition method according to claim 2, characterized in that: The calling of a preset discriminant filter learning network model to identify the initial image to obtain a recognition result, wherein the recognition result is a historical medical imaging diagnosis report, a medical test report, a drug package or others, including: Input the initial image into a preset discriminant filter learning network DFL-CNN model; Perform convolution and pooling operations through the convolution and pooling layer of the dense convolutional network DenseNet in the DFL-CNN model to obtain an initial feature map; Input the initial feature map into the first dense block of DenseNet to perform dense connection operation to obtain a first intermediate feature map; Inputting the first intermediate feature map into the first transition layer of DenseNet to perform a convolution pooling operation to obtain a first pooling feature map; Input the first pooled feature map into the second dense block of DenseNet to perform a dense connection operation to obtain a second intermediate feature map; Input the second intermediate feature map to the second transition layer of DenseNet for convolution and pooling operation to generate a second pooling feature map, continue to perform dense connection operation and convolution and pooling operation on the second pooling feature map, until the final feature map obtained is input to the fully connected layer and the discriminant filter learning DFL network to generate a target feature map; A recognition result of the initial image is generated according to the target feature map, and the recognition result is a historical medical imaging diagnosis report, a medical examination report, a drug package or others.

4. The medical image recognition method according to claim 1, characterized in that: When the initial image is a medical feature image, the medical feature image is segmented by a preset dual attention network model to obtain a medical foreground image, including: When the initial image is a medical feature image, the medical feature image is recognized by a preset residual network to generate an original feature matrix; Inputting the original feature matrix into the position attention model in the preset dual attention network model to generate a spatial feature matrix; Input the original feature matrix into the channel attention model in the preset dual attention network model to generate a channel feature matrix; Fusing the spatial feature matrix and the channel feature matrix to obtain a fused feature matrix; A medical prospect image is generated according to the fused feature matrix.

5. The medical image recognition method according to claim 4, characterized in that: The step of inputting the original feature matrix into a position attention model in a preset dual attention network model to generate a spatial feature matrix includes: Calling a position attention model in a preset dual attention model to identify the original feature matrix to obtain a spatial attention matrix, where the spatial attention matrix is ​​used to indicate a spatial dependency relationship between any two pixels in the medical feature image; Multiplying the spatial attention matrix with the original feature matrix to generate a spatial product matrix; The spatial product matrix and the original feature matrix are weightedly summed to generate a spatial feature matrix, where the spatial feature matrix is ​​used to indicate long-range context information.

6. The medical image recognition method according to claim 4, characterized in that: The step of inputting the original feature matrix into a channel attention model in a preset dual attention network model to generate a channel feature matrix includes: Calling a channel attention model in a preset dual attention model to identify the original feature matrix to obtain a channel attention matrix, where the channel attention matrix is ​​used to indicate a global dependency relationship between any channel graphs; Multiplying the channel attention matrix by the original feature matrix to generate a channel product matrix; The channel product matrix and the original feature matrix are weightedly summed to generate a channel feature matrix, where the channel feature matrix is ​​used to indicate long-range dependencies.

7. The medical image recognition method according to any one of claims 1 to 6, characterized in that: The method of correcting the medical foreground image by using perspective transformation to obtain a corrected medical foreground image includes: Gray-scale the medical foreground image to obtain a grayscale image; Performing edge detection on the grayscale image to obtain initial edge information; Performing Hough transform on the initial edge information to obtain target edge information of the medical foreground image; Calculating the target edge information to obtain the coordinates of four corner points; A perspective transformation is performed according to the coordinates of the four corner points to obtain a corrected medical foreground image.

8. A medical image recognition device, characterized in that: include: An image receiving module, used for receiving an initial image sent by a user terminal, wherein the initial image includes different types of report images, wherein the image specifications of different types of report images are different; An image classification module, used to classify the initial image by using a preset fine-grained classification model to determine the image type of the initial image, wherein the image type includes a medical feature image and a non-medical feature image; An image segmentation module, used for performing image segmentation on the medical feature image by using a preset dual attention network model to obtain a medical foreground image when the initial image is a medical feature image; An image correction module, used for correcting the medical foreground image by using perspective transformation to obtain a corrected medical foreground image; A text detection module, used for calling a preset differential binarization network model to perform text detection on the corrected medical prospect image to obtain a target text; A text recognition module, used to call a preset recognition model to perform text recognition on the target text to obtain a recognized field, wherein the preset recognition model is a model constructed based on a dense convolutional network DenseNet and connectionist temporal classification; A text correction module, used for checking the correction of the indicator field in the identified field according to a preset dictionary set to obtain a correction field; The text classification module is used to classify the correction field using a preset TextCNN model to obtain a classification result, and send the classification result to a target terminal, wherein the classification result is used to indicate the risk of the indicator field.

9. A medical image recognition device, characterized in that: The medical image recognition device comprises: a memory and at least one processor, the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor calls the instructions in the memory so that the medical image recognition device executes the medical image recognition method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed by a processor, implement the medical image recognition method as described in any one of claims 1 to 7.

Images