Medical image recognition model training method, medical image recognition method and equipment

Abstract

The invention provides a medical image recognition method. The method comprises the following steps: acquiring a fundus image to be recognized; calling a feature extraction model to perform feature extraction on the to-be-recognized fundus image to obtain a fundus feature map; inputting the fundus feature map into an attention mechanism model to obtain an attention weighted feature map; and calling a classification model to classify the attention weighted feature map to obtain a target disease label of the fundus image to be identified. The invention further provides a medical image recognition model training method and device. By using the method, the hidden biomarkers in the eye fundus image can be accurately detected, so that the diagnosis accuracy and timeliness of eye diseases can be improved.

Description

Technical Field

[0001] The present application belongs to the field of biomedicine and relates to image processing technology, and in particular to a medical image recognition model training method, a medical image recognition method and a device. Background Art

[0002] In the field of medical technology, fundus images have become an important tool to assist doctors in diagnosing diseases. Fundus images contain a wealth of information about the patient's eye health, including the morphology and pathological conditions of structures such as the retina, choroid and sclera. By observing and analyzing these images, doctors can diagnose various eye diseases, such as retinal holes, retinal hemorrhage and retinal detachment.

[0003] However, due to the complexity of fundus images and the diversity of lesions, the hidden biomarkers (potential lesions) in fundus images may be blurred and unclear. Since it is difficult to detect these hidden biomarkers from fundus images, the accuracy and timeliness of eye disease diagnosis are affected. Summary of the invention

[0004] In view of the above, it is necessary to provide a medical image recognition model training method, a medical image recognition method and a device that can solve the technical problem that it is difficult to detect hidden biomarkers in fundus images, resulting in poor diagnostic accuracy and timeliness of eye diseases.

[0005] On the one hand, the present application provides a medical image recognition model training method, the method comprising: obtaining multiple training images and preset disease labels corresponding to each training image, wherein each training image is a fundus image, training a medical image recognition network based on the multiple training images and the preset disease labels corresponding to each training image to obtain a medical image recognition model, wherein the medical image recognition network comprises a feature extraction network, an attention mechanism module and a classification network, and the training process of the medical image recognition network comprises: inputting each training image into the feature extraction network to obtain a training feature map of each training image, inputting each training feature map into the attention mechanism module to obtain To the training weighted feature map of each training feature map, call the classification network to classify each training image based on the training weighted feature map to obtain the predicted disease label corresponding to each training image, calculate the loss value according to each predicted disease label and the preset disease label corresponding to each predicted disease label, and adjust the feature extraction network, the attention mechanism module and the classification network according to the loss value to obtain the feature extraction model corresponding to the feature extraction network, the attention mechanism model corresponding to the attention mechanism module and the classification model corresponding to the classification network, and form the medical image recognition model based on the feature extraction model, the attention mechanism model and the classification model.

[0006] In some embodiments of the present application, the attention mechanism module also includes a training perception layer, wherein inputting each training feature map into the attention mechanism module to obtain a training weighted feature map of each training feature map includes: performing pooling processing on each training feature map to obtain a training pooling value, inputting each training pooling value into the training perception layer, and obtaining a training weight vector corresponding to each training feature map based on the weight of each neuron in the training perception layer to each training pooling value, and then generating a training weighted feature map of each training feature map based on each training feature map and the corresponding training weight vector.

[0007] In some embodiments of the present application, the pooling processing is performed on each of the training feature maps to obtain the training pooling value, each of the training pooling values ​​is input into the training perception layer, and according to the weight of each neuron of the training perception layer for each training pooling value, the training weight vector corresponding to each training feature map is obtained, including: performing global average pooling processing on each of the training feature maps to obtain the training global average pooling value, each training global average pooling value is input into the training perception layer, and according to the weight of each neuron of the perception layer for each global average pooling value, the training weight vector of each training feature map is obtained.

[0008] In some embodiments of the present application, the training perception layer also includes a first training perception layer and a second training perception layer, and the training weight vector includes a first training weight vector and a second training weight vector, wherein the pooling processing is performed on each of the training feature maps to obtain the training pooling value, each training pooling value is input into the training perception layer, and the training weight vector corresponding to each training feature map is obtained according to the weight of each neuron in the training perception layer for each training pooling value. The processing also includes: performing global average pooling processing on each of the training feature maps to obtain a training global average value, inputting each training global average value into the first training perception layer, and obtaining the first training weight vector of each training feature map according to the weight of each neuron in the first training perception layer for each training global average value, performing global maximum pooling processing on each of the training feature maps to obtain a training global maximum value, inputting each training global maximum value into the second training perception layer, and obtaining the second training weight vector of each training feature map according to the weight of each neuron in the second training perception layer for each training global maximum value.

[0009] In some embodiments of the present application, generating a training weighted feature map of each training feature map according to each training feature map and the corresponding training weight vector includes: operating each training feature map and the corresponding first training weight vector to obtain a first operation feature map corresponding to each training feature map, operating each training feature map and the corresponding second training weight vector to obtain a second operation feature map corresponding to each training feature map, and forming a training weighted feature map of each training feature map according to the first operation feature map and the second operation feature map corresponding to each training feature map.

[0010] Through the above implementation, the medical image recognition model is trained according to the training images (fundus images) and the preset disease labels corresponding to the training images, so that the medical image recognition network can learn the association between different disease features (for example, hidden biomarkers) in multiple fundus images and the preset disease labels, so that the trained medical image recognition model can comprehensively and accurately detect the diseases in the fundus images to be identified, thereby effectively assisting doctors in diagnosing the disease.

[0011] On the other hand, the present application provides a medical image recognition method, which includes: obtaining a fundus image to be recognized, calling a pre-trained medical image recognition model to recognize the fundus image to be recognized, and obtaining a target disease label of the fundus image to be recognized, the medical image recognition model includes a feature extraction model, an attention mechanism model and a classification model, and the generation of the target disease label includes: calling the feature extraction model to extract features from the fundus image to be recognized to obtain a fundus feature map, inputting the fundus feature map into the attention mechanism model to obtain an attention weighted feature map, calling the classification model to classify the attention weighted feature map, and obtaining the target disease label of the fundus image to be recognized.

[0012] In some embodiments of the present application, the attention mechanism model also includes a perception layer, and the inputting of the fundus feature map into the attention mechanism model to obtain the attention weighted feature map includes: performing pooling processing on the fundus feature map to obtain pooling values, inputting each pooling value into the perception layer, and obtaining a target weight vector of the fundus feature map based on the weight of each neuron in the perception layer for each pooling value, and then generating the attention weighted feature map based on the fundus feature map and the target weight vector.

[0013] In some embodiments of the present application, the pooling processing is performed on the fundus feature map to obtain the pooling value, each pooling value is input into the perception layer, and the target weight vector of the fundus feature map is obtained according to the weight of each neuron in the perception layer for each pooling value, including: performing global average pooling processing on the fundus feature map to obtain the global average pooling value, inputting each global average pooling value into the perception layer, and obtaining the target weight vector according to the weight of each neuron in the perception layer for each global average pooling value.

[0014] In some embodiments of the present application, the perception layer also includes a first perception layer and a second perception layer, the target weight vector includes a first target weight vector and a second target weight vector, the pooling processing is performed on the fundus feature map to obtain the pooling value, each pooling value is input into the perception layer, and the target weight vector of the fundus feature map is obtained according to the weight of each neuron in the perception layer for each pooling value. It also includes: performing global average pooling processing on the fundus feature map to obtain a global average value, inputting each global average value into the first perception layer, and obtaining the first target weight vector according to the weight of each neuron in the first perception layer for each global average value, performing global maximum pooling processing on the fundus feature map to obtain a global maximum value, inputting each global maximum value into the second perception layer, and obtaining the second target weight vector according to the weight of each neuron in the second perception layer for each global maximum value.

[0015] In some embodiments of the present application, generating the attention weighted feature map according to the fundus feature map and the target weight vector includes: operating the fundus feature map and the first target weight vector to obtain a first attention feature map, operating the fundus feature map and the second target weight vector to obtain a second attention feature map, and generating the attention weighted feature map according to the first attention feature map and the second attention feature map.

[0016] Through the above implementation, since the medical image recognition model is trained through training images (fundus images) and preset disease labels corresponding to the training images, the medical image recognition model learns the association between different disease features (for example, hidden biomarkers) in multiple fundus images and the preset disease labels. Therefore, the medical image recognition model can comprehensively and accurately detect the diseases in the fundus images to be identified, thereby effectively assisting doctors in diagnosing the disease.

[0017] On the other hand, the present application provides an electronic device, comprising: a memory storing at least one instruction, and a processor, obtaining the instructions stored in the memory to implement the medical image recognition model training method, or to implement the medical image recognition method. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application.

[0019] Figure 2 It is a flowchart of a medical image recognition method provided in one embodiment of the present application.

[0020] Figure 3 It is a schematic diagram of a fundus image to be identified provided in an embodiment of the present application.

[0021] Figure 4 This is a flowchart of generating a target weight vector provided by an embodiment of the present application.

[0022] Figure 5 This is a flowchart for generating an attention-weighted feature map provided by an embodiment of the present application.

[0023] Figure 6 It is a flowchart of a medical image recognition model training method provided in one embodiment of the present application.

[0024] Figure 7 This is a flowchart of generating a training weight vector provided by an embodiment of the present application.

[0025] Figure 8 This is a flowchart of generating a training weighted feature map provided by an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0028] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0029] In the field of medical technology, fundus images have become an important tool to assist doctors in diagnosing diseases. Fundus images contain a wealth of information about the patient's eye health, including the morphology and pathological conditions of structures such as the retina, choroid and sclera. By observing and analyzing these images, doctors can diagnose various eye diseases, such as retinal holes, retinal hemorrhage and retinal detachment.

[0030] However, due to the complexity of fundus images and the diversity of lesions, the hidden biomarkers (potential lesions) in fundus images may be blurred and unclear. Since it is difficult to detect these hidden biomarkers from fundus images, the accuracy and timeliness of eye disease diagnosis are affected.

[0031] In order to solve the above technical problems, the present application provides a medical image recognition model training method, a medical image recognition method and a device, which can accurately detect hidden biomarkers in fundus images, thereby improving the accuracy and timeliness of diagnosis of eye diseases. The virtual human body model driving method provided in the embodiment of the present application can be applied to one or more electronic devices.

[0032] like Figure 1 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 10 can be a mobile phone, a tablet computer, a laptop computer, or other electronic device. The present embodiment of the present application does not impose any restrictions on the specific type of the electronic device.

[0033] like Figure 1 As shown, the electronic device 10 may include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104 and a bus 105. The processor 103 is coupled to the communication module 101, the memory 102 and the input / output interface 104 through the bus 105.

[0034] The communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as universal serial bus (USB), controller area network bus (CAN, Controller Area Network). The wireless communication module may provide one or more wireless communication solutions such as wireless fidelity (Wi-Fi), Bluetooth (BT), mobile communication network, frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0035] The memory 102 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 103, and can be used to store executable programs (such as machine instructions) of other running programs, and can also be used to store user and application data. The random access memory may include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0036] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110. The non-volatile memory may include a disk storage device and a flash memory.

[0037] The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include multiple instructions. When the multiple instructions are executed by the processor 103, the medical image recognition model training method and the medical image recognition method executed on the electronic device 10 can be implemented.

[0038] In other embodiments, Figure 1The electronic device 10 shown also includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10 .

[0039] The processor 103 may include one or more processing units, for example, the processor 103 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), and / or a neural network processor (neural-network processing unit, NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0040] The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute a computer program stored in the memory 102 to implement the above-mentioned medical image recognition model training method and medical image recognition method.

[0041] The input / output interface 104 is used to provide a channel for user input or output. For example, the input / output interface 104 can be used to connect various input and output devices, such as a mouse, keyboard, touch device, display screen, etc., so that the user can enter information or visualize information.

[0042] The bus 105 is at least used to provide a channel for mutual communication among the communication module 101 , the memory 102 , the processor 103 , and the input / output interface 104 in the electronic device 10 .

[0043] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0044] like Figure 2 FIG. 1 is a flowchart of a medical image recognition method provided by an embodiment of the present application. According to different requirements, the order of each step in the flowchart can be adjusted according to the actual detection requirements, and some steps can be omitted. The execution subject of the method is an electronic device, such as Figure 1 An electronic device 1 is shown.

[0045] S11, obtaining a fundus image to be identified.

[0046] In at least one embodiment of the present application, the fundus image to be identified refers to an image that needs to identify and detect eye diseases, and the fundus image to be identified can be obtained by photographing the fundus of the eye with a fundus camera. Figure 3 FIG. 1 is a schematic diagram of a fundus image to be identified provided by an embodiment of the present application. Figure 3 It can be seen that the eyeball, retina and blood vessels can be seen from the fundus image to be identified.

[0047] In some embodiments of the present application, after acquiring the fundus image to be identified, the electronic device may call a pre-trained medical image recognition model to identify the fundus image to be identified, and obtain a target disease label of the fundus image to be identified, wherein the medical image recognition model includes a feature extraction model, an attention mechanism model, and a classification model. The target disease label refers to the disease (lesion) category in the fundus image to be identified, and the target disease label may be one or more. For example, the target disease label may be one or more of intraretinal effusion, subretinal effusion, retinal pigment cell detachment, and macular wrinkles.

[0048] S12, calling a feature extraction model to extract features from the fundus image to be identified, and obtaining a fundus feature map.

[0049] In at least one embodiment of the present application, the feature extraction model is a feature extraction module in a machine learning model. If the machine learning model is a CNN (Convolutional Neural Network) model, the CNN model may include, but is not limited to: LeNet model, AlexNet model, VGG model, GoogLeNet model, ResNet model, DenseNet model and Inception model, etc., and the present application does not impose any restrictions on this.

[0050] In some embodiments of the present application, the machine learning model includes a convolutional network layer, a residual block and a global average pooling layer, etc., wherein the convolutional network layer includes a series of convolution kernels (filters), the residual block can be multiple, and multiple residual blocks are repeatedly stacked, each residual block includes a batch normalization layer (BatchNormalization, BN), an activation function (ReLu) and multiple convolutional layers, and each convolutional layer in each residual block is connected to a batch normalization layer and an activation function.

[0051] In this embodiment, the electronic device calls the feature extraction model to perform feature extraction on the fundus image to be identified, and obtains the fundus feature map, including: calling the convolutional network layer to perform first feature extraction on the fundus image to be identified, to obtain the first feature map, performing feature fusion operation on the first feature map through multiple residual blocks, to obtain the target fused feature map, and inputting the fused feature map into the global average pooling layer to obtain the fundus feature map. Among them, the process of performing the feature fusion operation on the first feature map to obtain the target fused feature map includes: calling a residual block to perform second feature extraction on the first feature map, to obtain the second feature map, performing feature fusion on the first feature map and the second feature map, to obtain the first fused feature map, calling another residual block to perform third feature extraction on the initial feature map, to obtain the third feature map, performing feature fusion on the initial feature map and the third feature map, to obtain the second fused feature map, repeating the above steps until the feature map output by the last residual block is obtained, and determining the feature map obtained by fusion of the feature map output by the last residual block and the feature map input to the last residual block as the target fused feature map.

[0052] The operation of calling the residual block for feature extraction can refer to the relevant technology. Feature fusion can be performed by adding or subtracting two feature maps or concatenating (concatenate or concat) according to the channel dimension. This application does not limit the method of feature fusion.

[0053] S13, input the fundus feature map into the attention mechanism model to obtain the attention weighted feature map.

[0054] In at least one embodiment of the present application, the attention mechanism model includes a pooling layer, and the pooling layer can be one or more of the following pooling layers: global average pooling layer (GlobalAveragePooling, GAP), global maximum pooling layer (GlobalAveragePooling, GMP).

[0055] In some embodiments of the present application, the attention mechanism model also includes a pooling layer and a perception layer, wherein the electronic device inputs the fundus feature map into the attention mechanism model to obtain the attention weighted feature map, including: the electronic device performs pooling processing on the fundus feature map to obtain a pooling value, inputs each pooling value into the perception layer, and obtains a target weight vector of the fundus feature map based on the weight of each neuron in the perception layer for each pooling value, and generates an attention weighted feature map based on the fundus feature map and the target weight vector.

[0056] Specifically, if the pooling layer in the attention mechanism model is a global average pooling layer, the pooling value is the global average pooling value, the electronic device performs global average pooling on the fundus feature map to obtain the global average pooling value, and inputs each global average pooling value into the perception layer. Based on the weight of each neuron in the perception layer for each global average pooling value, a target weight vector is obtained, and the fundus feature map is operated with the target weight vector to obtain an attention weighted feature map.

[0057] Among them, the perception layer can be a fully connected layer, and the operation method between the fundus feature map and the target weight vector can be set by itself, and this application does not limit this. For example, the fundus feature map and the target weight vector can be multiplied, added, and subtracted to obtain an attention weighted feature map.

[0058] In this embodiment, important features (such as hidden biomarkers) and background information in the fundus feature map are identified and distinguished, and higher weights are assigned to important features in the fundus feature map, so that the size of the elements in the target weight vector can reflect the importance of each feature in the fundus feature map. By operating the fundus feature map and the target weight vector, an attention weighted feature map is obtained, which can make the important features in the attention weighted feature map more obvious than unimportant features such as background information.

[0059] S14, calling the classification model to classify the attention weighted feature map to obtain the target disease label of the fundus image to be identified.

[0060] In at least one embodiment of the present application, the classification model is a classification module in a machine learning model. If the machine learning model is a CNN (Convolutional Neural Network) model, the CNN model may include, but is not limited to: LeNet model, AlexNet model, VGG model, GoogLeNet model, ResNet model, DenseNet model and Inception model, etc., and the present application does not impose any restrictions on this.

[0061] In some embodiments of the present application, the classification model may include a fully connected layer (FC) and an activation function. The activation function may be a sigmoid function or a softmax function, which is not limited in the present application.

[0062] In other embodiments of the present application, the classification model may also include a fully connected layer and a Dense layer.

[0063] In some embodiments of the present application, the classification model classifies the attention weighted feature map to obtain multiple probability values, each probability value corresponds to a preset disease label, selects a target probability value from the multiple probability values, and determines the preset disease label corresponding to the target probability value as the target disease label.

[0064] Among them, the preset disease label can be one or more of intraretinal fluid, subretinal fluid, retinal pigment cell detachment and macular wrinkles. The method of selecting the target probability value from multiple probability values ​​can be set by yourself, and this application does not limit this. For example, the multiple probability values ​​output by the classification model are intraretinal fluid 0.1, subretinal fluid 0.2, retinal pigment cell detachment 0.01, and macular wrinkles 0.9. The maximum probability value 0.9 can be used as the target probability value, and macular wrinkles can be used as the target disease label.

[0065] In this embodiment, since important features in the attention weighted feature map are more obvious than unimportant features such as background information, classifying the attention weighted feature map through a classification model can improve the accuracy of the target disease label.

[0066] Through the above implementation, since the medical image recognition model is trained through training images (fundus images) and preset disease labels corresponding to the training images, the medical image recognition model learns the association between different disease features (for example, hidden biomarkers) in multiple fundus images and preset disease labels. Therefore, the medical image recognition model can comprehensively and accurately detect the diseases in the fundus images to be identified, thereby effectively assisting doctors in diagnosing the disease.

[0067] In at least one embodiment of the present application, if the pooling layer in the attention mechanism model includes a global average pooling layer and a global maximum pooling layer, the perception layer includes a first perception layer and a second perception layer, and the pooling value includes a global average value and a global maximum value, such as Figure 4 FIG. 1 is a flowchart of generating a target weight vector provided by an embodiment of the present application, which specifically includes the following steps:

[0068] S131, perform global average pooling processing on the fundus feature map to obtain a global average value, input each global average value into the first perception layer, and obtain a first target weight vector based on the weight of each neuron in the first perception layer for each global average value.

[0069] In some embodiments of the present application, the electronic device performs global average pooling processing on the fundus feature map to obtain a first target weight vector, including: the electronic device calculates the global average of all pixel values ​​of each channel in the fundus feature map, and inputs multiple global averages of all channels in the fundus feature map into the first perception layer (perceptron layer), and obtains the first target vector based on the weight of each neuron in the first perception layer for each global average. For example, if the size of the fundus feature map is H*W*C, where H represents the height of the fundus feature map, W represents the width of the fundus feature map, and C represents the number of channels of the fundus feature map. The electronic device calculates the global average of all pixel values ​​of each channel in the fundus feature map to obtain C global averages, and inputs the C global averages into the first perception layer, and obtains a first target weight vector of size 1*C based on the weight of each neuron in the first perception layer for each global average.

[0070] S132, perform global maximum pooling processing on the fundus feature map to obtain a global maximum value, input each global maximum value into the second perception layer, and obtain a second target weight vector based on the weight of each neuron in the second perception layer for each global maximum value.

[0071] In some embodiments of the present application, the electronic device performs global maximum pooling processing on the fundus feature map to obtain the second target weight vector, including: the electronic device determines the global maximum value of all pixel values ​​of each channel in the fundus feature map, and inputs multiple global maxima of all channels in the fundus feature map into the second perception layer, and obtains the second target vector based on the weight of each neuron in the second perception layer for each global maximum value. For example, following the above embodiment, if the size of the fundus feature map is H*W*C, where H represents the height of the fundus feature map, W represents the width of the fundus feature map, and C represents the number of channels of the fundus feature map. The electronic device traverses all pixel values ​​of each channel in the fundus feature map, and determines the global maximum value of all pixel values ​​of each channel to obtain C global maxima, and inputs the C global maxima into the second perception layer, and obtains a second target weight vector of size 1*C based on the weight of each neuron in the second perception layer for each global maximum value.

[0072] In at least one embodiment of the present application, Figure 5 FIG. 1 is a flowchart of generating an attention weighted feature map provided by an embodiment of the present application, which specifically includes the following steps:

[0073] S133, performing operation on the fundus feature map and the first target weight vector to obtain a first attention feature map.

[0074] In some embodiments of the present application, since the first target weight vector includes the global average value (weight) corresponding to each channel in the fundus feature map, the electronic device can perform operations on the pixel values ​​of the pixel points of each channel in the fundus feature map with the corresponding global average value in the first target weight vector by multiplication and addition, etc., to obtain the first channel feature map corresponding to each channel, and add the pixel values ​​of the corresponding pixel points in multiple first channel feature maps to obtain the first attention feature map.

[0075] S134, performing operation on the fundus feature map and the second target weight vector to obtain a second attention feature map.

[0076] In some embodiments of the present application, since the second target weight vector includes the global maximum value corresponding to each channel in the fundus feature map, the electronic device can perform operations on the pixel values ​​of the pixel points of each channel in the fundus feature map with the corresponding global maximum value in the second target weight vector by multiplication and addition, etc., to obtain a second channel feature map corresponding to each channel, and add the pixel values ​​of the corresponding pixel points in multiple second channel feature maps to obtain a second attention feature map.

[0077] S135, generating an attention weighted feature map based on the first attention feature map and the second attention feature map.

[0078] In some embodiments of the present application, the electronic device fuses the first attention feature map and the second attention feature map to obtain an attention weighted feature map. The first attention feature map and the second attention feature map may be fused by concatenating or concat according to the channel dimension to obtain the attention weighted feature map.

[0079] For example, the process of obtaining the first attention feature map or the second attention feature map may also refer to formula (1):

[0080] M c (x,y)=∑ k w c k f k (x,y); (1)

[0081] Among them, M c (x, y) represents the pixel value at the (x, y) position in the first attention feature map or the second attention feature map, w c k represents the kth target weight vector, f k (x,y) represents the pixel value at position (x,y) in the kth attention feature map.

[0082] In some embodiments of the present application, before using the pre-trained medical image recognition model to recognize the fundus image to be recognized, it is necessary to train the medical image recognition network corresponding to the medical image recognition model, wherein the medical image recognition network includes a feature extraction network corresponding to the feature extraction model, an attention mechanism module corresponding to the attention mechanism model, and a classification network corresponding to the classification model. Figure 6 As shown, it is a flow chart of a medical image recognition model training method provided by an embodiment of the present application. According to different requirements, the order of each step in the flow chart can be adjusted according to the actual detection requirements, and some steps can be omitted. The execution subject of the method is an electronic device, such as Figure 1 An electronic device 1 is shown.

[0083] S21, obtaining a plurality of training images and a preset disease label corresponding to each training image.

[0084] In some embodiments of the present application, each training image is a fundus image. The introduction of the preset disease label can refer to step S14, and each training image can have multiple preset disease labels. In some embodiments of the present application, in order to ensure the accuracy of the preset disease label, the preset disease label can be obtained by the doctor through diagnosis of the patient's fundus image and optical coherence tomography (OCT) image.

[0085] In some embodiments of the present application, the preset disease label can also be obtained by detecting each training image through a convolutional neural network model. The convolutional neural network model is pre-trained through optical synchronous tomography images and disease labels obtained by doctors from each optical synchronous tomography image diagnosis. The pre-trained convolutional neural network model can accurately detect each training image, thereby improving the convenience and accuracy of obtaining the preset disease label.

[0086] S22, input each training image into a feature extraction network to obtain a training feature map of each training image.

[0087] In at least one embodiment of the present application, the feature extraction model is a classification module in a machine learning model. If the machine learning model is a CNN (Convolutional Neural Network) model, the CNN model may include, but is not limited to: LeNet model, AlexNet model, VGG model, GoogLeNet model, ResNet model, DenseNet model and Inception model, etc., and the present application does not impose any restrictions on this.

[0088] In some embodiments of the present application, the generation process of each training feature map is substantially the same as the generation process of the fundus feature map, and thus the present application will not repeat the description.

[0089] S23, input each training feature map into the attention mechanism module to obtain a training weighted feature map of each training feature map.

[0090] The attention mechanism module includes a pooling layer, which can be one or more of the following pooling layers: global average pooling layer (GlobalAveragePooling, GAP), global maximum pooling layer (GlobalAveragePooling, GMP).

[0091] In some embodiments of the present application, the attention mechanism module also includes a training pooling layer and a training perception layer, wherein the electronic device inputs each training feature map into the attention mechanism module to obtain a training weighted feature map of each training feature map, including: the electronic device performs pooling processing on each training feature map to obtain a training pooling value, inputs each training pooling value into the training perception layer, and obtains a training weight vector corresponding to each training feature map based on the weight of each neuron in the training perception layer for each training pooling value, and generates a training weighted feature map for each training feature map based on each training feature map and the corresponding training weight vector.

[0092] Specifically, if the training pooling layer in the attention mechanism module is a training global average pooling layer, the training pooling value is a training global average pooling value, the electronic device performs global average pooling processing on each training feature map to obtain a training global average pooling value, and inputs each training global average pooling value into the training perception layer, and obtains a training weight vector corresponding to each training feature map based on the weight of each neuron in the perception layer for each global average pooling value, and then operates on each training feature map and the corresponding training weight vector to obtain a training weighted feature map of each training feature map.

[0093] In this embodiment, the operation method between each training feature map and the corresponding training weight vector can be set by itself, and this application does not limit this. For example, each training feature map can be multiplied by the corresponding training weight vector to obtain a training weighted feature map of each training feature map.

[0094] S24, calling the classification network to classify each training image based on the training weighted feature map to obtain a predicted disease label corresponding to each training image.

[0095] In some embodiments of the present application, the process of generating the predicted disease label is substantially the same as the process of generating the target disease label, and thus the present application will not repeat the description.

[0096] S25, calculating a loss value according to each predicted symptom label and a preset symptom label corresponding to each predicted symptom label, and adjusting a feature extraction network, an attention mechanism module, and a classification network according to the loss value.

[0097] In some embodiments of the present application, the loss value may be a cross-entropy loss value (CrossEntropy). The calculation method of the cross-entropy loss value may refer to the calculation formula of the cross-entropy loss function in the related art, and the present application will not describe it in detail here.

[0098] In some embodiments of the present application, parameters such as the learning rate, weight, and bias in the feature extraction network can be adjusted, parameters such as the pooling window and step size in the attention mechanism module can be adjusted, and parameters such as the weight and bias of the fully connected layer in the classification network can be adjusted.

[0099] S26, determining whether the loss value is within a preset range.

[0100] In this embodiment, the preset range can be set by yourself, and this application does not limit this. For example, the preset range can be [0.10.3]. If the loss value is not within the preset range, the process returns to step S22. If the loss value is within the preset range, step S27 is executed.

[0101] S27, stop adjusting, and obtain the feature extraction model corresponding to the feature extraction network, the attention mechanism model corresponding to the attention mechanism module, and the classification model corresponding to the classification network.

[0102] S28, based on feature extraction model, attention mechanism model and classification model, constitutes a medical image recognition model.

[0103] In this embodiment, the electronic device can combine or splice the feature extraction model, the attention mechanism model and the classification model to obtain a medical image recognition model.

[0104] Through the above implementation, the medical image recognition model is trained according to the training images (fundus images) and the preset disease labels corresponding to the training images, so that the medical image recognition network can learn the association between different disease features (for example, hidden biomarkers) in multiple fundus images and the preset disease labels, so that the trained medical image recognition model can comprehensively and accurately detect the diseases in the fundus images to be identified, thereby effectively assisting doctors in diagnosing the disease.

[0105] In at least one embodiment of the present application, if the pooling layer in the attention mechanism model includes a global average pooling layer and a global maximum pooling layer, the training perception layer includes a first training perception layer and a second training perception layer, and the training pooling value includes a training global average value and a training global maximum value, such as Figure 7FIG. 1 is a flowchart of generating a training weight vector provided by an embodiment of the present application, which specifically includes the following steps:

[0106] S231, perform global average pooling processing on each training feature map to obtain a training global average, input each training global average into the first training perception layer, and obtain a first training weight vector for each training feature map based on the weight of each neuron in the first training perception layer to each training global average.

[0107] In some embodiments of the present application, the generation process of the first training weight vector is substantially the same as the generation process of the first target weight vector, and thus the description thereof will not be repeated in this application.

[0108] S232, perform global maximum pooling processing on each training feature map to obtain a training global maximum, input each training global maximum into the second training perception layer, and obtain a second training weight vector for each training feature map based on the weight of each neuron in the second training perception layer for each training global maximum.

[0109] In some embodiments of the present application, the generation process of the second training weight vector is substantially the same as the generation process of the second target weight vector, and thus the description is not repeated in this application.

[0110] In at least one embodiment of the present application, Figure 8 FIG. 1 is a flowchart of generating a training weighted feature map provided by an embodiment of the present application, which specifically includes the following steps:

[0111] S233, performing operation on each training feature graph and the corresponding first training weight vector to obtain a first operation feature graph corresponding to each training feature graph.

[0112] In some embodiments of the present application, the generation process of the first operation feature map is basically the same as the generation process of the first attention feature map, so this application will not repeat the description.

[0113] S234, performing operation on each training feature graph and the corresponding second training weight vector to obtain a second operation feature graph corresponding to each training feature graph.

[0114] In some embodiments of the present application, the generation process of the second operation feature map is basically the same as the generation process of the second attention feature map, so this application will not repeat the description.

[0115] S235: Generate a training weighted feature graph for each training feature graph according to the first operation feature graph and the second operation feature graph corresponding to each training feature graph.

[0116] In some embodiments of the present application, the electronic device fuses each first operation feature map and the corresponding second operation feature map to obtain a training weighted feature map of each training feature map. Wherein, each first operation feature map and the corresponding second operation feature map can be fused by concatenating (concatenate or concat) according to the channel dimension to obtain a training weighted feature map of each training feature map.

[0117] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application.

[0118] The computer-readable storage medium may be an internal memory of the electronic device described in the above embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. equipped on the electronic device.

[0119] In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0120] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0121] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0122] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A medical image recognition model training method, characterized in that: The method comprises: Acquire multiple training images and preset disease labels corresponding to each training image, wherein each training image is a fundus image; The medical image recognition network is trained based on the multiple training images and the preset disease labels corresponding to each training image to obtain a medical image recognition model, wherein the medical image recognition network includes a feature extraction network, an attention mechanism module and a classification network, and the training process of the medical image recognition network includes: Inputting each training image into the feature extraction network to obtain a training feature map of each training image; Input each training feature map into the attention mechanism module to obtain a training weighted feature map of each training feature map; Calling the classification network to classify each training image based on the training weighted feature map to obtain a predicted disease label corresponding to each training image; Calculate a loss value according to each predicted symptom label and a preset symptom label corresponding to each predicted symptom label, and adjust the feature extraction network, the attention mechanism module and the classification network according to the loss value to obtain a feature extraction model corresponding to the feature extraction network, an attention mechanism model corresponding to the attention mechanism module and a classification model corresponding to the classification network; Based on the feature extraction model, the attention mechanism model and the classification model, the medical image recognition model is composed.

2. The medical image recognition model training method according to claim 1, characterized in that: The attention mechanism module further includes a training perception layer, wherein the step of inputting each training feature map into the attention mechanism module to obtain a training weighted feature map of each training feature map includes: Perform pooling processing on each of the training feature maps to obtain a training pooling value, input each of the training pooling values ​​into the training perception layer, and obtain a training weight vector corresponding to each of the training feature maps based on the weight of each neuron in the training perception layer for each of the training pooling values, and then generate a training weighted feature map for each of the training feature maps based on each of the training feature maps and the corresponding training weight vectors.

3. The medical image recognition model training method according to claim 2, characterized in that: The performing pooling processing on each training feature map to obtain the training pooling value, inputting each training pooling value into the training perception layer, and obtaining the training weight vector corresponding to each training feature map according to the weight of each neuron in the training perception layer to each training pooling value includes: Perform global average pooling processing on each of the training feature maps to obtain a training global average pooling value, input each training global average pooling value into the training perception layer, and obtain a training weight vector for each training feature map based on the weight of each neuron in the perception layer for each global average pooling value.

4. The medical image recognition model training method according to claim 2, characterized in that: The training perception layer further includes a first training perception layer and a second training perception layer, and the training weight vector includes a first training weight vector and a second training weight vector, wherein the pooling process is performed on each training feature map to obtain the training pooling value, each training pooling value is input into the training perception layer, and according to the weight of each neuron of the training perception layer to each training pooling value, the training weight vector corresponding to each training feature map is obtained, which also includes: Performing global average pooling processing on each of the training feature maps to obtain a training global average, inputting each of the training global averages into the first training perception layer, and obtaining a first training weight vector for each of the training feature maps according to a weight of each neuron in the first training perception layer to each of the training global averages; Perform global maximum pooling processing on each of the training feature maps to obtain a training global maximum, input each training global maximum into the second training perception layer, and obtain a second training weight vector for each training feature map based on the weight of each neuron in the second training perception layer for each training global maximum.

5. The medical image recognition model training method according to claim 4, characterized in that: The step of generating a training weighted feature graph of each training feature graph according to each training feature graph and a corresponding training weight vector comprises: Performing operations on each of the training feature graphs and the corresponding first training weight vectors to obtain a first operation feature graph corresponding to each of the training feature graphs; Performing operations on each of the training feature graphs and the corresponding second training weight vectors to obtain a second operation feature graph corresponding to each of the training feature graphs; A training weighted feature map of each training feature map is formed according to the first operation feature map and the second operation feature map corresponding to each training feature map.

6. A medical image recognition method, characterized in that: The method comprises: Acquire a fundus image to be identified; Calling a pre-trained medical image recognition model to recognize the fundus image to be recognized, and obtaining a target disease label of the fundus image to be recognized, wherein the medical image recognition model includes a feature extraction model, an attention mechanism model, and a classification model, and the generation of the target disease label includes: Calling the feature extraction model to extract features from the fundus image to be identified to obtain a fundus feature map; Inputting the fundus feature map into the attention mechanism model to obtain an attention weighted feature map; The classification model is called to classify the attention weighted feature map to obtain a target disease label of the fundus image to be identified.

7. The medical image recognition method according to claim 6, characterized in that: The attention mechanism model also includes a perception layer, wherein the step of inputting the fundus feature map into the attention mechanism model to obtain the attention weighted feature map includes: The fundus feature map is pooled to obtain pooled values, each pooled value is input into the perception layer, and the target weight vector of the fundus feature map is obtained according to the weight of each neuron in the perception layer for each pooled value, and the attention weighted feature map is generated according to the fundus feature map and the target weight vector.

8. The medical image recognition method according to claim 7, characterized in that: The pooling process is performed on the fundus feature map to obtain the pooling value, each pooling value is input to the perception layer, and the target weight vector of the fundus feature map is obtained according to the weight of each neuron of the perception layer to each pooling value, including: Performing global average pooling processing on the fundus feature map to obtain global average pooling values, inputting each global average pooling value into the perception layer, and obtaining the target weight vector based on the weight of each neuron in the perception layer for each global average pooling value.

9. The medical image recognition method according to claim 7, characterized in that: The perception layer further includes a first perception layer and a second perception layer, the target weight vector includes a first target weight vector and a second target weight vector, the pooling process is performed on the fundus feature map to obtain the pooling value, each pooling value is input to the perception layer, and according to the weight of each neuron of the perception layer to each pooling value, the target weight vector of the fundus feature map is obtained, and further includes: Performing global average pooling processing on the fundus feature map to obtain a global average value, inputting each global average value into the first perception layer, and obtaining the first target weight vector according to the weight of each neuron in the first perception layer to each global average value; Perform global maximum pooling processing on the fundus feature map to obtain a global maximum value, input each global maximum value into the second perception layer, and obtain the second target weight vector based on the weight of each neuron in the second perception layer to each global maximum value.

10. The medical image recognition method according to claim 9, characterized in that: The step of generating the attention weighted feature map according to the fundus feature map and the target weight vector comprises: Performing operation on the fundus feature map and the first target weight vector to obtain a first attention feature map; Performing operation on the fundus feature map and the second target weight vector to obtain a second attention feature map; The attention weighted feature map is generated according to the first attention feature map and the second attention feature map.

11. An electronic device, characterized in that: The electronic device comprises: a memory storing at least one instruction; and A processor obtains instructions stored in the memory to implement the medical image recognition model training method as described in any one of claims 1 to 5, or implements the medical image recognition method as described in claims 6 to 10.

Images