A disease identification method and device based on image

Abstract

The embodiment of the present application discloses an image-based disease recognition method and device, wherein the method comprises: performing H feature fusions on the N first modal feature images and the M second modal feature images respectively, obtaining H fused modal feature images to be identified, classifying the H fused modal feature images to be identified according to preset disease categories, obtaining probability values ​​corresponding to each disease category in the preset disease categories, and the probability values ​​characterize the possibility of each category of disease in the preset categories. Thus, at least two modal feature images are fused, and the disease category is automatically detected based on the fused modal feature images, avoiding manual detection, and improving the accuracy of image disease category detection, that is, improving the efficiency of disease category detection, while also avoiding manual selection of images and manual annotation of lesions and lesion categories of images.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image-based disease identification method and device. Background Art

[0002] At present, fundus diseases are classified based on fundus color photographs and OCT images in clinical medicine. The main means of classification includes manual labeling, which is to manually label the captured images, then make disease judgments, and then determine the types of diseases contained in the fundus color photographs and OCT images.

[0003] Existing multimodal automatic detection of fundus diseases includes processing of fundus color photographs and OCT images, but only one OCT image is input and needs to be manually selected. OCT images are often a sequence, and using only one OCT image will lose a lot of OCT modality information. In addition, the obtained OCT image information is manually annotated, resulting in the inability to fully utilize the existing data, resulting in low detection accuracy. Summary of the invention

[0004] In response to the above technical problems, the embodiments of the present application propose an image-based disease recognition method and device to solve the problem that the above method cannot fully utilize existing data, resulting in low detection accuracy.

[0005] A first aspect of an embodiment of the present application provides an image-based disease recognition method, comprising:

[0006] Obtaining N first modality feature maps and M second modality feature maps, where N and M are both integers greater than or equal to 1, wherein the N first modality feature maps are feature maps of N first modality sub-images, respectively, and the N first modality sub-images are obtained according to the first modality image;

[0007] Performing H feature fusions on the N first modal feature graphs and the M second modal feature graphs respectively, to obtain H fused modal feature graphs to be identified, where H is an integer greater than or equal to 2, and the H fused modal feature graphs to be identified all include the first modal feature and the second modal feature, and the weights of the same modal feature in the H fused modal feature graphs to be identified are different;

[0008] The H fused modal feature maps to be identified are classified according to preset disease categories to obtain a probability value corresponding to each disease category in the preset disease category, and the probability value represents the possibility of each disease category in the preset category.

[0009] In one embodiment, for any fused modality feature graph to be identified, the step of fusing the N first modality feature graphs and the M second modality feature graphs includes:

[0010] The N first modal feature graphs are respectively converted into the N third modal feature graphs, and the M second modal feature graphs are respectively converted into the M fourth modal feature graphs, and the N third modal feature graphs and the M fourth modal feature graphs are modally comparable;

[0011] respectively configuring first initial weights for the N third modal feature graphs, and respectively configuring second initial weights for the M fourth modal feature graphs, wherein each of the first initial weights represents the importance of the corresponding third modal feature graph, and each of the second initial weights represents the importance of the corresponding fourth modal feature graph;

[0012] The N third modality feature maps and the M fourth modality feature maps are fused according to the first initial weight and the second initial weight to obtain a fused modality feature map to be identified.

[0013] In one embodiment, it is characterized in that the N third modality feature maps and the M fourth modality feature maps are fused according to the first initial weight and the second initial weight to obtain a fused modality feature map to be identified, including:

[0014] Calculating results of multiplying the N third modal feature graphs by corresponding first initial weights, and calculating results of multiplying the M fourth modal feature graphs by corresponding second initial weights;

[0015] All multiplication results are added together to obtain the fused modality feature map to be identified.

[0016] In one embodiment, the method comprises:

[0017] Respectively obtaining a first importance coefficient and a second importance coefficient of each of the N first modal feature graphs, wherein the first importance coefficient represents the importance of the corresponding first modal feature graph, and the second importance coefficient represents the regional importance of the corresponding first modal feature graph;

[0018] A first disease prediction graph is generated according to the first importance coefficient and the second importance coefficient of each first modality feature graph, wherein the first disease prediction graph represents a disease manifestation graph under the first modality.

[0019] In one embodiment, the method further comprises:

[0020] Respectively obtaining a first importance coefficient and a second importance coefficient of each of the M second modal feature maps, wherein the first importance coefficient represents the importance of the corresponding second modal feature map, and the second importance coefficient represents the regional importance of the corresponding second modal feature map;

[0021] A second disease prediction map is generated according to the first importance coefficient and the second importance coefficient of each second modality feature map, wherein the second disease prediction map represents a disease manifestation map under the second modality.

[0022] In one embodiment, for any first modal feature graph, obtaining a first importance coefficient of the corresponding first modal feature graph includes:

[0023] Obtain an average value of H first initial weights corresponding to the first modal feature graph to obtain a first importance coefficient of the first modal feature graph, wherein the H first initial weights are obtained in a process of performing H times feature fusion on the N first modal feature graphs and the M second modal feature graphs.

[0024] In one embodiment, the N first modality sub-images are obtained according to the first modality image, including:

[0025] The first modality image is cropped to obtain N1 first modality sub-images, and the N1 first modality sub-images are flipped respectively to obtain N2 first modality sub-images, where the sum of N1 and N2 is N.

[0026] In one embodiment, the acquiring N first modal feature graphs and M second modal feature graphs includes:

[0027] Performing feature extraction on the N first modality sub-images using a first feature extraction model to obtain the N first modality feature graphs and a second importance coefficient of each first modality feature graph;

[0028] Feature extraction is performed on the M second modality sub-images using a second feature extraction model to obtain the M second modality feature maps and a second importance coefficient of each second modality feature map.

[0029] In one embodiment, the H fusion modality feature maps to be identified are classified according to preset disease categories to obtain a probability value corresponding to each disease category in the preset disease category, including:

[0030] If the corresponding probability value is greater than 0.5, the disease type corresponding to the corresponding probability value is determined as the target type.

[0031] A second aspect of an embodiment of the present application provides an image-based disease identification device, the device comprising:

[0032] An image acquisition module, used to acquire N first modality feature maps and M second modality feature maps, where N and M are both integers greater than or equal to 1, wherein the N first modality feature maps are feature maps of N first modality sub-images, respectively, and the N first modality sub-images are obtained based on the first modality image;

[0033] An image fusion module, used for performing feature fusion H times on the N first modal feature maps and the M second modal feature maps respectively, to obtain H fused modal feature maps to be identified, where H is an integer greater than or equal to 2, and the H fused modal feature maps to be identified all include the first modal feature and the second modal feature, and the weights of the same modal feature in the H fused modal feature maps to be identified are different;

[0034] The image classification module is used to classify the H fused modal feature maps to be identified according to preset disease categories, and obtain the probability value corresponding to each disease category in the preset disease category, wherein the probability value represents the possibility of each disease category in the preset category.

[0035] In an embodiment of the present application, H fused modal feature maps to be identified are obtained by performing feature fusion H times on the N first modal feature maps and the M second modal feature maps, respectively. The H fused modal feature maps to be identified all contain the first modal feature and the second modal feature, and the weights of the same modal feature in the H fused modal feature maps to be identified are different, thereby achieving the fusion of at least two modal feature images, and automatically detecting the disease category according to the fused modal feature image to avoid manual detection. The H fused modal feature maps to be identified are classified according to preset disease categories to obtain probability values ​​corresponding to each disease category in the preset disease category, and the probability values ​​represent the possibility of each disease category in the preset category, thereby improving the accuracy of image disease category detection, that is, improving the efficiency of disease category detection, and also avoiding manual selection of images and manual annotation of lesions and lesion categories of images. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The features and advantages of the present application will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present application in any way. In the accompanying drawings:

[0037] Figure 1 This is a flowchart of an image-based disease identification method provided by the present application;

[0038] Figure 2 This is a schematic diagram of the fusion process of two modal feature maps of an image-based disease recognition method provided by this application. DETAILED DESCRIPTION

[0039] In the following description, many specific details of the application are set forth by example, so as to provide a thorough understanding of the relevant disclosure. However, for those of ordinary skill in the art, the application can be implemented without these details as is apparent. It should be understood that the use of "system", "device", "unit" and / or "module" terms in the application is a method for distinguishing different parts, elements, parts or assemblies of different levels in a sequential arrangement. However, if other expressions can achieve the same purpose, these terms can be replaced by other expressions.

[0040] It should be understood that when a device, unit or module is referred to as being "on," "connected to," or "coupled to" another device, unit or module, it may be directly on, connected to, coupled to, or communicating with other devices, units or modules, or there may be intermediate devices, units or modules, unless the context clearly indicates an exception. For example, the term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0041] The terms used in this application are only for describing specific embodiments and are not intended to limit the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates an exception, the words "one", "a", "a kind" and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified features, wholes, steps, operations, elements and / or components, and such expressions do not constitute an exclusive list, and other features, wholes, steps, operations, elements and / or components may also be included.

[0042] These and other features and characteristics, methods of operation, functions of related elements of the structure, combinations of parts, and economies of manufacture of the present application may be better understood with reference to the following description and drawings, which form a part of the specification. However, it is to be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of protection of the present application. It is to be understood that the drawings are not drawn to scale.

[0043] This application describes an image-based disease recognition method, such as Figure 1 As shown, the method includes:

[0044] Step S10 obtains N first modal feature maps and M second modal feature maps, where N and M are both integers greater than or equal to 1, wherein the N first modal feature maps are feature maps of N first modal sub-images, and the N first modal sub-images are obtained based on the first modal image.

[0045] Step S20 performs H feature fusions on the N first modal feature maps and the M second modal feature maps respectively to obtain H fused modal feature maps to be identified, where H is an integer greater than or equal to 2. The H fused modal feature maps to be identified all contain the first modal features and the second modal features, and the weights of the same modal features in the H fused modal feature maps to be identified are different.

[0046] Step S30 classifies the H fused modal feature maps to be identified according to preset disease categories, and obtains a probability value corresponding to each disease category in the preset disease category, wherein the probability value represents the possibility of each disease category in the preset category.

[0047] It can be seen that the present implementation method can realize the fusion of at least two modal feature images, and automatically detect the disease category based on the fused modal feature image, thereby avoiding manual detection and improving the accuracy of image disease category detection, that is, improving the efficiency of disease category detection, while also avoiding manual selection of images and manual labeling of image lesions and lesion categories.

[0048] Through step S10, N first modal feature graphs and M second modal feature graphs are obtained, where N and M are both integers greater than or equal to 1. The present application adopts the same number of first modal feature graphs and second modal feature graphs, because with the same number of feature graphs, the subsequent fusion of N first modal feature graphs and M second modal feature graphs will be more uniform, and the fused features obtained will be more obvious, thereby improving the accuracy of identifying diseases. Of course, in other embodiments, the number of first modal feature graphs and the number of second modal feature graphs may also be different.

[0049] Step S20 performs feature fusion H times on the N first modal feature graphs and the M second modal feature graphs respectively to obtain H fused modal feature graphs to be identified, thereby realizing the fusion between two different modal features. By fusing two different modal feature graphs, information complementarity between the two modal feature graphs is achieved, which is more rapid and beneficial to subsequent disease identification, and the fusion between the two modal feature graphs is more complete.

[0050] Through step S30, the probability value corresponding to each disease category in the preset disease category is obtained, and the probability value represents the possibility of each disease category in the preset category, and finally the disease category is determined, thereby avoiding manual detection and improving the accuracy of image disease category detection, that is, improving the efficiency of disease category detection, and also avoiding manual selection of images and manual annotation of image lesions and lesion categories.

[0051] See the following examples Figure 1 as well as Figure 2In some embodiments, for any fused modal feature graph to be identified, the feature fusion of the N first modal feature graphs and the M second modal feature graphs includes: converting the N first modal feature graphs into the N third modal feature graphs, and converting the M second modal feature graphs into the M fourth modal feature graphs, respectively, and the N third modal feature graphs and the M fourth modal feature graphs are comparable in modality.

[0052] Specifically, the present application is a new module for multimodal feature fusion that completes image fusion and classification through the MM-MIL module. The MM-MIL module inherits the interpretability of MIL based on example attention. It fuses example-level CFP / OCT features with example attention weights into instance-level feature vectors, that is, the N first modality feature maps and the M second modality feature maps are feature fused to form a vector feature map. The MM-MIL module is divided into two parts: the first part is a cross-modal mapping part. The cross-modal mapping part accepts the input (n×d) of two modalities, that is, the input (n×d) first modality feature maps and the input (n×d) second modality feature maps, and the (n×d) first modality feature maps and (n×d) second modality feature maps of the two modal inputs are respectively subjected to fully connected layers and layer normalization to remove modal information of different modalities, so that the modalities are comparable, thereby realizing the fusion of the two modal maps.

[0053] The first initial weights are respectively configured for the N third modal feature graphs, and the second initial weights are respectively configured for the M fourth modal feature graphs, each of the first initial weights represents the importance of the corresponding third modal feature graph, and each of the second initial weights represents the importance of the corresponding fourth modal feature graph. The N third modal feature graphs and the M fourth modal feature graphs are fused according to the first initial weights and the second initial weights to obtain a fused modal feature graph to be identified.

[0054] Specifically, the second part of the MM-MIL module is the example attention calculation part, which concatenates the two features output by the first part to obtain a feature of size 2n×d. After two linear layers and activation function layers, the weight of each example is obtained, which is 2n×1. The weight of each example is multiplied by the original feature of each example to obtain the final module output of size 1×d. By using multiple MM-MIL modules, the features learned by each module have different focuses, that is, the features obtained by the fused modality feature map to be identified are different. Combining the information of each module can make the model learn more comprehensively, make decisions more accurate, and obtain more accurate disease types.

[0055] In one embodiment, the N third modal feature graphs and the M fourth modal feature graphs are fused according to the first initial weight and the second initial weight to obtain a fused modal feature graph to be identified, including: respectively calculating the results of multiplying the N third modal feature graphs with the corresponding first initial weights, and respectively calculating the results of multiplying the M fourth modal feature graphs with the corresponding second initial weights. All multiplication results are added together to obtain the fused modal feature graph to be identified.

[0056] Specifically, the MM-MIL module calculates the results of multiplying the N third modal feature maps by the corresponding first initial weights, and respectively calculates the results of multiplying the M fourth modal feature maps by the corresponding second initial weights, and finally obtains the fused modal feature map to be identified, which is further divided into two steps: the first step is to multiply the modal features (2n×d) obtained by the example attention score calculation part by the feature 2n×1 of the weight of each example obtained after passing through the linear layer and the activation function layer twice, and obtain a feature of size 2n×d; the second step is to add the features of the first step on the first dimension (length 2n) to obtain a feature of size 1×d as the output of the module.

[0057] In one embodiment, the method includes: respectively obtaining a first importance coefficient and a second importance coefficient of each of the N first modal feature graphs, wherein the first importance coefficient represents the importance of the corresponding first modal feature graph, and the second importance coefficient represents the regional importance of the corresponding first modal feature graph. By accurately predicting the type of disease based on the dual important features of the importance of the first modal feature graph and the regional importance of the first modal feature graph, the accuracy of the type of disease is improved.

[0058] A first disease prediction graph is generated according to the first importance coefficient and the second importance coefficient of each first modality feature graph, wherein the first disease prediction graph represents a disease manifestation graph under the first modality.

[0059] It should be noted that, by generating a first disease prediction map according to the first importance coefficient and the second importance coefficient of each first modal feature map, the first disease prediction map represents the disease manifestation map under the first modality. In special emergency situations, the modal feature map can be viewed by this method to first predict whether the eye has a disease, making full preparations for subsequent eye treatment. In fact, by visualizing the first modal feature map or the second modal feature map, a preliminary screening and judgment is made for the final confirmation of the type of eye disease, saving a lot of time for the subsequent treatment.

[0060] In one embodiment, the first importance coefficient and the second importance coefficient of each of the M second modal feature graphs are obtained respectively, wherein the first importance coefficient represents the importance of the corresponding second modal feature graph, and the second importance coefficient represents the regional importance of the corresponding second modal feature graph. By using the dual important features of the importance of the second modal feature graph and the regional importance of the second modal feature graph, the disease type can be accurately predicted, thereby improving the accuracy of the disease type.

[0061] A second disease prediction map is generated according to the first importance coefficient and the second importance coefficient of each second modality feature map, wherein the second disease prediction map represents a disease manifestation map under the second modality.

[0062] It should be noted that, by generating a second disease prediction map according to the first importance coefficient and the second importance coefficient of each second modal feature map, the second disease prediction map represents the disease manifestation map under the first modality. In special emergency situations, the modal feature map can be viewed by this method to first predict whether the eye has a disease, making full preparations for subsequent eye treatment. In fact, by visualizing the first modal feature map or the second modal feature map, a preliminary screening and judgment is made for the final confirmation of the type of eye disease, saving a lot of time for the subsequent treatment.

[0063] In one embodiment, for any first modal feature graph, obtaining a first importance coefficient of the corresponding first modal feature graph includes:

[0064] The average value of the H first initial weights corresponding to the first modal feature map is obtained to obtain the first importance coefficient of the first modal feature map, wherein the H first initial weights are obtained in the process of performing H feature fusions on the N first modal feature maps and the M second modal feature maps. The first initial weights are averaged and pooled. Average pooling is an averaging operation. Averaging is to average the fusion feature map results output by multiple MM-MIL modules. By comparing the average value with the preset score, it is determined whether the feature map contains a disease.

[0065] In one embodiment, the N first modality sub-images are obtained based on the first modality image, including: cropping the first modality image to obtain N1 first modality sub-images, flipping the N1 first modality sub-images to obtain N2 first modality sub-images, and the sum of N1 and N2 is N.

[0066] Optionally, the N first modality sub-images are obtained by pseudo-oversampling the first modality image, and the first modality image is cropped to obtain N1 first modality sub-images, wherein the first modality image needs to be cropped at the four corners and the center to obtain N1 first modality sub-images.

[0067] In some embodiments, it is characterized in that the acquiring of N first modal feature graphs and M second modal feature graphs comprises:

[0068] Feature extraction is performed on the N first modality sub-images using a first feature extraction model to obtain the N first modality feature graphs and a second importance coefficient of each first modality feature graph.

[0069] Feature extraction is performed on the M second modality sub-images using a second feature extraction model to obtain the M second modality feature maps and a second importance coefficient of each second modality feature map.

[0070] In another embodiment, feature extraction is performed on N pairs of first modality sub-images and feature extraction is performed on the M pairs of second modality sub-images, and the extracted feature maps are serialized respectively, and then the serialized first modality feature maps and second modality feature maps are respectively sent to two different 2D-CNNs (two-dimensional convolutional neural networks) in parallel, wherein the size of the extracted feature map is w×h×d, such as the size of the feature maps of the N second modality feature sequences is N×w×h×d and M×w×h×d (d=2048, h=w=8), and N and M are 12 respectively, and 12 represents the number of images. ResNet-50 (residual neural network) is used in this solution. Optionally, other feature extraction models can also be used.

[0071] Furthermore, spatial global average pooling is performed on the N first modal feature maps and the M second modal feature maps respectively, so that the first modal feature map and the second modal feature map obtain feature vectors of their own dimensions. The first modal feature map and the second modal feature map are fused through the MM-MIL module to obtain the modal feature map to be identified, thereby achieving the fusion of N first modal feature maps and M second modal feature maps to complement each other's information, thereby further increasing the accuracy of disease judgment.

[0072] In one embodiment, the H fused modal feature maps to be identified are classified according to preset disease categories to obtain probability values ​​corresponding to each disease category in the preset disease categories, including: if the corresponding probability value is greater than 0.5, the disease category corresponding to the corresponding probability value is determined as the target category.

[0073] Preferably, before obtaining the probability of each modal feature, each modal feature needs to be converted into a corresponding weight size through a linear layer, and its output is successively passed through average pooling and sigmoid (S-type function) to finally calculate the average probability of the disease category. If the probability is greater than 0.5, the eye is confirmed to have a disease, and the target disease is determined according to the disease category corresponding to the corresponding modal feature. In fact, it is necessary to compare the preset modal features with the corresponding modal features of the corresponding modal features with the calculated probabilities of each modal feature. The modal features of the preset modal features are compared with the modal features of the disease that needs to be confirmed, and the comparison is often made between modal features with the same weight or similar weights. This will shorten the comparison time and improve the efficiency of confirming the disease category.

[0074] The present application also provides a device for disease identification based on an image, the device comprising:

[0075] An image acquisition module, used to acquire N first modality feature maps and M second modality feature maps, where N and M are both integers greater than or equal to 1, wherein the N first modality feature maps are feature maps of N first modality sub-images, respectively, and the N first modality sub-images are obtained based on the first modality image;

[0076] An image fusion module, used for performing feature fusion H times on the N first modal feature maps and the M second modal feature maps respectively, to obtain H fused modal feature maps to be identified, where H is an integer greater than or equal to 2, and the H fused modal feature maps to be identified all include the first modal feature and the second modal feature, and the weights of the same modal feature in the H fused modal feature maps to be identified are different;

[0077] The image classification module is used to classify the H fused modal feature maps to be identified according to preset disease categories, and obtain the probability value corresponding to each disease category in the preset disease category, wherein the probability value represents the possibility of each disease category in the preset category.

[0078] The two modality images are processed and finally fused through the image acquisition module, the image fusion module and the image classification module, so that at least two modalities can automatically detect the disease category, avoiding manual detection and improving the accuracy of image disease category detection, that is, improving the efficiency of disease category detection, while also avoiding manual selection of images and manual labeling of image lesions and lesion categories.

[0079] It should be noted that the image acquisition module includes acquiring N first modal feature maps and N second modal feature maps, and sending the acquired N first modal feature maps and N second modal feature maps to two different 2D-CNN networks in sequence and parallel, wherein the sizes of the extracted N first modal feature maps and N second modal feature maps are fixed, and the sizes of the N first modal feature maps and N second modal feature maps in each sequence are also related to the number of modal feature maps. Then, the N first modal feature maps and N second modal feature maps are globally averaged and pooled in space to obtain the modal feature map vectorization of the N first modal feature maps and N second modal feature maps.

[0080] The image fusion module includes two parts, the first part is the cross-modal mapping part, and the second part is the weight calculation part. The first part receives the vectorized N first modal feature maps and N second modal feature maps, respectively receives the input of the two modalities (n×d), and respectively passes the two modal features through the fully connected layer and the layer standardization to remove the information of different modalities between the two modal features, so that the information between the remaining two modal features can be compared, and then the information between the two modal features can be clearly contrasted. The second part splices the two modal feature maps output by the first part to obtain a feature of size 2n×d, and obtains the corresponding weight of each feature map after two linear layers and activation function layers, and obtains a feature of size 2n×1. Finally, the corresponding weight of each modal feature is multiplied by the N first modal feature maps and the N second modal feature maps to calculate a feature of size 1×d.

[0081] Optionally, the fusion module is a MM-MIL module, which is a new module for fusing at least two modal feature maps. It inherits the interpretability of MIL based on example attention. It fuses example-level CFP (color fundus picture) / OCT (optical coherence tomography) modal features with example attention weights into an instance-level feature vector. The above weights are all represented as example attention weights here. Among them, the fusion into instance-level feature vectors needs to be sent to h MM-MIL modules together through the output of SW-GAP (spatial global average pooling) of at least two modalities. In the MM-MIL module, 2n instance-level features (2n×d) are aggregated into 1 feature (1×d) with multimodal information. Each feature is converted into a category decision score (1×m) as output through a linear layer. The output category decision score (1×m) is then averaged and activated by sigmoid (which can be called an S-type function) to obtain the final probability prediction score, that is, the above-mentioned disease category probability value. Average pooling is an averaging operation. For this model, the output results of multiple MM-MIL modules are averaged. If the average score of a certain category is greater than 0.5, the model believes that the eye contains the disease. Using multiple MM-MIL modules, each module has different features. Combining the information of each module can make the model learn more comprehensively and make more accurate decisions.

[0082] Furthermore, the features of the MM-MIL module need to be calculated by averaging the attention weights calculated in the h MM-MILs, that is, the h attention scores of each example, to obtain the attention weight of each example. Based on the attention weight, we can see the contribution of each example to the entire model, that is, to what extent each example affects the decision of the model.

[0083] Specifically, the calculation method can be divided into two steps: the first step is to multiply the example attention score (2n×1) by the feature after modal mapping (2n×d) element by element to obtain a feature of size 2n×d.

[0084] The second step adds the features of the first step in the first dimension (length 2n) to obtain 1×d features as the output of the module. Since these two steps are actually equivalent to matrix multiplication, only one multiplication sign is drawn on the module diagram.

[0085] The image classification module compares the preset modal features with the modal features of the disease that needs to be confirmed, and then confirms the disease category. Moreover, the comparison is often made between modal features with the same or similar weights. This will shorten the comparison time and improve the efficiency of confirming the disease category.

[0086] The disease classification module obtains the probability value corresponding to each disease category in the preset disease category, and the probability value represents the possibility of each disease category in the preset category. It is necessary to classify the H fused modal feature maps to be identified, wherein the fused modal feature maps to be identified need to be averaged and compared with the preset probability size. If it is greater than the preset probability, it is confirmed that the eye image has a disease, and the type of disease is determined based on the modal feature map, wherein the preset probability size is 0.5. In fact, it is necessary to first average and pool the weighted probabilities of the modal features, and then activate them through sigmoid (S-type function) to obtain the weighted probability of the averaged modal feature map, and compare the weighted average probability of the modal feature map with the preset probability.

[0087] It should be understood that the above specific embodiments of the present application are only used to illustrate or explain the principles of the present application, and do not constitute a limitation to the present application. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present application should be included in the protection scope of the present application. In addition, the claims attached to the present application are intended to cover all changes and modifications that fall within the scope and boundaries of the attached claims, or the equivalent forms of such scope and boundaries.

Claims

1. An image-based disease identification device, characterized in that: The device comprises: An image acquisition module, used to acquire N first modality feature maps and M second modality feature maps, where N and M are both integers greater than or equal to 1, wherein the N first modality feature maps are feature maps of N first modality sub-images, respectively, and the N first modality sub-images are obtained based on the first modality image; An image fusion module, used for performing feature fusion H times on the N first modal feature maps and the M second modal feature maps respectively, to obtain H fused modal feature maps to be identified, where H is an integer greater than or equal to 2, and the H fused modal feature maps to be identified all include the first modal feature and the second modal feature, and the weights of the same modal feature in the H fused modal feature maps to be identified are different; The specific process of obtaining H fused modal feature maps to be identified includes: performing spatial global average pooling on N first modal feature maps and M second modal feature maps respectively, so that the first modal feature map and the second modal feature map obtain feature vectors of their own dimensions, and fusing the first modal feature map and the second modal feature map through the MM-MIL module to obtain the fused modal feature map to be identified; Respectively obtaining a first importance coefficient and a second importance coefficient of each of the N first modal feature graphs, wherein the first importance coefficient represents the importance of the corresponding first modal feature graph, and the second importance coefficient represents the regional importance of the corresponding first modal feature graph; Generating a first disease prediction graph according to the first importance coefficient and the second importance coefficient of each first modality feature graph, wherein the first disease prediction graph represents a disease manifestation graph under the first modality; Respectively obtaining a first importance coefficient and a second importance coefficient of each of the M second modal feature maps, wherein the first importance coefficient represents the importance of the corresponding second modal feature map, and the second importance coefficient represents the regional importance of the corresponding second modal feature map; Generating a second disease prediction graph according to the first importance coefficient and the second importance coefficient of each second modality feature graph, wherein the second disease prediction graph represents a disease manifestation graph under the second modality; The image classification module is used to classify the H fused modal feature maps to be identified according to preset disease categories, and obtain the probability value corresponding to each disease category in the preset disease category, wherein the probability value represents the possibility of each disease category in the preset disease category.

2. The image-based disease identification device according to claim 1, characterized in that: For any fused modal feature graph to be identified, the step of fusing the N first modal feature graphs and the M second modal feature graphs includes: Respectively converting the N first modal feature graphs into N third modal feature graphs, and respectively converting the M second modal feature graphs into M fourth modal feature graphs, wherein the N third modal feature graphs and the M fourth modal feature graphs are comparable in modality; respectively configuring first initial weights for the N third modal feature graphs, and respectively configuring second initial weights for the M fourth modal feature graphs, wherein each of the first initial weights represents the importance of the corresponding third modal feature graph, and each of the second initial weights represents the importance of the corresponding fourth modal feature graph; The N third modality feature maps and the M fourth modality feature maps are fused according to the first initial weight and the second initial weight to obtain a fused modality feature map to be identified.

3. The image-based disease identification device according to claim 2, characterized in that: The method comprises fusing the N third modality feature maps and the M fourth modality feature maps according to the first initial weight and the second initial weight to obtain a fused modality feature map to be identified, comprising: Calculating results of multiplying the N third modal feature graphs by corresponding first initial weights, and calculating results of multiplying the M fourth modal feature graphs by corresponding second initial weights; All multiplication results are added together to obtain the fused modality feature map to be identified.

4. According to the image-based disease identification device of claim 2 or 3, for any first modal feature graph, obtaining the first importance coefficient of the corresponding first modal feature graph comprises: Obtain an average value of H first initial weights corresponding to the first modal feature graph to obtain a first importance coefficient of the first modal feature graph, wherein the H first initial weights are obtained in a process of performing H times feature fusion on the N first modal feature graphs and the M second modal feature graphs.

5. The image-based disease identification device according to claim 1, characterized in that: The N first modality sub-images are obtained according to the first modality image, including: The first modality image is cropped to obtain N1 first modality sub-images, and the N1 first modality sub-images are flipped respectively to obtain N2 first modality sub-images, where the sum of N1 and N2 is N.

6. The image-based disease identification device according to any one of claims 1 to 3 and 5, characterized in that: The obtaining of N first modal feature graphs and M second modal feature graphs includes: Performing feature extraction on the N first modality sub-images using a first feature extraction model to obtain the N first modality feature graphs and a second importance coefficient of each first modality feature graph; Feature extraction is performed on the M second modality sub-images using a second feature extraction model to obtain the M second modality feature maps and a second importance coefficient of each second modality feature map.

7. The image-based disease identification device according to claim 1, characterized in that: The H fusion modality feature maps to be identified are classified according to the preset disease categories to obtain the probability value corresponding to each disease category in the preset disease categories, including: If the corresponding probability value is greater than 0.5, the disease type corresponding to the corresponding probability value is determined as the target type.

Images