Intelligent prediction method and system across modal medical images

Abstract

The application provides a cross-modal medical image intelligent prediction method and system, the method comprising: performing feature extraction on each modality medical image respectively, obtaining multi-scale features of each modality, and mapping the multi-scale features to corresponding high latitude feature spaces; based on the attention mechanism and the high latitude features of different modal medical images, obtaining the correlation information between different modal medical images; based on the correlation information between different modalities, obtaining complementary features, fusing the complementary features, and obtaining the integrated features after fusion; reasoning and predicting the integrated features to obtain the intelligent prediction result of the cross-modal medical image. The application fully utilizes the cross-modal medical image information by obtaining the correlation information between the cross-modal medical images, extracting the complementary features based on the correlation information, and fusing to obtain the integrated features, realizes more comprehensive and accurate prediction, and assists doctors in obtaining more accurate judgment results for patients.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a cross-modality medical image intelligent prediction method and system. BACKGROUND

[0002] Currently, medical image modalities are diverse, including CT, MRI, ultrasound, PET, etc. However, the data difference between different modalities is significant, and there are great differences in resolution, imaging principle, data representation form, etc., which makes it a big problem to integrate multi-modality data. For example, CT focuses on displaying bones and calcified lesions, and its image presents high density resolution; while MRI has advantages in soft tissue imaging, and can provide rich tissue texture and functional information. In traditional analysis methods, only one kind of modality data can often be processed alone, and it is difficult to fully play the complementary value of multi-modality data, resulting in incomplete information utilization and affecting the accuracy and reliability of disease prediction. Semantic understanding and association dilemma There is no unified standard for the semantic expression of the same anatomical structure or pathological feature in each modality medical image. The structural features in CT image may correspond to different signal features in MRI image. This semantic difference makes it extremely difficult to associate and comprehensively understand the information across modalities. At present, in medical image analysis, most studies are still limited to intra-modality semantic analysis, lacking effective cross-modality semantic mapping and association mechanism. This leads to the inability to comprehensively and accurately extract semantic information related to diseases from multi-modality images when facing complex diseases, limiting the in-depth understanding and accurate judgment of intelligent prediction systems for diseases.

[0003] Therefore, how to overcome the above-mentioned technical problems and defects becomes a problem that needs to be focused on. SUMMARY

[0004] In order to overcome the above-mentioned problems in the prior art, the present application provides a cross-modality medical image intelligent prediction method and system, which adopts the following technical solutions:

[0005] In the first aspect, the present application provides a cross-modality medical image intelligent prediction method, comprising:

[0006] Respectively extracting features of each modality medical image, obtaining multi-scale features of each modality, and mapping the multi-scale features to corresponding high latitude feature spaces;

[0007] Based on the attention mechanism and the high latitude features of different modality medical images, obtaining the association information between different modality medical images;

[0008] Based on the association information between different modalities, obtaining complementary features, fusing the complementary features, and obtaining fused comprehensive features;

[0009] Reasoning and prediction are performed on comprehensive features to obtain intelligent prediction results of cross-modal medical images.

[0010] Further, before feature extraction is performed on each modality medical image, the following is included:

[0011] Noise removal and image enhancement are performed on medical image data of different modalities to obtain a first medical image set;

[0012] Color images in the first medical image set are converted to grayscale images to obtain a second medical image set;

[0013] Normalization processing is performed on the second medical image set to obtain a third medical image set.

[0014] Further, normalization processing is performed on the second medical image set to obtain a third medical image set, including:

[0015] The maximum and minimum normalization method is used to perform normalization processing on the second medical image set, and the maximum and minimum normalization formula is: where x max represents the maximum pixel value in the image, x min represents the minimum pixel value in the image, x represents the original data, and x m represents the normalized data of the original data.

[0016] Further, feature extraction is performed on each modality medical image to obtain multi-scale features of each modality, and the multi-scale features are mapped to corresponding high-latitude feature spaces, including:

[0017] The third medical image set is input into corresponding branch channels of a deep convolutional neural network model; based on different convolution kernels and different step lengths of different branch channels, multi-scale feature extraction is performed on corresponding modality medical images;

[0018] The multi-scale feature maps are down-sampled by the pooling layer of the corresponding branch channel to obtain key feature maps of the corresponding medical images;

[0019] The key feature maps obtained by the pooling layer that are below the preset threshold resolution are up-sampled to the preset resolution, and the up-sampled feature maps are weighted and fused with the feature maps of the previous layer to obtain new feature maps; the new feature maps of different scales and the key feature maps are weighted and fused to generate high-latitude features of the corresponding modality medical images.

[0020] Further, the weighted fusion formula is: p x-1 = α1D(p x ) + α2p x-1 , where D(·) is an inverse convolution transformation function, α1 and α2 are weight coefficients, p x-1 is the feature map of the previous layer, and px is the current feature map.

[0021] Further, based on the attention mechanism and the high-dimensional features of different modal medical images, the association information between different modal medical images is obtained, including:

[0022] The high-dimensional features of a modal medical image are taken as query features, and the high-dimensional features of medical images of other modalities are taken as key features and value features, and a cross-attention module is constructed.

[0023] In each cross-attention mechanism module, a similarity matrix between the query features and the key features is obtained based on a dot product similarity calculation, and the similarity matrix is converted into an attention weight matrix through a softmax function.

[0024] Based on the attention weight, the value features are weighted and summed to obtain the feature representation of the medical image of other modalities associated with the query features.

[0025] Further, based on the association information between different modalities, complementary features are obtained, the complementary features are fused, and the fused comprehensive features are obtained, including:

[0026] Features whose weights are lower than a preset threshold in one modality but higher than the preset threshold in another modality are extracted, and the features are taken as complementary features. The complementary features are simply spliced to obtain an initial fusion feature vector. The initial fusion feature vector is taken as the input of a multilayer perceptron to obtain the fused comprehensive features.

[0027] In a second aspect, the present application also provides a cross-modal medical image intelligent prediction system, including:

[0028] A feature extraction module is configured to extract features from each modality of medical images respectively, obtain multi-scale features of each modality, and map the multi-scale features to corresponding high-dimensional feature spaces.

[0029] An association information acquisition module is configured to obtain association information between different modal medical images based on the attention mechanism and the high-dimensional features of different modal medical images.

[0030] An intelligent prediction result acquisition module is configured to obtain complementary features based on the association information between different modalities, fuse the complementary features, obtain fused comprehensive features, and infer and predict the comprehensive features to obtain intelligent prediction results of cross-modal medical images.

[0031] In a third aspect, the present application provides an electronic device, including:

[0032] One or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs comprising instructions that, when executed by the device, cause the device to perform the method of the first aspect.

[0033] In a fourth aspect, the present application provides a computer readable storage medium, having stored therein a computer program, which, when executed on a computer, causes the computer to perform the method of the first aspect.

[0034] In a fifth aspect, the present application provides a computer program for performing the method of the first aspect when the computer program is executed on a computer.

[0035] In a possible design, the program in the fifth aspect can be stored, in whole or in part, on a storage medium packaged together with the processor, or stored, in part or in whole, on a storage medium not packaged together with the processor.

[0036] The present application has the following beneficial effects:

[0037] 1. The present application extracts features through a deep convolutional neural network, and then fuses through a multilayer perceptron, avoiding directly using a multilayer perceptron to extract features from medical images, and losing the spatial locality of medical images.

[0038] 2. The present application extracts features from each modality medical image respectively, obtains multi-scale features of each modality, and maps the multi-scale features to corresponding high latitude feature spaces; obtains the correlation information between different modal medical images based on the attention mechanism and the high latitude features of different modal medical images; obtains complementary features based on the correlation information between different modalities, fuses the complementary features, and obtains the integrated features after fusion; reasons and predicts the integrated features to obtain the intelligent prediction result of the cross-modality medical image. The present application obtains the correlation information between the cross-modality medical images, extracts the complementary features based on the correlation information, and fuses to obtain the integrated features, which fully utilizes the cross-modality medical image information, realizes more comprehensive and accurate prediction, and assists doctors to obtain more accurate judgment results for patients. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 An exemplary system architecture diagram to which embodiments of the present application can be applied;

[0040] Figure 2 A flowchart of the cross-modality medical image intelligent prediction method of the embodiments of the present application;

[0041] Figure 3A cross-modal medical image preprocessing flowchart of an embodiment of the present application;

[0042] Figure 4 A high-latitude feature generation flowchart of an embodiment of the present application;

[0043] Figure 5 A system flowchart of an embodiment of the present application;

[0044] Figure 6 A computer device schematic diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application; the description and claims herein and the above description of drawings herein utilize terms such as "comprising", "having" and "including" to convey construction that is inclusive of, but not limited to, such features unless otherwise indicated; the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0046] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in an embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all directed to the same embodiment, or to a single alternative embodiment.

[0047] In order for those skilled in the art to better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings.

[0048] As shown in Figure 1 The system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a communication link medium between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0049] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0050] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smartphones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, and the like.

[0051] The server 105 can be a server providing various services, such as a background server providing support for a page displayed on the terminal devices 101, 102, and 103.

[0052] It should be noted that the cross-modality medical image intelligent prediction method provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the cross-modality medical image intelligent prediction system is generally arranged in a server / terminal device.

[0053] It should be understood that Figure 1 The number of terminal devices, networks, and servers in

[0054] For ease of understanding, the basic technical concepts related to the embodiments of the present application are described as follows: medical image intelligent prediction is one of the important applications of artificial intelligence technology in the medical field. The core of medical image intelligent prediction is to analyze and process medical images through the combination of computer vision and deep learning, thereby assisting doctors in early disease diagnosis, risk assessment, and treatment plan planning, and providing more accurate and efficient medical services for patients.

[0055] The embodiments of the present application relate to the application of artificial intelligence in the medical field. By extracting features from cross-modality medical images and converting the extracted features through a network, the correlation information between different modality image data is mined. The comprehensive feature representation of the cross-modality medical image is obtained through the full connection layer of the deep convolution network, and the intelligent prediction task of the cross-modality medical image is completed through reasoning on the comprehensive features, thereby assisting doctors in patient disease diagnosis, risk assessment, and treatment plan planning.

[0056] With reference to Figure 2 , a cross-modality medical image intelligent prediction method flowchart of the present application is shown, and the method includes the following steps:

[0057] At step 201, feature extraction is performed on each modality medical image respectively, multi-scale features of each modality are obtained, and high-latitude features corresponding to the multi-scale features are generated.

[0058] In the field of intelligent prediction, various medical images are usually needed. Image information collected by medical images is analyzed to understand the internal structure information of the patient's body. Medical images help doctors diagnose patients. Medical images usually include color ultrasound, X-ray examination, CT scan, magnetic resonance imaging, etc. Different types of medical images have different characteristics. Cross-modal integration of different types of medical images can achieve comprehensive and in-depth understanding and accurate judgment of the health status of patients.

[0059] In one possible implementation, before feature extraction is performed on each modality medical image, please refer to Figure 3 , the specific content includes:

[0060] At step 31, noise removal and image enhancement are performed on medical image data of different modalities to obtain a first medical image set.

[0061] At step 32, color images in the first medical image set are converted to grayscale images to obtain a second medical image set. Since noise exists when different medical images are obtained, or the obtained pictures have a blur phenomenon. At this time, noise removal and image enhancement need to be performed on the obtained medical images, so that the data features of the medical images can be accurately obtained in the feature extraction process. Mean filtering, median filtering and other methods can be used to remove noise from medical images. Medical image enhancement can be achieved by adjusting the brightness, contrast, saturation and other parameters of the medical image to highlight the important features of the medical image. The specific method is not described in detail here.

[0062] At step 33, the second medical image set is normalized to obtain a third medical image set.

[0063] The second medical image set can be normalized using the maximum and minimum normalization method. The maximum and minimum normalization formula is: where x max represents the maximum pixel value in the image, x min represents the minimum pixel value in the image, x represents the original data, and x m represents the normalized data of the original data. By removing noise and enhancing images from medical images, converting color images in the medical image set to grayscale images and performing normalization, the efficiency of subsequent medical image feature extraction can be improved.

[0064] In the embodiments of the present application, the preprocessed images of each modality are subjected to feature extraction by a deep convolutional neural network.

[0065] In the embodiments of the present application, the deep convolutional neural network comprises a plurality of convolutional layers, pooling layers and fully connected layers; the convolutional layers are subjected to convolution operation by sliding the convolution kernel on the medical images, thereby extracting the local features of different medical images; and the pooling layers are used to reduce the data dimension of different medical images while retaining the important features of different medical images. Since the medical images of different modalities have their own feature information, when the medical images of each modality are subjected to feature extraction, a multi-modal input channel is set for the deep convolutional neural network model, and an independent input branch is set for each modality of medical images, each branch comprising a series of convolutional layers and pooling layers for extracting the features of the corresponding modality of medical images.

[0066] In one possible implementation, multi-scale features of each modality are acquired, and corresponding high-dimensional features are generated based on the multi-scale features, please refer to Figure 4 , the specific content includes:

[0067] Step 41, input the third medical image set into the branch channel corresponding to the deep convolutional neural network model.

[0068] Step 42, based on different convolution kernels and different step lengths of different branch channels, multi-scale feature extraction of the medical images of the corresponding modality is performed. For example, a 3x3 or 5x5 convolution kernel can be used, wherein the 3x3 convolution kernel can acquire local information of the corresponding medical image, and the 5x5 convolution kernel can acquire feature information of a wider area in the medical image.

[0069] Step 43, the multi-scale feature maps are down-sampled by the pooling layer of the corresponding branch channel to acquire the key feature maps of the corresponding medical images.

[0070] Step 44, the feature maps with a resolution lower than the preset threshold (low resolution) acquired by the pooling layer are up-sampled to the preset resolution (high resolution), and the up-sampled feature maps are weighted and fused with the feature maps of the previous layer to acquire new feature maps.

[0071] Step 45, the new feature maps of different scales and the key feature maps are weighted and fused to generate high-dimensional features of the medical images of the corresponding modality. By up-sampling the feature maps with low resolution to the preset resolution (i.e. high resolution), the multi-scale information of the medical images of different modalities can be acquired, and high-dimensional feature extraction of the feature maps can be realized.

[0072] The weighted fusion formula of the present application is: p x-1 =α1D(p x )+α2p x-1where D(·) is a deconvolution transformation function, a1, a2 are weight coefficients, p x-1 is the feature map of the previous layer, p x is the current feature map.

[0073] In step 202, based on the attention mechanism and the high-latitude features of different modal medical images, the association information between different modal medical images is obtained.

[0074] The attention mechanism is a deep learning technology that simulates the attention allocation in the human visual and cognitive system. The core idea is to dynamically adjust the weight distribution of input data, so that the model can focus on the most important information part, thereby improving the processing efficiency and performance. In the field of image processing, the attention mechanism significantly improves the model's understanding ability and task execution effect by capturing key regions and features in images. By introducing the attention mechanism, the application can deeply understand and mine the correlation between different modal medical images.

[0075] In the embodiments of the application, based on the attention mechanism and the high-latitude features of different modal medical images, the association information between different modal medical images is obtained, including:

[0076] The high-latitude features of a modal medical image are taken as query features, and the high-latitude features of medical images of other modalities are taken as key features and value features to construct a cross-attention module; the relationship between the high-latitude features of different modal medical images is constructed; (the cross-attention module includes: constructing query features for each modal medical image, and constructing key features and value features from the high-latitude features of medical images of other modalities). Taking CT and MRI images as an example, some anatomical structure features of CT images may be related to corresponding position or function-related features in MRI images. By taking CT image features as queries and MRI image features as keys and values, the relevant part of the MRI image can be found by calculating the similarity between the query and the key, thereby realizing the information interaction between cross-modal medical images.

[0077] In each cross-attention mechanism module, a similarity matrix between the query features and the key features is obtained based on a dot product similarity calculation, and the similarity matrix is converted into an attention weight matrix by a softmax function. The similarity calculation can measure the similarity between two feature vectors, and the greater the value, the more similar the two features. The softmax function converts the similarity matrix into an attention weight matrix in the form of a probability distribution, so that the sum of the attention weights of each query feature to all key features is 1. In this way, the attention weight can reflect the attention degree of the query feature to each key feature in the key feature space, that is, which key features are more important for the current query feature, providing a weight basis for the acquisition of associated information. The softmax function is an activation function widely used in neural networks and machine learning, mainly used to convert an input vector into a probability distribution. Its core role is to map a set of real values to a normalized probability distribution, so that the value of each element is between 0 and 1, and the sum of all elements is 1.

[0078] For each feature vector q i of a modality medical image F i and each feature vector k j of another modality medical image F j , the dot product q i ·k j of the two modalities medical images is calculated to obtain a similarity matrix. The similarity matrix is converted into an attention weight matrix A by a softmax function, so that A ij represents the attention weight of the i-th feature of F i to the j-th feature in F j , and ∑ j A ij = 1.

[0079] The value features are weighted and summed based on the attention weights to obtain the feature representation of the other modality medical image associated with the query feature. The application filters and aggregates information related to the query feature by weighting and summing the value features based on the attention weights. Since the attention weights represent the attention degree of the query feature to each key feature, the process of weighted summing is equivalent to extracting the part related to the query feature from the value features according to these weights, and combining them into a new feature representation. For example, when the CT image feature is used as the query, the MRI image feature representation associated with the CT image feature obtained by weighted summing contains information in the MRI image that is strongly associated with the CT image feature, thereby achieving the purpose of acquiring associated information from different modalities. The value features F j are weighted and summed using the attention weight matrix A to obtain the F i feature representation F j associated with F j-related . Fj-related =∑ j A ij ·v j , where v j is the eigenvector of F j , F j-related contains the F i medical image features associated with the F j medical image features, F j-related represents the association information between the F i , F j two medical image high-dimensional features.

[0080] The above iteration operation is performed on the medical image high-dimensional features in each attention module respectively to obtain the association information between different modal medical images.

[0081] In step 203, based on the association information between different modalities, complementary features are obtained, the complementary features are fused, and the fused comprehensive features are obtained; the comprehensive features are inferred and predicted to obtain an intelligent prediction result of the cross-modal medical image.

[0082] In the embodiments of the present application, based on the association information between different modalities, complementary features are obtained, the complementary features are fused, and the fused comprehensive features are obtained, including:

[0083] Extract the features whose weight in one modality is lower than a preset threshold, but whose weight in another modality is higher than the preset threshold, and take the features as complementary features. For example, the texture features and shape features of CT, the functional features and signal intensity features of MRI, the metabolic features of PET, etc. Then, the complementary features are fused to obtain more comprehensive image information representation.

[0084] Before inputting the complementary features into the multi-layer perception machine, the complementary features are first spliced and fused, and the spliced features are taken as the input of the multi-layer perception machine. In the process of fusing the complementary features, it is hoped that more original information can be retained, so the present application adopts the splicing method to initially fuse the complementary features.

[0085] The complementary features are simply spliced to obtain an initial fusion feature vector. Assuming that there are n different types of complementary features, the dimensions of which are d1, d2,..., d n , the dimension of the spliced initial feature vector is D = d1 + d +,... + d n . For example, a medical image feature vector with a dimension of 10 and a physiological index feature vector with a dimension of 5 are spliced to obtain a new feature vector with a dimension of 10 + 5 = 15.

[0086] The initial fusion feature vector is taken as the input of the multi-layer perception machine to obtain the fused comprehensive features.

[0087] In the embodiment of the present application, the initial fusion feature vector is taken as the input of the multilayer perceptron to obtain the fused comprehensive feature, including:

[0088] The spliced initial feature vector is input into the input layer of the multilayer perceptron, in the first hidden layer, the input feature is multiplied by the weight matrix of the layer, and then the bias vector is added, and then the output of the hidden layer is obtained through the activation function; this output is taken as the input of the next hidden layer, and the above process is repeated until all hidden layers are passed through, and the output of the last hidden layer is the fused comprehensive feature. The initial feature vector of the present application passes through the hidden layers in turn, in each hidden layer, the neurons perform weighted summation on the input initial feature vector and process it through the activation function. The neurons of different hidden layers combine and transform the input features through the learned weights to realize feature fusion.

[0089] The multilayer perceptron (MLP) is a deep learning model based on feedforward neural network, which is composed of multiple neuron layers, usually including input layer, hidden layer and output layer.

[0090] The present application can comprehensively utilize the advantages of cross-modal medical images to more comprehensively reflect the pathological information of patients and reduce the occurrence of missed diagnosis and misdiagnosis. For example, in the diagnosis of brain diseases, CT can well show structural abnormalities such as skull fracture, while MRI is more sensitive to small lesions in the brain parenchyma such as early cerebral infarction and brain tumor. Extracting complementary features can combine the unique advantages of different modalities to more comprehensively cover the information about the brain condition.

[0091] In the embodiment of the present application, the comprehensive feature is inferred and predicted to obtain the intelligent prediction result of the cross-modal medical image, including:

[0092] The feature vector fused through the hidden layer is transmitted to the output layer; when the prediction result is a classification task, the Softmax function of the output layer of the multilayer perceptron calculates the probability of each class, and the class with the highest probability is the prediction result; when the prediction result is a regression task, the output layer of the multilayer perceptron directly outputs a numerical value as the predicted value of the target variable.

[0093] For the initial feature vector X=(x1, x2,..., x i ...,xn ), the weight matrix of the first hidden layer is W l = (w l ij ), the bias b l = (b l 1, b l 2,...,b l m ). Where n is the dimension of the input features, m is the number of neurons in the first hidden layer, w l ij represents the connection weight from the jth neuron of the (l-1)th layer to the ith neuron of the lth layer, which determines the degree of influence of the previous layer of neurons on the current layer of neurons, the number of rows of the matrix is equal to the number of neurons in the lth layer, and the number of columns is equal to the number of neurons in the (l-1)th layer; b l m is the bias of the mth neuron in the lth layer. The input of the lth hidden layer is z l = W l x l-1 +b l , where x l-1 is the output of the (l-1)th layer, and for the first hidden layer, x 0 = x, i.e. the initial feature vector of the input. The output of the lth layer is x l = f(z l ), where f is an activation function, which is used to perform a nonlinear transformation on the weighted input of the neuron, so that the neural network can learn more complex function relationships; common activation functions such as ReLU function f(x) = max(0, x) set values less than 0 to 0 and keep values greater than 0 unchanged; through the activation function, the neuron can produce a nonlinear output, thereby improving the expression ability of the neural network.

[0094] Through the sequential calculation of multiple hidden layers, the features are continuously combined and transformed, realizing feature fusion; finally, when reaching the output layer, the output obtained is the final fused features. If it is a classification task, the output layer usually uses a Softmax function to convert the output into a probability distribution, i.e. where zi is the ith element of the output layer before Softmax processing, zj is the jth element of the output layer before Softmax processing; k is the number of output layer neurons, representing the number of categories; y i represents the probability that the sample belongs to the ith category, and the sum of the probabilities of all categories is 1. If it is a regression task, the output layer is usually linear output, i.e. y = W O X L +b o , where L is the total number of hidden layers, W O and b ois the weight matrix and bias vector of the output layer, and y is the final predicted continuous numerical result.

[0095] The present application can fully utilize the unique information of different modal medical images by fusing complementary features, synthesize the unique information of medical images of different modalities, and then comprehensively analyze, evaluate and predict, so that the analysis, evaluation and prediction results are more accurate.

[0096] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. When the program is executed, the processes of the above-mentioned embodiments can be included. The storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0097] It should be understood that although each step in the flowchart of the accompanying drawings is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other sequences. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or sub-steps or stages of other steps.

[0098] With reference to the above Figure 5 , the cross-modal medical image intelligent prediction system described in the embodiment comprises:

[0099] The feature extraction module 501 is configured to extract features from each modality medical image respectively, obtain multi-scale features of each modality, and map the multi-scale features to corresponding high-latitude feature spaces.

[0100] The correlation information acquisition module 502 is configured to acquire correlation information between different modal medical images based on the attention mechanism and the high-latitude features of different modal medical images.

[0101] The intelligent prediction result acquisition module 503 is configured to acquire complementary features based on the correlation information between different modalities, fuse the complementary features, acquire fused comprehensive features, perform reasoning and prediction on the comprehensive features, and acquire intelligent prediction results of cross-modal medical images.

[0102] To solve the above technical problems, the embodiment of the present application further provides a computer device. For details, please refer to Figure 6 , Figure 6 The basic structure block diagram of the computer device of the embodiment is shown in FIG. 1.

[0103] The computer device 6 comprises a memory 6a, a processor 6b and a network interface 6c which are connected to each other through a system bus. It should be noted that only the computer device 6 with components 6a-6c is shown in the figure, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be alternatively implemented. Among them, the computer device herein can be understood by those skilled in the art as a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0104] The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server and other computing devices. The computer device can interact with the user through a keyboard, a mouse, a remote controller, a touchpad or a voice control device.

[0105] The memory 6a includes at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 6a can be an internal storage unit of the computer device 6, such as a hard disk or a memory of the computer device 6. In other embodiments, the memory 6a can also be an external storage device of the computer device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 6. Of course, the memory 6a can also include both the internal storage unit and the external storage device of the computer device 6. In the present embodiment, the memory 6a is generally used to store an operating system and various application software installed on the computer device 6, such as a program code of the cross-modality medical image intelligent prediction method, etc. In addition, the memory 6a can also be used to temporarily store various data that have been output or will be output.

[0106] The processor 6b can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 6b is generally used to control the overall operation of the computer device 6. In the present embodiment, the processor 6b is used to run the program code or process data stored in the memory 6a, such as running the program code of the cross-modality medical image intelligent prediction method.

[0107] The network interface 6c can include a wireless network interface or a wired network interface, and is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0108] The present application also provides another embodiment, i.e., to provide a non-volatile computer readable storage medium storing a program of a cross-modality medical image intelligent prediction method, which can be executed by at least one processor to enable the at least one processor to perform the steps of the cross-modality medical image intelligent prediction method as described above.

[0109] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the method described in each embodiment of the present application.

[0110] Obviously, the above-described embodiments are only some of the embodiments of the present application, not all the embodiments, and the drawings show the preferred embodiments of the present application, but do not limit the patent scope of the present application. The present application can be implemented in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing specific embodiments, or make equivalent replacements to some of the technical features. Any equivalent structure made by using the content of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the scope of the patent protection of the present application.

Claims

1. A cross-modal medical image intelligent prediction method, characterized in that, include: Feature extraction is performed on each modality of medical images to obtain multi-scale features of each modality, and the multi-scale features are mapped to the corresponding high-dimensional feature space, including inputting the third medical image set into the corresponding branch channel of the deep convolutional neural network model; Based on different convolutional kernels and different timings of different branch channels, multi-scale feature extraction is performed on the corresponding modal medical images. The multi-scale feature maps are downsampled through pooling layers of the corresponding branch channels to obtain key feature maps of the corresponding medical images. Feature maps with resolutions below a preset threshold obtained from the key feature maps of the pooling layers are upsampled to a preset resolution, and the upsampled feature maps are weighted and fused with the feature maps of the previous layer to obtain new feature maps. The new feature maps of different scales and the key feature maps are weighted and fused to generate high-dimensional features of the corresponding modal medical images. The weighted fusion formula is: ,in It is the deconvolution transformation function. , These are the weighting coefficients. This is the feature map of the previous layer. This is the current feature map; Before extracting features from each modality of medical images, the process includes: removing noise and enhancing images from medical image data of different modalities to obtain a first medical image set; converting color images in the first medical image set to grayscale images to obtain a second medical image set; and normalizing the second medical image set to obtain a third medical image set. Based on the attention mechanism and the high-dimensional features of different modal medical images, the correlation information between different modal medical images is obtained; Based on the correlation information between different modalities, complementary features are obtained, and the complementary features are fused to obtain the fused comprehensive features. The comprehensive features are then used for inference and prediction to obtain intelligent prediction results for cross-modal medical images.

2. The cross-modal medical image intelligent prediction method according to claim 1, characterized in that, The second medical image set is normalized to obtain the third medical image set, which includes: The second medical image set was normalized using the max-min normalization method, where the max-min normalization formula is: ,in This represents the maximum pixel value in the image. Represents the minimum pixel value in the image. Represents the original data. This represents the data after the original data has been normalized.

3. The cross-modal medical image intelligent prediction method according to claim 1, characterized in that, Based on attention mechanisms and the high-dimensional features of different modalities of medical images, correlation information between different modalities of medical images is obtained, including: A cross-attention module is constructed by using high-dimensional features of one modality of medical images as query features and high-dimensional features of other modalities of medical images as key and value features. In each cross-attention mechanism module, the similarity matrix between the query feature and the key feature is obtained based on the dot product similarity calculation, and the similarity matrix is ​​transformed into an attention weight matrix through the softmax function; By weighting and summing the value features based on attention weights, feature representations of other modal medical images associated with the query features are obtained.

4. The cross-modal medical image intelligent prediction method according to claim 1, characterized in that, Based on the correlation information between different modalities, complementary features are obtained, and these complementary features are fused to obtain the fused comprehensive features, including: Extract features that have a weight below a preset threshold in one modality but a weight above a preset threshold in another modality, and use these features as complementary features; The complementary features are simply concatenated to obtain the initial fused feature vector; The initial fused feature vector is used as input to a multilayer perceptron to obtain the fused comprehensive features.

5. A cross-modal medical image intelligent prediction system, used to implement the cross-modal medical image intelligent prediction method of claims 1-4, characterized in that, include: The feature extraction module is used to extract features from medical images of each modality, obtain multi-scale features of each modality, and map the multi-scale features to the corresponding high-dimensional feature space. The association information acquisition module is used to acquire association information between different modalities of medical images based on attention mechanisms and high-dimensional features of different modalities of medical images; The intelligent prediction result acquisition module is used to acquire complementary features based on the correlation information between different modalities, fuse the complementary features, and obtain the fused comprehensive features. By reasoning and predicting comprehensive features, intelligent prediction results for cross-modal medical images can be obtained.

6. An electronic device, characterized in that, include: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs including instructions that, when executed by the device, cause the device to perform the steps of the cross-modal medical image intelligent prediction method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the steps of the cross-modal medical image intelligent prediction method as described in any one of claims 1 to 4.

Images