Bone tumor multi-modal image intelligent diagnosis system and method based on deep learning

Abstract

The invention relates to the field of bone tumor multi-modal image intelligent diagnosis, in particular to a bone tumor multi-modal image intelligent diagnosis system and method based on deep learning. The method comprises the steps of obtaining and standardizing image sample data of different modes of a patient, and obtaining single-mode features based on the standardized image sample data of the different modes so as to generate cross-mode collaborative features; generating a fusion feature vector based on the single-mode feature and the cross-mode collaborative feature; obtaining an individual feature vector of the patient, and integrating the individual feature vector of the patient with the fusion feature vector to generate a preliminary diagnosis result; and obtaining a final diagnosis result based on the preliminary diagnosis result. The problem that the image data has deviation in numerical value due to different imaging principles and physical characteristics of images of different modalities is solved; the image features of different modalities may have different influences and correlations; and a complex relation between image features cannot be fully captured by a simple feature weighting and superposition method.

Description

Technical Field

[0001] The present invention relates to the field of multimodal imaging intelligent diagnosis of bone tumors, and in particular to a multimodal imaging intelligent diagnosis system and method for bone tumors based on deep learning. Background Art

[0002] Bone tumor is a malignant disease that seriously affects the quality of life of patients. Early and accurate diagnosis is the key to improving patient survival rate and prognosis. With the development of modern imaging technology, multimodal imaging methods such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and X-ray play a vital role in the diagnosis of bone tumors. Multimodal imaging can provide information on the morphology, metabolic characteristics, blood flow, and tumor microenvironment of tumors, allowing doctors to analyze the patient's condition from multiple angles. However, the analysis of multimodal imaging requires high diagnostic experience of doctors. The accuracy and consistency of doctors' diagnosis depend largely on their own experience. Different doctors may have different interpretations of images, resulting in inconsistent diagnostic results. The integration and processing of different modality imaging features is also relatively complex.

[0003] At the same time, the existing multimodal imaging diagnosis and analysis methods have the following problems: due to the different imaging principles and physical properties of images of different modalities (such as MRI, CT and PET), there are numerical deviations in the image data, which affects the subsequent feature extraction and analysis, and thus affects the accuracy of the diagnosis; image features of different modalities may have different influences and correlations, and how to effectively extract and fuse image features of different modalities is a challenge; simple feature weighting and superposition methods cannot fully capture the complex relationship between image features, so flexible and effective feature fusion methods are needed; in medical diagnosis, how to effectively combine the individual characteristics of patients with multimodal imaging features to improve diagnostic accuracy. Summary of the invention

[0004] The present invention provides a bone tumor multimodal imaging intelligent diagnosis system and method based on deep learning to solve the problems that different imaging principles and physical properties of images of different modalities lead to numerical deviations in image data, thereby affecting subsequent feature extraction and analysis, and further affecting the accuracy of diagnosis; image features of different modalities may have different influences and correlations; simple feature weighting and superposition methods cannot fully capture the complex relationship between image features; and how to effectively combine individual characteristics of patients with multimodal imaging features.

[0005] The multimodal imaging intelligent diagnosis system and method for bone tumors based on deep learning of the present invention specifically include the following technical solutions: The multimodal imaging intelligent diagnosis method for bone tumors based on deep learning includes the following steps: S1: Obtain image sample data of different modalities of the patient, use an adaptive normalization algorithm to standardize the image sample data of different modalities, obtain standardized image sample data of different modalities, and form standardized image sample data sets of different modalities; S2: Based on the standardized image sample datasets of different modalities, the single modality features are obtained through an improved deep convolutional neural network; based on the single modality features, a dynamic adaptive modality combination method is designed to generate cross-modality collaborative features; S3: Based on single-modal features and cross-modal collaborative features, a feature fusion algorithm with dynamic weighted nonlinear combination is used to generate a fused feature vector; S4: Obtain the patient's individual information, extract features from the patient's individual information and integrate them into the patient's individual feature vector; use a personalized diagnostic network to integrate the patient's individual feature vector with the fusion feature vector to generate a preliminary diagnosis result; based on the preliminary diagnosis result, obtain the final diagnosis result.

[0006] Preferably, the S1 specifically includes: Obtain the patient's image sample data of different modalities to form image sample data sets of different modalities; use an adaptive normalization algorithm to standardize the image sample data of different modalities. The specific formula is as follows: , Among them, Represents the standardized The first mode Image sample data; Indicates The first mode Image sample data; Indicates The mean of the modes; Indicates The variance of the modes; represents the adaptive balance coefficient; Indicates the number of modalities currently being processed; It means summing the variance of all modes.

[0007] Preferably, the S2 specifically includes: The specific formula of the improved deep convolutional neural network is: , in, Indicates The first mode The characteristics of the image samples; represents the rectified linear unit activation function; Indicates The convolutional layer weight matrix of the modality; Represents the standardized The first mode Image sample data; Indicates The convolutional layer bias term of the modality; Represents the activation function, which is used to generate dynamic weights; The convolutional layer weight matrix representing dynamic weight calculation; Represents the bias term for dynamic weight calculation.

[0008] Preferably, the S2 specifically includes: The generation formula of cross-modal collaborative features is: , in, Indicates The first mode The image samples and The first mode Cross-modal collaborative features generated by image samples; represents the hyperbolic tangent activation function; Indicates the use to generate The first mode and the The weight matrix of the cross-modal collaborative features of the modalities; Indicates The first mode The characteristics of the image samples and the The first mode The sum of the features of image samples; Indicates that it is used for The first mode and the Bias term for cross-modal collaborative feature generation of modalities; The weight matrix representing the modal interaction is used to extract the difference information between the modalities; Indicates The first mode The characteristics of the image samples and the The first mode The difference in characteristics of image samples; Represents the bias term used for modal interaction.

[0009] Preferably, the S3 specifically includes: The dynamic weighted nonlinear combination feature fusion algorithm adopts a dynamic weighting strategy to weight the features of image samples of each modality, and performs nonlinear mapping on the features of image samples of all modalities to generate a fused feature vector; the calculation formula of the fused feature vector is: , in, represents the fused feature vector; Indicates Adaptive weights of the modalities; represents the hyperbolic tangent function, which is used to provide nonlinear mapping; Indicates The first mode characteristics that affect the sample; Indicates The number of image samples of each modality; Indicates The number of image samples of each modality; Indicates the number of modalities currently being processed; Indicates The first mode and the Adaptive weights of cross-modal collaborative features between modalities; Indicates The first mode The image samples and The first mode Cross-modal collaborative features generated from image samples.

[0010] Preferably, the S4 specifically includes: The individual feature weight matrix is ​​used to weight the patient's individual feature vector to obtain the weighted patient individual feature vector; the fusion feature weight matrix is ​​used to weight the fusion feature vector to obtain the weighted fusion feature vector; and a nonlinear transformation is introduced to generate the probability distribution of the patient's disease category.

[0011] Preferably, the S4 specifically includes: The calculation formula for the preliminary diagnosis result is: , in, Represents the preliminary diagnosis result, which is used to reflect the probability of the patient being classified into different disease categories; Indicates the conversion of any real vector into a probability distribution; represents the activation function; represents the individual feature weight matrix; represents the individual feature vector of the patient; Represents the fusion feature weight matrix; represents the fused feature vector; represents the bias term of the personalized diagnosis network; Based on the preliminary diagnosis results, the disease category with the highest probability is selected as the final diagnosis result.

[0012] The multimodal imaging intelligent diagnosis system for bone tumors based on deep learning includes the following parts: Data acquisition module, adaptive standardization module, multimodal image feature extraction module, patient individual feature extraction module, multimodal image feature fusion module, personalized diagnosis network module; Data acquisition module: acquires the image sample data of different modalities of the patient and the individual information of the patient, outputs the image sample data of different modalities of the patient to the adaptive standardization module, and outputs the individual information of the patient to the individual feature extraction module of the patient; Adaptive standardization module: Based on the image sample data of different modalities of the patient, the image sample data of different modalities are standardized using an adaptive standardization algorithm to obtain standardized image sample data of different modalities, and the standardized image sample data of different modalities are output to the multimodal image feature extraction module; Multimodal image feature extraction module: The standardized image sample data of different modalities are processed through an improved deep convolutional neural network to obtain single-modal features; based on the single-modal features, a dynamic adaptive modality combination method is designed to generate cross-modal collaborative features by introducing interactive features between modalities, and the single-modal features and cross-modal collaborative features are output to the multimodal image feature fusion module; Multimodal image feature fusion module: Based on the single-modal features and cross-modal collaborative features of the multimodal image feature extraction module, the fused feature vector is calculated using the feature fusion algorithm of dynamic weighted nonlinear combination, and the fused feature vector is output to the personalized diagnosis network module; Patient individual feature extraction module: extracts features from the patient's individual information to obtain the patient's individual feature vector, and outputs the patient's individual feature vector to the personalized diagnosis network module; Personalized diagnosis network module: Based on the patient's individual feature vector and fusion feature vector, a personalized diagnosis network is used to generate preliminary diagnosis results, and based on the preliminary diagnosis results, a final diagnosis result is generated.

[0013] The beneficial effects of the technical solution of the present invention are: 1. By using an adaptive standardization algorithm to standardize image sample data of different modalities, the problem of image sample data of each modality not being unified in scale is effectively eliminated, so that the accuracy of subsequent feature extraction and diagnosis is improved. Especially in the intelligent diagnosis of complex diseases (such as bone tumors), the standardized image sample data of different modalities can be analyzed under a unified standard, which significantly improves the accuracy of image diagnosis.

[0014] 2. The standardized image sample data of different modalities are processed by an improved deep convolutional neural network to obtain single-modal features. Adaptive weights are introduced to enable the image sample features of different modalities to automatically adjust their weights during feature fusion, thereby enhancing the expressive power of the fused features.

[0015] 3. Based on single-modal features, a dynamic adaptive modal combination method is designed. By calculating the sum and difference of modal features, the common and unique information between modalities is represented, and rich cross-modal collaborative features are generated. The dynamic adaptive modal combination method improves the synergy between modalities, making the effect of feature fusion obvious, and providing a comprehensive feature representation for subsequent intelligent diagnosis.

[0016] 4. The single modal features and cross-modal collaborative features are dynamically weighted fused using a feature fusion algorithm of dynamic weighted nonlinear combination, and the learning ability of complex relationships is enhanced through nonlinear mapping. The final generated fusion feature vector significantly improves the accuracy and robustness of the multimodal imaging intelligent diagnosis system for bone tumors.

[0017] 5. The individual feature vectors of patients and the fusion feature vectors are integrated through the personalized diagnosis network to generate preliminary diagnosis results that meet the actual situation of the patients, which not only improves the accuracy of diagnosis, but also enhances the adaptability to individual differences of patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a structural diagram of the multimodal imaging intelligent diagnosis system for bone tumors based on deep learning according to the present invention; Figure 2 This is a flow chart of the deep learning-based multimodal imaging intelligent diagnosis method for bone tumors described in the present invention. DETAILED DESCRIPTION

[0019] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0021] The specific scheme of the multimodal imaging intelligent diagnosis system and method for bone tumors based on deep learning provided by the present invention is described in detail below with reference to the accompanying drawings.

[0022] See attached Figure 1 , which shows a structure diagram of a multimodal imaging intelligent diagnosis system for bone tumors based on deep learning provided by an embodiment of the present invention. The system includes the following parts: Data acquisition module, adaptive standardization module, multimodal image feature extraction module, patient individual feature extraction module, multimodal image feature fusion module, personalized diagnosis network module; Data acquisition module: acquires the image sample data of different modalities of the patient and the individual information of the patient, outputs the image sample data of different modalities of the patient to the adaptive standardization module, and outputs the individual information of the patient to the individual feature extraction module of the patient; Adaptive standardization module: Based on the image sample data of different modalities of the patient in the data acquisition module, the image sample data of different modalities are standardized using an adaptive standardization algorithm to obtain standardized image sample data of different modalities, and the standardized image sample data of different modalities are output to the multimodal image feature extraction module; Multimodal image feature extraction module: The standardized image sample data of different modalities are processed through an improved deep convolutional neural network to obtain single-modal features. Based on the single-modal features, a dynamic adaptive modality combination method is designed to generate cross-modal collaborative features by introducing interactive features between modalities. The single-modal features and cross-modal collaborative features are output to the multimodal image feature fusion module. Multimodal image feature fusion module: Based on the single-modal features and cross-modal collaborative features of the multimodal image feature extraction module, the fused feature vector is calculated using the feature fusion algorithm of dynamic weighted nonlinear combination, and the fused feature vector is output to the personalized diagnosis network module; Patient individual feature extraction module: extracts features from the individual information of patients from the data acquisition module to obtain individual feature vectors of patients, and outputs the individual feature vectors of patients to the personalized diagnosis network module; Personalized diagnosis network module; based on the patient individual feature vector of the patient individual feature extraction module and the fused feature vector of the multimodal image feature fusion module, a personalized diagnosis network is used to generate a preliminary diagnosis result, and based on the preliminary diagnosis result, a final diagnosis result is generated.

[0023] See attached Figure 2 , which shows a flow chart of a multimodal imaging intelligent diagnosis method for bone tumors based on deep learning provided by an embodiment of the present invention, the method comprising the following steps: S1. Obtain image sample data of different modalities of the patient, use an adaptive normalization algorithm to standardize the image sample data of different modalities, obtain standardized image sample data of different modalities, and form a standardized image sample data set of different modalities.

[0024] Obtain image sample data of different modalities of patients to form image sample data sets of different modalities , Indicates The first mode Image sample data, , Indicates The number of image samples of each modality; Since the imaging principles and physical characteristics of various modalities, such as magnetic resonance imaging (MRI), computed tomography (CT) and positron emission tomography (PET), are different, resulting in numerical deviations in the images of various modalities, in order to ensure that the image sample data of different modalities can be analyzed under a unified standard and make the subsequent feature extraction and diagnosis accurate, it is necessary to use an adaptive normalization algorithm to standardize the image sample data of different modalities, which is crucial to improving the accuracy of imaging diagnosis, especially in cases involving complex diseases (such as bone tumors); The adaptive normalization algorithm obtains the distribution characteristics of the image sample data by calculating the mean and variance of the image sample data of each modality. The mean reflects the central tendency of the image sample data, while the variance indicates the degree of dispersion of the image sample data. The process of calculating the mean is to add up all the image sample data and divide it by the number of image samples to obtain the average value of each modality. The calculation of the variance is to sum the squares of the differences between each image sample and the mean, and then divide it by the number of samples to evaluate the distribution breadth of the image sample data. After obtaining the mean and variance, standardization is continued. The core of standardization is to convert each image sample data to a unified scale. By subtracting each image sample from the corresponding mean, the influence of the mean is eliminated and the image sample data is centralized. At the same time, the variance is also considered to avoid errors caused by variance differences between modalities. Specifically, the square root of the variance is used to standardize the image sample data, thereby adjusting the image sample data to a standard distribution. In addition, in order to enhance the flexibility and adaptability of the adaptive standardization algorithm, an adaptive balance coefficient is introduced. The adaptive balance coefficient ensures that the standardized processing can adapt to the changes in the image sample data of each modality by weighted averaging the variances of all modalities. The standardization formula is: , in, Represents the standardized The first mode Image sample data; Indicates The first mode Image sample data; Indicates The mean value of the image sample data of the modality is used to reflect the The central tendency of the image sample data of the modality is calculated as follows: , Indicates The number of image samples of each modality; Indicates The variance of the image sample data of the modality is used to describe the The distribution breadth of the image sample data of the modality is calculated as follows: ; Represents the adaptive balance coefficient, which is used to control the influence of variance on standardization, enhance the stability of standardization, and ensure the balance between image sample data of different modalities. It can be set according to the specific implementation scenario and is not limited here; Indicates the number of modalities currently being processed; Indicates the sum of the variances of all modality image sample data, which is used to enhance the robustness of the standardization process; After standardization, the standardized image sample data of different modalities are integrated into a standardized image sample data set of different modalities. The formula is as follows: , in, Represents the first Image sample datasets of various modalities; The adaptive normalization algorithm improves the accuracy and effectiveness of data analysis by solving the scale unification problem of multimodal imaging sample data, providing important support for the intelligent diagnosis of bone tumors.

[0025] S2. Based on the standardized image sample datasets of different modalities, the single modal features are obtained through an improved deep convolutional neural network. Based on the single modal features, a dynamic adaptive modal combination method is designed to generate cross-modal collaborative features.

[0026] Based on the standardized image sample data sets of different modalities, the standardized image sample data of each modality is processed by an improved deep convolutional neural network to obtain single-modal features; The improved deep convolutional neural network adopts dynamic feature weighting to automatically adjust the weights of different features, and performs convolution operations on standardized image sample data of different modalities. The weight matrix of the convolutional layer is multiplied by the standardized image sample data of different modalities, and then a convolutional layer bias term is added to introduce additional flexibility; by applying nonlinear activation functions (such as ReLU), the expressive power of the improved deep convolutional neural network is enhanced to capture complex patterns and output feature representations after nonlinear transformation; in order to introduce dynamic weights, use function, The output value of the function is between 0 and 1, which means that each feature can be given different weights according to its importance when being integrated; The extraction formula of unimodal features is: , in, Indicates The first mode The characteristics of the image samples; Represents the rectified linear unit activation function, which is used to introduce nonlinearity and enhance the expressive power of the improved deep convolutional neural network; Indicates The convolution layer weight matrix of this modality is used to perform the convolution operation and extract the local features of the image sample data. It is generated through deep learning training. Its dimension depends on the size of the convolution kernel and the size of the input feature map. It can be set according to the specific implementation scenario and is not limited here. Represents the standardized The first mode Image sample data; Indicates A convolutional layer bias term of a certain modality, used to adjust the offset of the convolution output, improve flexibility, and is generated through deep learning training; Represents the activation function, which is used to generate dynamic weights so that the feature weighting is between 0 and 1, limiting the range of influence of the feature; The convolutional layer weight matrix representing dynamic weight calculation is used to generate dynamic weights to reflect the importance of features, which is generated through deep learning training; Represents the bias term of dynamic weight calculation, which is used to adjust the output of dynamic weight calculation to increase flexibility and is generated through deep learning training.

[0027] Based on single-modal features, a dynamic adaptive modal combination method is designed to generate rich cross-modal collaborative features by introducing interactive features between modalities. Features extracted from different modalities are combined. Instead of simply adding the features together, a new feature representation is generated using a dynamic weighting method that combines the similarities and differences between the modalities. Specifically, the common and unique information between the modalities is represented by calculating the sum and difference of the modal features. The combined features are then activated through nonlinear activation functions, such as the hyperbolic tangent activation function, The activation function is used to further enhance the expressive power of cross-modal collaborative features. The generation formula of cross-modal collaborative features is: , in, Indicates The first mode The image samples and The first mode The cross-modal collaborative features generated by image samples are used to capture the similarities and differences between modalities; represents the hyperbolic tangent activation function, which is used to enhance the expression of the relationship between different features; Indicates the use to generate The first mode and the The weight matrix of the cross-modal collaborative features of the modalities is generated through deep learning training to adjust the extraction of common information between modalities; Indicates The first mode The characteristics of the image samples and the The first mode The sum of the features of the image samples is used to combine the common information of the two modalities, capture the commonalities of the modalities, and form a new feature representation; Indicates that it is used for The first mode and the The bias term for cross-modal collaborative feature generation of the modalities is used to increase flexibility, adjust the output results, and generate through deep learning training; Represents the activation function, which is used to generate dynamic weights so that the feature weighting is between 0 and 1, limiting the range of influence of the feature; The weight matrix representing the modal interaction is used to extract the difference information between the modalities and further strengthen the relationship between the features; Indicates The first mode The characteristics of the image samples and the The first mode The difference of the features of the image samples is used to capture the information between the modalities and help generate rich feature representations; Represents the bias term for modal interaction, which is used to increase flexibility and adjust the output results so that unique information extraction is flexible and effective, and is generated through deep learning training.

[0028] The above steps effectively enhance the synergy between different modal features. All generated cross-modal collaborative features will be integrated into a cross-modal collaborative feature set. , providing a basis for subsequent feature fusion.

[0029] S3. Based on single-modal features and cross-modal collaborative features, a feature fusion algorithm with dynamic weighted nonlinear combination is used to generate a fused feature vector.

[0030] Based on single-modality features and cross-modality collaborative features, a feature fusion algorithm of dynamic weighted nonlinear combination is used to calculate the fusion feature vector, aiming to effectively integrate the feature information in the image sample data of each modality to improve the accuracy and reliability of the multimodal imaging intelligent diagnosis system for bone tumors.

[0031] The feature fusion algorithm of dynamic weighted nonlinear combination adopts a dynamic weighting strategy to weight the features of image samples of each modality. Specifically, for each modality, an adaptive weight is calculated to reflect the contribution of the features of image samples of each modality to the final fusion features. By adjusting the adaptive weight, the expression ability of the fusion features is optimized.

[0032] The cross-modal collaborative features are further processed. The cross-modal collaborative features are generated by the interactive information between different modalities. The cross-modal collaborative features can capture the similarities and differences between modalities. By introducing cross-modal collaborative features, rich feature information can be obtained.

[0033] When performing fusion calculations, nonlinear mapping is performed on the features of image samples of all modalities to enhance the learning ability of complex relationships. The use of hyperbolic tangent function for nonlinear mapping can effectively handle the nonlinear relationship between the features of image samples, not only considering the contribution of single-modal features, but also combining the influence of cross-modal collaborative features, thereby generating a comprehensive fusion feature vector. The calculation formula of the fused feature vector is: , in, It represents the fusion feature vector, which contains the information of single-modal features and cross-modal collaborative features, reflecting the comprehensive effect of all modalities; Indicates The adaptive weights of the modalities are used to adjust the influence of the features of the image samples of each modality in the fusion features. They can be set according to the specific implementation scenario and are not limited here. represents the hyperbolic tangent function, which is used to provide nonlinear mapping; Indicates The first mode characteristics that affect the sample; Indicates The number of image samples of each modality; Indicates The number of image samples of each modality; Indicates the number of modalities currently being processed; Indicates The first mode and the The adaptive weight of the cross-modal collaborative feature between the modalities is used to adjust the influence of the cross-modal collaborative feature on the fusion feature vector, which can be set according to the specific implementation scenario and is not limited here; Indicates The first mode The image samples and The first mode Cross-modal collaborative features generated from image samples.

[0034] The feature fusion algorithm of dynamic weighted nonlinear combination ensures the flexibility and dynamic adaptability of feature fusion, thereby improving the accuracy and robustness of the multimodal imaging intelligent diagnosis system for bone tumors and is suitable for intelligent diagnosis.

[0035] S4. Obtain the patient's individual information, extract features from the patient's individual information and integrate them into the patient's individual feature vector; use a personalized diagnostic network to integrate the patient's individual feature vector with the fusion feature vector to generate a preliminary diagnosis result; based on the preliminary diagnosis result, obtain the final diagnosis result.

[0036] In order to improve the accuracy and personalization of medical diagnosis, especially when dealing with complex diseases (such as bone tumors), it can significantly improve the treatment effect of patients. A personalized diagnostic network is used to integrate the patient's individual feature vector with the fusion feature vector to generate a preliminary diagnosis result.

[0037] The personalized diagnosis network obtains the patient's individual information from the electronic health record system. The patient's individual information includes but is not limited to the patient's basic information (such as age, gender), health status (such as weight, height), medical history (such as previous medical history, family medical history), living habits (such as diet, exercise) and psychological state, and extracts features from the patient's individual information and integrates them into a patient individual feature vector. The acquisition of the patient's individual information and the feature extraction and integration methods are all technical means well known to those skilled in the art and will not be elaborated here.

[0038] In order to extract important information from the patient's individual feature vector, the individual feature weight matrix is ​​used and point multiplication is performed with the individual feature vector to obtain the weighted patient individual feature vector, which condenses the importance of each individual feature to the diagnosis. At the same time, the fused feature vector is processed by the fused feature weight matrix to obtain the weighted fused feature vector, which adjusts the contribution of the fused feature in the final diagnostic result and ensures that the influence of the image sample features of each modality in the diagnosis is appropriate.

[0039] The weighted individual feature vector of the patient is added to the weighted fusion feature vector, which can reflect the personalized characteristics of the patient and the combined effect of multimodal imaging; In order to enhance the expressive power of the personalized diagnosis network, the activation function is used to introduce nonlinear transformation to capture complex patterns and relationships, so that the personalized diagnosis network can handle complex diagnostic tasks and improve accuracy. The nonlinearly transformed features will be input into a conversion layer to generate the probability distribution of the patient's disease category. Specifically, The activation function converts any real vector into a probability distribution so that the sum of the output probabilities is 1; The calculation formula for the preliminary diagnosis result is: , in, Represents the preliminary diagnosis result, which is used to reflect the probability of the patient being classified into different disease categories; Indicates converting any real number vector into a probability distribution to ensure that the probability value of each disease category is between 0 and 1, and the sum is 1; represents the activation function, which is used to introduce nonlinearity to enhance the stability and convergence of the personalized diagnosis network; Represents the individual feature weight matrix, which is used to extract important information from the patient's individual characteristics (such as age, gender, and medical history) to regulate the impact on the initial diagnosis results. It is generated through deep learning training; represents the individual feature vector of the patient; Represents the fusion feature weight matrix, which is used to adjust the contribution of the fusion feature in the preliminary diagnosis results and is generated through deep learning training; represents the fused feature vector; Represents the bias term of the personalized diagnosis network, which is used to ensure the balance of the output under specific conditions and is generated through deep learning training.

[0040] Based on the preliminary diagnosis results, the disease category with the highest probability is selected as the final diagnosis result, which not only provides specific diagnostic information but also reflects the relative credibility of different diagnostic results.

[0041] In summary, a multimodal imaging intelligent diagnosis system and method for bone tumors based on deep learning has been completed.

[0042] The order of the embodiments of the invention is for description only and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0043] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0044] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A multimodal imaging intelligent diagnosis method for bone tumors based on deep learning, characterized in that: The following steps are involved: S1: Obtain image sample data of different modalities of the patient, use an adaptive normalization algorithm to standardize the image sample data of different modalities, obtain standardized image sample data of different modalities, and form standardized image sample data sets of different modalities; S2: Based on the standardized image sample datasets of different modalities, the single modality features are obtained through an improved deep convolutional neural network; based on the single modality features, a dynamic adaptive modality combination method is designed to generate cross-modality collaborative features; S3: Based on single-modal features and cross-modal collaborative features, a feature fusion algorithm with dynamic weighted nonlinear combination is used to generate a fused feature vector; S4: Obtain individual information of the patient, extract features of the individual information of the patient and integrate them into an individual feature vector of the patient; A personalized diagnostic network is used to integrate the patient's individual feature vector with the fusion feature vector to generate a preliminary diagnostic result; based on the preliminary diagnostic result, the final diagnostic result is obtained.

2. The method for intelligent diagnosis of bone tumors based on multimodal imaging based on deep learning according to claim 1, characterized in that: The S1 specifically includes: Obtain the patient's image sample data of different modalities to form image sample data sets of different modalities; use an adaptive normalization algorithm to standardize the image sample data of different modalities. The specific formula is as follows: ， in, Represents the standardized The first mode Image sample data; Indicates The first mode Image sample data; Indicates The mean of the modes; Indicates The variance of the modes; represents the adaptive balance coefficient; Indicates the number of modalities currently being processed; It means summing the variance of all modes.

3. The method for intelligent diagnosis of bone tumors based on deep learning multimodal imaging according to claim 1, characterized in that: The S2 specifically includes: The specific formula of the improved deep convolutional neural network is: ， in, Indicates The first mode The characteristics of the image samples; represents the rectified linear unit activation function; Indicates The convolutional layer weight matrix of the modality; Represents the standardized The first mode Image sample data; Indicates The convolutional layer bias term of the modality; Represents the activation function, which is used to generate dynamic weights; The convolutional layer weight matrix representing dynamic weight calculation; Represents the bias term for dynamic weight calculation.

4. The method for intelligent diagnosis of bone tumors based on multimodal imaging based on deep learning according to claim 1, characterized in that: The S2 specifically includes: The generation formula of cross-modal collaborative features is: ， in, Indicates The first mode The image samples and The first mode Cross-modal collaborative features generated by image samples; represents the hyperbolic tangent activation function; Indicates the use to generate The first mode and the The weight matrix of the cross-modal collaborative features of the modalities; Indicates The first mode The characteristics of the image samples and the The first mode The sum of the features of image samples; Indicates that it is used for The first mode and the Bias term for cross-modal collaborative feature generation of modalities; The weight matrix representing the modal interaction is used to extract the difference information between the modalities; Indicates The first mode The characteristics of the image samples and the The first mode The difference in characteristics of image samples; Represents the bias term used for modal interaction.

5. The method for intelligent diagnosis of bone tumors based on multimodal imaging based on deep learning according to claim 1, characterized in that: The S3 specifically includes: The feature fusion algorithm of dynamic weighted nonlinear combination adopts a dynamic weighting strategy to weight the features of image samples of each modality, and performs nonlinear mapping on the features of image samples of all modalities to generate a fused feature vector; the calculation formula of the fused feature vector is: ， in, represents the fused feature vector; Indicates Adaptive weights of the modalities; represents the hyperbolic tangent function, which is used to provide nonlinear mapping; Indicates The first mode characteristics that affect the sample; Indicates The number of image samples of each modality; Indicates The number of image samples of each modality; Indicates the number of modalities currently being processed; Indicates The first mode and the Adaptive weights of cross-modal collaborative features between modalities; Indicates The first mode The image samples and The first mode Cross-modal collaborative features generated from image samples.

6. The method for intelligent diagnosis of bone tumors based on multimodal imaging based on deep learning according to claim 1, characterized in that: The S4 specifically includes: The individual feature weight matrix is ​​used to weight the patient's individual feature vector to obtain the weighted patient individual feature vector; the fusion feature weight matrix is ​​used to weight the fusion feature vector to obtain the weighted fusion feature vector; and a nonlinear transformation is introduced to generate the probability distribution of the patient's disease category.

7. The method for intelligent diagnosis of bone tumors based on deep learning multimodal imaging according to claim 6, characterized in that: The S4 specifically includes: The calculation formula for the preliminary diagnosis result is: ， in, Represents the preliminary diagnosis result, which is used to reflect the probability of the patient being classified into different disease categories; Indicates the conversion of any real vector into a probability distribution; represents the activation function; represents the individual feature weight matrix; represents the individual feature vector of the patient; Represents the fusion feature weight matrix; represents the fused feature vector; represents the bias term of the personalized diagnosis network; Based on the preliminary diagnosis results, the disease category with the highest probability is selected as the final diagnosis result.

8. A multimodal imaging intelligent diagnosis system for bone tumors based on deep learning, applied to the multimodal imaging intelligent diagnosis method for bone tumors based on deep learning as claimed in claim 1, characterized in that: Includes the following sections: Data acquisition module, adaptive standardization module, multimodal image feature extraction module, patient individual feature extraction module, multimodal image feature fusion module, personalized diagnosis network module; Data acquisition module: acquires the image sample data of different modalities of the patient and the individual information of the patient, outputs the image sample data of different modalities of the patient to the adaptive standardization module, and outputs the individual information of the patient to the individual feature extraction module of the patient; Adaptive standardization module: Based on the image sample data of different modalities of the patient, the image sample data of different modalities are standardized using an adaptive standardization algorithm to obtain standardized image sample data of different modalities, and the standardized image sample data of different modalities are output to the multimodal image feature extraction module; Multimodal image feature extraction module: The standardized image sample data of different modalities are processed through an improved deep convolutional neural network to obtain single-modal features; based on the single-modal features, a dynamic adaptive modality combination method is designed to generate cross-modal collaborative features by introducing interactive features between modalities, and the single-modal features and cross-modal collaborative features are output to the multimodal image feature fusion module; Multimodal image feature fusion module: Based on the single-modal features and cross-modal collaborative features of the multimodal image feature extraction module, the fused feature vector is calculated using the feature fusion algorithm of dynamic weighted nonlinear combination, and the fused feature vector is output to the personalized diagnosis network module; Patient individual feature extraction module: extracts features from the patient's individual information to obtain the patient's individual feature vector, and outputs the patient's individual feature vector to the personalized diagnosis network module; Personalized diagnosis network module: Based on the patient's individual feature vector and fusion feature vector, a personalized diagnosis network is used to generate preliminary diagnosis results, and based on the preliminary diagnosis results, a final diagnosis result is generated.

Images