Disease diagnosis method based on image and clinical data and related device

Abstract

The application provides a disease diagnosis method based on image and clinical data and a related device. The method comprises: performing feature extraction processing on a medical image based on a spatial dimension and a time dimension to obtain a medical image feature, wherein the medical image is a medical image of a specific body organ of a patient; performing feature extraction processing on patient clinical data to obtain a clinical data feature, wherein the patient clinical data comprises clinical data of the patient and / or clinical data of a specific disease patient, and the specific disease patient refers to a patient who has a specific disease in the specific body organ; fusing the image feature and the clinical data feature to obtain a fusion feature, and performing classification based on the fusion feature to obtain a classification result, wherein the classification result is used to indicate whether the patient has the specific disease. The method can improve the disease diagnosis accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a disease diagnosis method based on image and clinical data and a related device. BACKGROUND

[0002] Intelligent disease diagnosis based on artificial intelligence technology is a common application scenario of artificial intelligence technology in the medical field.

[0003] A conventional intelligent disease diagnosis scheme is to process a patient's medical image to determine whether the patient has a disease or to determine the patient's condition. However, the above disease diagnosis scheme often has a diagnosis error, and the diagnosis accuracy is not high. SUMMARY

[0004] To solve the above technical problem, the present application provides a disease diagnosis method based on image and clinical data and a related device, which can improve the accuracy of disease diagnosis.

[0005] The first aspect of the present application provides a disease diagnosis method based on image and clinical data, comprising:

[0006] performing feature extraction processing on a medical image based on spatial and temporal dimensions to obtain a medical image feature, wherein the medical image is a medical image of a specific body organ of a patient;

[0007] performing feature extraction processing on patient clinical data to obtain a clinical data feature, wherein the patient clinical data includes clinical data of the patient and / or clinical data of a specific disease patient, the specific disease patient referring to a patient with a specific disease in the specific body organ;

[0008] fusing the image feature and the clinical data feature to obtain a fusion feature, and performing classification based on the fusion feature to obtain a classification result, the classification result being used to indicate whether the patient has the specific disease.

[0009] In some implementations, the medical image includes a medical image sequence, and the feature extraction processing on the medical image based on the spatial and temporal dimensions to obtain the medical image feature includes:

[0010] performing feature extraction processing on each medical image frame in the medical image sequence based on the spatial dimension to obtain an image feature of each medical image frame;

[0011] arranging the image features of each medical image frame in the order of each medical image frame in the medical image sequence to obtain an image feature sequence;

[0012] The image feature sequence is subjected to time-dimension based feature extraction processing to obtain medical image features.

[0013] In some implementations, each medical image frame in the medical image sequence is subjected to spatial-dimension based feature extraction processing to obtain image features of the medical image frame, including:

[0014] Each medical image frame in the medical image sequence is processed by a convolutional neural network to obtain image features of the medical image frame.

[0015] The convolutional neural network is configured to obtain high-dimensional feature vectors of the image by extracting spatial features at different levels of the image.

[0016] In some implementations, the image feature sequence is subjected to time-dimension based feature extraction processing to obtain medical image features, including:

[0017] The image feature sequence is processed by a long short-term memory network to obtain medical image features.

[0018] The long short-term memory network is configured to capture features that change over time and capture dependencies between features.

[0019] In some implementations, medical image is subjected to spatial-dimension and time-dimension based feature extraction processing to obtain medical image features, patient clinical data is subjected to feature extraction processing to obtain clinical data features, the image features and the clinical data features are fused to obtain fused features, and the medical image is classified based on the fused features to obtain a classification result, including:

[0020] The medical image and the patient clinical data are input into a pre-trained diagnosis model, so that the diagnosis model performs spatial-dimension and time-dimension based feature extraction processing on the medical image to obtain medical image features, performs feature extraction processing on the patient clinical data to obtain clinical data features, fuses the image features and the clinical data features to obtain fused features, and classifies the medical image based on the fused features to obtain a classification result.

[0021] In some implementations, the method further includes:

[0022] Obtaining a diagnosis result corresponding to the medical image.

[0023] Based on the diagnosis result and the classification result, the parameters of the diagnosis model are optimized.

[0024] The second aspect of the present application proposes a disease diagnosis device based on image and clinical data, including:

[0025] a first processing unit configured to perform a feature extraction process on a medical image based on a spatial dimension and a temporal dimension to obtain an image feature, wherein the medical image is a medical image of a specific organ of a patient;

[0026] a second processing unit configured to perform a feature extraction process on patient clinical data to obtain a clinical data feature, wherein the patient clinical data comprises clinical data of the patient and / or clinical data of a specific disease patient, wherein the specific disease patient refers to a patient suffering from a specific disease of the specific organ;

[0027] a classification processing unit configured to fuse the image feature and the clinical data feature to obtain a fused feature, and perform classification based on the fused feature to obtain a classification result, wherein the classification result is used to indicate whether the patient suffers from the specific disease.

[0028] The third aspect of the present application provides a disease diagnosis device, comprising:

[0029] a data acquisition module and a processor connected to the data acquisition module;

[0030] the data acquisition module is configured to acquire a medical image and patient clinical data;

[0031] the processor is configured to execute the disease diagnosis method based on the image and the clinical data.

[0032] The fourth aspect of the present application provides an electronic device, comprising:

[0033] a memory and a processor;

[0034] The memory is connected to the processor and is configured to store a program.

[0035] The processor is configured to execute the disease diagnosis method based on the image and the clinical data by running the program in the memory.

[0036] The fifth aspect of the present application provides a computer program product, comprising computer program instructions, wherein the computer program instructions are configured to make a processor execute the disease diagnosis method based on the image and the clinical data when the computer program instructions are run by the processor.

[0037] The disease diagnosis method based on images and clinical data provided in the embodiments of the present application performs feature extraction processing on the medical images of a patient based on spatial dimensions and time dimensions, so that more detailed and comprehensive image features can be extracted from the medical images. On this basis, the embodiments of the present application further fuse the clinical data of the patient with the medical image features of the patient, and then perform classification of a specific disease based on the fused features. In the fused features, more abundant and detailed data information for diagnosing the specific disease is contained, so that the classification result of the specific disease is more accurate, and the diagnosis accuracy of the specific disease is improved.

[0038] In addition, since the disease diagnosis method based on images and clinical data provided in the embodiments of the present application applies clinical data to disease diagnosis, when the disease diagnosis result (classification result) is obtained, the diagnosis result can be explained according to the clinical data used to obtain the diagnosis result, so that the explainability of the diagnosis result (classification result) is improved. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0040] Figure 1 A flowchart of a disease diagnosis method based on images and clinical data provided in the embodiments of the present application.

[0041] Figure 2 An architecture diagram of a diagnosis model provided in the embodiments of the present application.

[0042] Figure 3 A structure diagram of a disease diagnosis device based on images and clinical data provided in the embodiments of the present application.

[0043] Figure 4 A structure diagram of an electronic device provided in the embodiments of the present application. DETAILED DESCRIPTION

[0044] The technical solutions of the embodiments of the present application are applicable to the application scenario of disease diagnosis based on medical images, and the disease diagnosis accuracy can be improved by using the technical solutions of the embodiments of the present application.

[0045] Disease diagnosis based on medical images is usually performed by image processing technology, that is, the medical images of a specific body organ are processed and analyzed, so as to identify the tissue abnormalities or lesions in the images, and then determine whether the body organ has a lesion and the corresponding disease degree.

[0046] However, due to the low quality of medical images, unreasonable processing of images and other reasons, the accuracy of disease diagnosis based on medical images is not high, and diagnosis errors often occur.

[0047] To solve the above technical problems, the embodiments of the present application propose a disease diagnosis method based on images and clinical data. On the one hand, the processing of medical images is optimized, so that more useful information for disease diagnosis can be extracted from medical images for disease diagnosis. On the other hand, the method combines medical images and clinical data for disease diagnosis, so that more diverse data information is used for disease diagnosis, thereby significantly improving the accuracy of disease diagnosis.

[0048] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0049] The embodiments of the present application propose a disease diagnosis method based on images and clinical data, which can be applied to devices or equipment with automatic diagnosis function, such as automatic diagnosis and treatment equipment, computers, servers, intelligent terminal equipment, etc. For example, the disease diagnosis method based on images and clinical data proposed in the embodiments of the present application can be in the form of software program running on the above devices or equipment, so that the above devices or equipment execute the disease diagnosis method based on images and clinical data proposed in the embodiments of the present application by running the software program. The embodiments of the present application mainly introduce the specific processing process of the disease diagnosis method based on images and clinical data. The application scenarios and implementation manners of the method are not limited.

[0050] In subsequent embodiments, the disease diagnosis method based on images and clinical data proposed in the present application will be introduced by taking liver cell carcinoma diagnosis as an example. Those skilled in the art can understand that the disease diagnosis method based on images and clinical data proposed in the present application is not limited to the diagnosis of a certain disease, but can be applied to the diagnosis of diseases of any body organs or tissues.

[0051] Referring to Figure 1 The disease diagnosis method based on images and clinical data proposed in the embodiments of the present application includes:

[0052] S101, performing feature extraction processing based on spatial dimension and time dimension on the medical image to obtain a medical image feature.

[0053] The medical image is a medical image of a specific organ of a patient, such as a medical image of a liver of a patient in the diagnosis of hepatocellular carcinoma. The medical image can be any type of medical image, such as a B-ultrasound image, a CT image, etc. In addition, the medical image can be one or more medical images. When the medical image is more than one, it can be more than one medical image of the same type or more than one medical image of different types.

[0054] In some embodiments, after obtaining the medical image, the obtained medical image can be first standardized, including adjusting the image size and gray value, and performing data enhancement operations such as rotation, flipping, scaling, etc. Alternatively, the region of interest can be extracted by a segmentation algorithm, such as extracting the liver region, to ensure that consistent and high-quality image data can be used for disease diagnosis and improve the accuracy of subsequent disease diagnosis.

[0055] The above-mentioned feature extraction processing of the medical image based on the spatial dimension refers to feature extraction of the medical image from the spatial dimension, specifically extracting high-dimensional spatial features, converting complex medical images into meaningful feature representations, and retaining important spatial information. Feature extraction of the medical image based on the spatial dimension can capture key information in different spatial positions in the medical image from the spatial dimension, such as image texture, color, shape, edge, etc., and can capture the association between these key information, etc., thereby comprehensively obtaining key information in the medical image.

[0056] For example, the image content at different positions of the medical image is extracted, such as texture features, color features, shape features, etc. of the image content at different positions, and then the image features of the image content at different positions are analyzed and fused for relevance, i.e. feature extraction based on the spatial dimension.

[0057] Alternatively, the medical image can be processed by a neural network with spatial dimension-based feature extraction function to obtain image features extracted based on the spatial dimension. For example, by using a convolutional neural network CNN, a graph neural network GNN, a recurrent neural network RNN, a Transformer model, a residual network ResNet, etc., the medical image can be processed based on the spatial dimension to obtain image features.

[0058] The time-dimension based feature extraction processing of the medical image refers to extracting features of the medical image from the time dimension, so as to capture the changes of the image features over time. The time-dimension based feature extraction processing of the medical image can capture the change characteristics of the key information of the medical image over time, such as the changes of the texture, color, shape, edge, etc. of the image over time, and can combine the image features at different times, so as to comprehensively obtain the key information in the medical image.

[0059] For example, the time-dimension based feature extraction processing of the medical image is implemented by extracting features of the medical images of the same part at different times, and then fusing the image features extracted at different times.

[0060] Alternatively, the medical image can be processed by means of a neural network with a time-dimension based feature extraction function to obtain the image features extracted based on the time dimension. For example, the 3D convolutional neural network 3D-CNN, the long short-term memory network LSTM, the Transformer model, etc. can be used to perform the time-dimension based feature extraction processing of the medical image to obtain the image features.

[0061] In some embodiments, the time-dimension and space-dimension based feature extraction processing of the medical image can be performed by first performing the space-dimension based feature extraction processing of the medical image, and then performing the time-dimension based feature extraction processing of the extracted features to obtain the final medical image features. Alternatively, the time-dimension based feature extraction processing of the medical image can be performed first, and then the space-dimension based feature extraction processing of the extracted features can be performed to obtain the final medical image features. Alternatively, the space-dimension based feature extraction processing and the time-dimension based feature extraction processing of the medical image can be performed simultaneously, and then the extracted space-dimension features and time-dimension features can be fused to obtain the final medical image features.

[0062] S102, performing feature extraction processing on the patient clinical data to obtain clinical data features.

[0063] The patient clinical data includes the clinical data of the patient, and / or the clinical data of a patient with a specific disease, wherein the patient with a specific disease refers to a patient with a specific disease in a specific body organ.

[0064] Specifically, through the HIS (Hospital Information System), EMR (Electronic Medical Record), LIS (Laboratory Information System) and other systems of a medical institution, the clinical data of a patient corresponding to the medical image can be obtained, and / or the clinical data of a patient with a specific disease can be obtained.

[0065] The patient with a specific disease refers to a patient whose specific body organ suffers from a specific disease. For example, when the medical image is a liver medical image of a patient A, the patient with a specific disease can be a patient B who is diagnosed with hepatocellular carcinoma.

[0066] Since the clinical data of a patient can include various data, and some data are useless for diagnosing a specific disease, in some other embodiments, the clinical data can be specifically the clinical data of a patient corresponding to a specific examination item. The specific examination item is a clinical examination item for judging a specific disease.

[0067] For example, assuming that the technical solutions of the embodiments of the present application are used for diagnosing hepatocellular carcinoma, when a patient suffers from hepatocellular carcinoma, the patient usually has manifestations in some physical symptoms, signs, and laboratory examination items.

[0068] Among the physical symptoms, early hepatocellular carcinoma can have no obvious symptoms, and is often found in physical examination or accidental medical image examination. When the tumor grows, the patient can feel discomfort or pain in the right upper abdomen, especially when the tumor is located on the surface or edge of the liver. In the late stage of liver cancer, fatigue and weight loss are common. Loss of appetite, nausea and vomiting are related to the affected digestive system.

[0069] Among the signs, a patient with hepatocellular carcinoma usually shows symptoms such as abdominal distension, hepatomegaly, jaundice, ascites, and splenomegaly.

[0070] In the laboratory examination items, the clinical detection data of the liver function, tumor markers, hepatitis B and C virus detection and other items of a patient with hepatocellular carcinoma are abnormal. For example, in the liver function examination, the levels of AST (aspartate aminotransferase) and ALT (alanine aminotransferase) are elevated, and the bilirubin is elevated. In the tumor marker examination, the elevated AFP (alpha-fetoprotein) level is a typical manifestation of hepatocellular carcinoma, but not all patients with hepatocellular carcinoma have elevated AFP. In the hepatitis B and C virus detection item, many cases of hepatocellular carcinoma are related to hepatitis B and C virus infection.

[0071] Therefore, the above-mentioned somatic symptoms, signs, and data of laboratory experience detection items are useful for diagnosing hepatocellular carcinoma. When obtaining the clinical data of the patient, the clinical data of the above-mentioned somatic symptoms, signs, and laboratory experience detection items can be obtained. Other clinical data that is not useful for diagnosing hepatocellular carcinoma can not be obtained, thereby avoiding redundancy and useless clinical data affecting the efficiency and accuracy of the diagnosis of hepatocellular carcinoma.

[0072] After obtaining the above-mentioned patient clinical data, the obtained patient clinical data is processed by feature engineering to realize the feature of the patient clinical data, and obtain the clinical data feature.

[0073] S103, the image features and the clinical data features are fused to obtain fusion features, and classification is performed based on the fusion features to obtain a classification result.

[0074] Specifically, after obtaining the medical image features and the clinical data features by performing steps S101 and S102, the two kinds of features are fused, such as splicing the medical image features and the clinical data features, to obtain fusion features.

[0075] Then, classification processing is performed based on the fusion features, specifically, the probability of the patient having a specific type of disease and not having a specific type of disease is predicted, and a classification result is obtained. The classification result is used to indicate whether the patient corresponding to the above-mentioned medical image has the above-mentioned specific disease. For example, the liver image features of the patient and the clinical data features of the patient are fused, and then classification of having hepatocellular carcinoma and not having hepatocellular carcinoma is performed based on the fusion features to obtain a classification result, which is used to indicate whether the patient has hepatocellular carcinoma.

[0076] In some embodiments, in the case where the above-mentioned patient clinical data only includes the clinical data of the patient, the fusion features obtained by the above-mentioned feature fusion processing not only include the medical image features of the patient, but also include the clinical data features of the patient, that is, the fusion features include more rich reference information for diagnosing a specific disease, thereby having higher diagnosis accuracy for diagnosing the specific disease only by the medical image features of the patient.

[0077] In the case where the above-mentioned patient clinical data only includes the clinical data of the patient with a specific disease, the fusion features obtained by the above-mentioned feature fusion processing not only include the medical image features of the patient, but also include the clinical data information of the patient with the specific disease, so that when classification is performed based on the fusion features, there is a clear diagnosis standard or diagnosis example of the specific disease to determine whether the patient has the specific disease, thereby being able to improve the diagnosis accuracy of the specific disease.

[0078] In the case of the patient clinical data including the patient's clinical data and the clinical data of the patient with the specific disease, the fusion feature obtained through the above feature fusion processing not only contains the medical image feature of the patient, but also contains the clinical data feature of the patient and the clinical data of the patient with the specific disease, so as to enrich the reference information for diagnosing whether the patient has the specific disease, and to carry the diagnosis standard or diagnosis example of the specific disease in the fusion feature, so as to more accurately judge whether the patient has the specific disease based on the fusion feature.

[0079] As can be seen from the above, the disease diagnosis method based on image and clinical data proposed in the embodiments of the present application performs feature extraction processing on the medical image of the patient based on the spatial dimension and the time dimension, so as to extract more detailed and comprehensive image features from the medical image. On this basis, the embodiments of the present application further fuse the patient clinical data with the patient medical image feature, and then classify the specific disease based on the fusion feature. In the above fusion feature, more abundant and more detailed data information for diagnosing the specific disease is contained, so as to make the classification result of the specific disease more accurate, and improve the diagnosis accuracy of the specific disease.

[0080] In addition, since the disease diagnosis method based on image and clinical data proposed in the embodiments of the present application applies clinical data to disease diagnosis, when the disease diagnosis result (classification result) is obtained, the diagnosis result can be explained according to the clinical data used to obtain the diagnosis result, so as to improve the explainability of the diagnosis result (classification result).

[0081] In another embodiment, the above medical image includes a medical image sequence of the patient, such as a CT image, a magnetic resonance MRI image, and the like. The single image of these images contains more clear and rich image information, and the image sequence further contains more rich image information.

[0082] For example, the CT or MRI device is used to scan the patient's body image, the scanning sequence includes a non-contrast agent enhancement period (to provide a baseline image), an arterial phase (to scan after 20-40 seconds to capture the hepatic arterial blood flow), a portal vein phase (to scan after 60-70 seconds to display the portal vein and its branches), a delay phase or a balance phase (to scan after 3-5 minutes to evaluate the contrast agent elution), to obtain CT or MRI images at different time points, and form time sequence data.

[0083] For the above medical image sequence, the image features can be extracted through the following steps A1-A3:

[0084] A1, respectively perform spatial dimension-based feature extraction processing on each medical image frame in the medical image sequence to obtain image features of each medical image frame.

[0085] Specifically, for each image frame in the medical image sequence, spatial dimension-based feature extraction processing is performed respectively. The image feature extraction can be performed by using the spatial feature extraction manner introduced in the above embodiments, to obtain the image features of each medical image frame.

[0086] As a preferred implementation, the embodiments of the present application perform processing on each medical image frame in the medical image sequence by using a convolutional neural network, to obtain the image features of each medical image frame.

[0087] Specifically, the convolutional neural network is used to obtain a high-dimensional feature vector of an image by extracting spatial features at different levels of the image.

[0088] After the medical image sequence is subjected to standardization processing such as size adjustment and grayscale value normalization, the medical image frames in the medical image sequence are input into a pre-trained convolutional neural network ResNet, so that the convolutional neural network performs convolution operation on the medical image frames to extract spatial features at different levels. These convolutional layers can detect image features at different scales and complexities, such as edges, textures and shapes, etc.

[0089] Then, after convolution and pooling operations, multi-level feature maps are obtained, which represent the spatial features of the image at different levels.

[0090] Finally, the extracted features are generated into a high-dimensional feature vector by a fully connected layer, i.e., the image features of the medical image frame are obtained.

[0091] For each medical image frame in the medical image sequence, the image features thereof can be obtained by using the above convolutional neural network.

[0092] A2、According to the order of each medical image frame in the medical image sequence, the image features of each medical image frame are arranged to obtain an image feature sequence.

[0093] Specifically, according to the order relationship of each medical image frame in the medical image sequence, the image features of each medical image frame are arranged to obtain an image feature sequence.

[0094] A3、The image feature sequence is subjected to time dimension-based feature extraction processing to obtain medical image features.

[0095] Specifically, the image feature sequence is subjected to feature extraction processing based on a time dimension, that is, each image feature frame in the image feature sequence is subjected to context-related feature extraction processing, to obtain medical image features corresponding to the image feature sequence. The image feature extraction can be performed by using the time feature extraction manner introduced in the above embodiments, to obtain the medical image features.

[0096] As a preferred implementation, the embodiments of the present application process the image feature sequence by using a long short-term memory network (LSTM) to obtain medical image features.

[0097] The LSTM network is used to capture features changing over time and capture the dependency between features, so that more useful feature components can be extracted from the features of medical image frames at different times.

[0098] The image feature sequence is input into the LSTM network, and the LSTM network analyzes the image feature sequence by using its memory cells and gate mechanism, captures the long-term dependency between features, extracts important patterns and trends changing over time, and obtains the final medical image features.

[0099] In some other embodiments, the disease diagnosis method based on image and clinical data proposed by the embodiments of the present application can be implemented by training a diagnosis model.

[0100] The diagnosis model can include the convolutional neural network ResNet and the LSTM network described above, Figure 2 An architecture schematic diagram of the diagnosis model is shown.

[0101] Referring to Figure 2 As shown, after obtaining the medical image of the patient and the clinical data of the patient, the medical image and the clinical data of the patient are input into the diagnosis model.

[0102] The convolutional neural network in the diagnosis model first performs feature extraction processing based on a spatial dimension on the medical image frames, to obtain image features of the medical image frames. Then, the LSTM network performs feature extraction processing based on a time dimension on an image feature sequence constituted by the image features of the medical image frames, to obtain medical image features.

[0103] On the other hand, the featureization processing module in the diagnosis model performs featureization processing on the clinical data of the patient, to obtain clinical data features.

[0104] Then, the medical image features output by the LSTM network and the clinical data features are fused by using a fully connected layer, to obtain fused features.

[0105] After the feature fusion, one or more fully connected layers are connected to perform further feature processing and conversion, which can be used to reduce dimension and perform complex nonlinear transformation to extract key information for classification.

[0106] After the last fully connected layer, a softmax layer is used for classification prediction, outputting the predicted probability for each class (having a specific disease and not having a specific disease), i.e., obtaining the probability distribution of each class, where the probability of each class represents the prediction confidence of the model for the class. The class with the highest probability is selected as the final classification result.

[0107] When training the above-mentioned diagnostic model, the patient medical image and the patient clinical data are used as training samples, and the patient diagnosis result is used as a label. The diagnostic model is trained for disease diagnosis by using a loss function such as cross-entropy, and the model parameters are optimized until the model diagnosis accuracy meets the requirements.

[0108] In actual application of the above-mentioned diagnostic model for disease diagnosis, the diagnostic model can be used as an aid for doctors to diagnose diseases, and the diagnostic model is continuously optimized during application.

[0109] For example, after inputting the medical image and the patient clinical data into the above-mentioned diagnostic model to obtain the classification result, the diagnosis result corresponding to the above-mentioned medical image is obtained, which can be the diagnosis result obtained by a doctor diagnosing the patient corresponding to the above-mentioned medical image.

[0110] Then, based on the classification result output by the diagnostic model and the above-mentioned diagnosis result, the parameters of the diagnostic model are optimized.

[0111] Specifically, the classification result output by the diagnostic model and the above-mentioned diagnosis result are compared to determine whether they are consistent. If they are consistent, it means that the classification result output by the diagnostic model is correct, and the diagnostic model can not be optimized at this time. If they are not consistent, it means that the classification result output by the diagnostic model is incorrect, and the parameters of the diagnostic model can be optimized by calculating the loss function of the classification result output by the diagnostic model and the above-mentioned diagnosis result.

[0112] Through the above-mentioned model optimization operation, the above-mentioned diagnostic model can be continuously optimized during application, so that the diagnosis accuracy of the diagnostic model is higher and higher.

[0113] Corresponding to the above-mentioned method, the embodiments of the present application also provide a disease diagnosis device based on image and clinical data, as shown in Figure 3 The device comprises:

[0114] The first processing unit 100 is configured to perform spatial dimension and time dimension based feature extraction processing on the medical image to obtain medical image features, wherein the medical image is a medical image of a specific body organ of a patient.

[0115] The second processing unit 110 is configured to perform feature extraction processing on the patient clinical data to obtain clinical data features, wherein the patient clinical data includes clinical data of the patient and / or clinical data of a specific disease patient, wherein the specific disease patient refers to a patient suffering from a specific disease of the specific body organ.

[0116] The classification processing unit 120 is configured to fuse the image features and the clinical data features to obtain fused features, and perform classification based on the fused features to obtain a classification result, wherein the classification result is used to indicate whether the patient suffers from the specific disease.

[0117] In some implementations, the medical image includes a medical image sequence, and the spatial dimension and time dimension based feature extraction processing on the medical image to obtain medical image features includes:

[0118] performing spatial dimension based feature extraction processing on each medical image frame in the medical image sequence to obtain image features of each medical image frame;

[0119] arranging the image features of each medical image frame in a sequence according to the order of each medical image frame in the medical image sequence to obtain an image feature sequence;

[0120] performing time dimension based feature extraction processing on the image feature sequence to obtain the medical image features.

[0121] In some implementations, the spatial dimension based feature extraction processing on each medical image frame in the medical image sequence to obtain image features of each medical image frame includes:

[0122] processing each medical image frame in the medical image sequence by a convolutional neural network to obtain image features of each medical image frame;

[0123] wherein the convolutional neural network is configured to obtain a high-dimensional feature vector of an image by extracting spatial features of different levels of the image.

[0124] In some implementations, the time dimension based feature extraction processing on the image feature sequence to obtain the medical image features includes:

[0125] processing the image feature sequence by a long short-term memory network to obtain the medical image features;

[0126] The long short-term memory network is used to capture features changing over time and capture dependencies between features.

[0127] In some embodiments, another image and clinical data based disease diagnosis device is also proposed, which is used for performing spatial dimension and time dimension based feature extraction processing on a medical image to obtain image features, wherein the medical image is a medical image of a specific body organ of a patient; performing feature extraction processing on patient clinical data to obtain clinical data features, wherein the patient clinical data includes clinical data of the patient and / or clinical data of a specific disease patient, and the specific disease patient refers to a patient with a specific disease in the specific body organ; fusing the image features and the clinical data features to obtain fused features, and performing classification based on the fused features to obtain a classification result, wherein the classification result is used to indicate whether the patient has the specific disease.

[0128] In some implementations, the device runs a pre-trained diagnosis model, and the device performs spatial dimension and time dimension based feature extraction processing on a medical image to obtain image features, performs feature extraction processing on patient clinical data to obtain clinical data features, and fuses the image features and the clinical data features to obtain fused features, and performs classification on the medical image based on the fused features to obtain a classification result, including:

[0129] The medical image and the patient clinical data are input into the pre-trained diagnosis model, so that the diagnosis model performs spatial dimension and time dimension based feature extraction processing on the medical image to obtain image features, performs feature extraction processing on the patient clinical data to obtain clinical data features, fuses the image features and the clinical data features to obtain fused features, and performs classification on the medical image based on the fused features to obtain a classification result.

[0130] In some implementations, the device is also used for:

[0131] Obtaining a diagnosis result corresponding to the medical image;

[0132] Based on the diagnosis result and the classification result, performing parameter optimization on the diagnosis model.

[0133] The disease diagnosis apparatus based on image and clinical data provided by the embodiment is of the same application concept as the disease diagnosis method based on image and clinical data provided by the above embodiments of the application, can execute the disease diagnosis method based on image and clinical data provided by any of the above embodiments of the application, and has the corresponding function modules and beneficial effects of the execution method. The technical details not described in detail in the embodiment can be found in the specific processing content of the disease diagnosis method based on image and clinical data provided by the above embodiments of the application, which will not be described here.

[0134] The functions implemented by the first processing unit 100, the second processing unit 110 and the classification processing unit 120 described above can be implemented by the same or different processors, and the embodiments of the application are not limited.

[0135] It should be understood that the units in the above apparatus can be implemented in the form of processor calling software. For example, the apparatus includes a processor connected with a memory, the memory stores instructions, and the processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of the units of the apparatus. The processor can be a general processor, such as a CPU or a microprocessor, and the memory can be an internal memory of the apparatus or an external memory of the apparatus. Alternatively, the units in the apparatus can be implemented in the form of hardware circuit. The functions of some or all of the units can be implemented by designing the hardware circuit. The hardware circuit can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the units are implemented by designing the logical relationship of the elements in the circuit. For another example, in another implementation, the hardware circuit can be implemented by a PLD. Taking an FPGA as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to implement the functions of some or all of the units. All the units of the above apparatus can be implemented in the form of processor calling software, or all the units can be implemented in the form of hardware circuit, or part of the units can be implemented in the form of processor calling software, and the remaining part can be implemented in the form of hardware circuit.

[0136] In the embodiments of the present application, the processor is a circuit with signal processing capability. In one implementation, the processor can be a circuit with instruction reading and running capability, such as a CPU, a microprocessor, a GPU, or a DSP, etc. In another implementation, the processor can implement certain functions through a logic relationship of a hardware circuit, which is fixed or can be reconfigured. For example, the processor is a hardware circuit implemented by an ASIC or a PLD, such as an FPGA, etc. In the reconfigurable hardware circuit, the processor loads a configuration document to implement the hardware circuit configuration. It can be understood that the processor loads instructions to implement the functions of the above units.

[0137] It can be seen that each unit in the above apparatus can be one or more processors (or processing circuits) configured to implement the above methods, such as a CPU, a GPU, an NPU, a TPU, a DPU, a microprocessor, a DSP, an ASIC, an FPGA, or a combination of at least two of these processor forms.

[0138] In addition, each unit in the above apparatus can be integrated together or can be independently implemented. In one implementation, these units are integrated together to implement a SOC. The SOC can include at least one processor for implementing any of the above methods or the functions of the units of the apparatus. The at least one processor can be different, such as including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0139] The embodiments of the present application also propose a control apparatus, which includes a processor and an interface circuit. The processor in the control apparatus is connected with an input-output component through the interface circuit of the control apparatus.

[0140] The input-output component specifically refers to a functional component capable of acquiring or collecting medical images and patient clinical data.

[0141] The above interface circuit can be any interface circuit capable of realizing data communication functions, such as a USB interface circuit, a Type-C interface circuit, a serial port circuit, a PCIE circuit, etc.

[0142] The processor in the control apparatus is also a circuit with signal processing capability. It processes the medical images and patient clinical data acquired by the input-output component to obtain a disease diagnosis result by executing any of the disease diagnosis methods based on images and clinical data introduced in the above embodiments. The specific implementation of the processor can refer to the above processor implementation, and the embodiments of the present application are not strictly limited.

[0143] The embodiment of the present application provides a disease diagnosis device, which comprises a data acquisition module and a processor connected with the data acquisition module.

[0144] The data acquisition module is used for acquiring medical images and patient clinical data, which can be image acquisition devices used for collecting the medical images and data acquisition devices used for collecting the patient clinical data, or the data acquisition module can also be a data receiving port used for receiving the medical images and the patient clinical data uploaded by a user.

[0145] The processor is used for processing the medical images and the patient clinical data acquired by the data acquisition module to obtain a disease diagnosis result by executing any one of the disease diagnosis methods based on the images and the clinical data in any one of the above embodiments.

[0146] The specific processing process of the processor can be referred to the description of the method embodiments, and the specific implementation mode of the processor can also be referred to the description of the above embodiments.

[0147] The disease diagnosis device can be a terminal device with a disease diagnosis function, for example, a self-service inquiry terminal, a server, a computer, an intelligent terminal and the like.

[0148] Another embodiment of the present application further provides an electronic device, as shown in the Figure 4 The device comprises:

[0149] a memory 200 and a processor 210.

[0150] The memory 200 is connected with the processor 210 and is used for storing programs.

[0151] The processor 210 is used for realizing the disease diagnosis method based on the images and the clinical data disclosed in any one of the above embodiments by running the programs stored in the memory 200.

[0152] Specifically, the electronic device can further comprise a bus, a communication interface 220, an input device 230 and an output device 240.

[0153] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected with each other through the bus.

[0154] The bus can comprise a channel for transmitting information between various components of the computer system.

[0155] The processor 210 can be a general processor, such as a general central processing unit (CPU), a microprocessor, or the like, or can be an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of programs of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a ready-to-use programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0156] The processor 210 can include a main processor and can also include a baseband chip, a modem, etc.

[0157] The memory 200 stores programs for executing the technical solutions of the present application, and can also store operating systems and other key services. Specifically, the programs can include program codes, and the program codes include computer operation instructions. More specifically, the memory 200 can include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk memory, a flash, and the like.

[0158] The input device 230 can include a device that receives data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor, etc.

[0159] The output device 240 can include a device that allows information to be output to a user, such as a display screen, a printer, a speaker, etc.

[0160] The communication interface 220 can include a device using any transceiver, such as an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc., to communicate with other devices or communication networks.

[0161] The processor 210 executes the programs stored in the memory 200 and calls other devices, which can be used to implement each step of any of the disease diagnosis methods based on image and clinical data provided by the above-mentioned embodiments of the present application.

[0162] The embodiments of the present application also propose a chip including a processor and a data interface, the processor reading and running programs stored on the memory through the data interface to execute the disease diagnosis method based on image and clinical data introduced in any of the above-mentioned embodiments. The specific processing process and its beneficial effects can be referred to the above-mentioned embodiments of the disease diagnosis method based on image and clinical data.

[0163] In addition to the methods and devices described above, embodiments of the present application can also be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform steps of the disease diagnosis method based on image and clinical data described in any of the embodiments of the present application.

[0164] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, etc., and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0165] In addition, embodiments of the present application can also be a storage medium having a computer program stored thereon, and the computer program is executed by a processor to perform steps of the disease diagnosis method based on image and clinical data described in any of the embodiments of the present application.

[0166] For each of the foregoing method embodiments, in order to simply describe, it is expressed as a combination of a series of actions, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0167] It should be noted that each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same and similar parts between each embodiment can be referred to each other. For device embodiments, since they are basically similar to method embodiments, they are described more simply, and the relevant parts refer to the part of the description of the method embodiments.

[0168] The steps in the method of each embodiment of the present application can be adjusted, combined and deleted in sequence according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0169] The modules and sub-modules in the device and terminal of each embodiment of the present application can be combined, divided and deleted according to actual needs.

[0170] It should be understood that the disclosed terminal, device and method can be implemented in other ways. For example, the terminal embodiments described above are merely illustrative. For example, the division of modules or sub-modules is merely a logical function division. In actual implementation, another division manner can be used. For example, a plurality of sub-modules or modules can be combined or integrated into another module, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed modules can be indirect coupling or communication connection through some interfaces, devices or modules, and can be electrical, mechanical or other forms.

[0171] The modules or sub-modules described as separate components can or can not be physically separate, and the components of the modules or sub-modules can or can not be physical modules or sub-modules, i.e. can be located in one place or distributed on a plurality of network modules or sub-modules. Some or all of the modules or sub-modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0172] In addition, the functional modules or sub-modules in each embodiment of the present application can be integrated into a processing module, or each module or sub-module can exist physically, or two or more modules or sub-modules can be integrated into one module. The integrated module or sub-module can be realized in the form of hardware or software functional module or sub-module.

[0173] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0174] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein can be directly implemented by hardware, software units executed by a processor, or a combination of both. The software units can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0175] Finally, it should be noted that, in this document, the term "only" is used simply to set off from one entity or action to another in order to avoid the use of the term "and / or" or the like for the sake of clarity. In no way should the term "only" be interpreted as implying that there is an implied exclusion of any referenced entity or action. Moreover, the terms "comprising", "including", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0176] The above description of disclosed embodiments provides enabling teaching for making or using the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A disease diagnosis method based on image and clinical data, characterized by, The method comprises: performing spatial dimension and time dimension based feature extraction processing on a medical image to obtain medical image features, wherein the medical image is a medical image of a specific body organ of a patient; the medical image comprises a medical image sequence obtained by scanning the patient at different time points in a single examination process; performing feature extraction processing on patient clinical data to obtain clinical data features, wherein the patient clinical data comprises clinical data of the patient and clinical data of a specific disease patient, and the specific disease patient refers to a patient suffering from a specific disease in the specific body organ; fusing the medical image features and the clinical data features to obtain fused features, and performing classification based on the fused features to obtain a classification result, wherein the classification result is used to indicate whether the patient suffers from the specific disease.

2. The method of claim 1, wherein, The method of performing spatial dimension and time dimension based feature extraction processing on a medical image to obtain medical image features comprises: performing spatial dimension based feature extraction processing on each medical image frame in the medical image sequence to obtain image features of each medical image frame; arranging the image features of each medical image frame in sequence according to the order of each medical image frame in the medical image sequence to obtain an image feature sequence; performing time dimension based feature extraction processing on the image feature sequence to obtain medical image features.

3. The method of claim 2, wherein, The method of performing spatial dimension based feature extraction processing on each medical image frame in the medical image sequence to obtain image features of each medical image frame comprises: processing each medical image frame in the medical image sequence by using a convolutional neural network to obtain image features of each medical image frame; wherein the convolutional neural network is used to obtain a high-dimensional feature vector of an image by extracting spatial features of different levels of the image.

4. The method of claim 2, wherein, The method of performing time dimension based feature extraction processing on the image feature sequence to obtain medical image features comprises: processing the image feature sequence by using a long short-term memory network to obtain medical image features; wherein the long short-term memory network is used to capture time-varying features and capture the dependency between features.

5. The method according to any one of claims 1 to 4, characterized in that, The method of performing spatial dimension and time dimension based feature extraction processing on a medical image to obtain medical image features, performing feature extraction processing on patient clinical data to obtain clinical data features, and fusing the medical image features and the clinical data features to obtain fused features, and performing classification on the medical image based on the fused features to obtain a classification result comprises: inputting the medical image and the patient clinical data into a pre-trained diagnosis model, so that the diagnosis model performs spatial dimension and time dimension based feature extraction processing on the medical image to obtain medical image features, performs feature extraction processing on the patient clinical data to obtain clinical data features, fuses the medical image features and the clinical data features to obtain fused features, and performs classification on the medical image based on the fused features to obtain a classification result.

6. The method of claim 5, wherein, The method further comprises: obtaining a diagnosis result corresponding to the medical image; Based on the diagnosis result and the classification result, parameters of the diagnosis model are optimized.

7. A disease diagnosis apparatus based on image and clinical data, characterized by, The method comprises the steps of: The first processing unit is configured to perform spatial dimension and time dimension based feature extraction processing on the medical image to obtain medical image features, wherein the medical image is a medical image of a specific body organ of a patient; and the medical image comprises a medical image sequence obtained by scanning the patient at different time points in a single examination process. The second processing unit is configured to perform feature extraction processing on patient clinical data to obtain clinical data features, wherein the patient clinical data comprises clinical data of the patient and clinical data of a specific disease patient, and the specific disease patient refers to a patient who has a specific disease in the specific body organ. The classification processing unit is configured to fuse the medical image features and the clinical data features to obtain fused features, and perform classification based on the fused features to obtain a classification result, wherein the classification result is used to indicate whether the patient has the specific disease.

8. A disease diagnostic device characterized by comprising: The method comprises the steps of: The data acquisition module is configured to acquire medical images and patient clinical data. The processor is configured to execute the disease diagnosis method based on images and clinical data according to any one of claims 1 to 6. The method comprises the steps of:

9. An electronic device, comprising: The memory and the processor are connected, and the memory is used to store programs. The processor is configured to execute the disease diagnosis method based on images and clinical data according to any one of claims 1 to 6 by running the programs in the memory. The computer program instructions are configured to enable the processor to execute the disease diagnosis method based on images and clinical data according to any one of claims 1 to 6 when the computer program instructions are run by the processor. ​ 10. A computer program product, characterised in that, ​

Images