Disease diagnosis method based on images and clinical data and related device

By combining feature extraction and fusion classification of medical images and clinical data in intelligent disease diagnosis, the problem of low diagnostic accuracy in the prior art is solved, and a more accurate and reliable disease diagnosis is achieved.

CN118983077AActive Publication Date: 2024-11-19ANHUI IFLYHEALTH CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202411024413.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-19
Estimated Expiration
2044-07-29

AI Technical Summary

Technical Problem

Existing intelligent diagnosis methods for diseases often have diagnostic errors in medical image processing, the diagnostic accuracy is not high, and the clinical data cannot be effectively utilized, resulting in the diagnostic results being unreliable enough.

Method used

The medical image is processed using feature extraction methods based on spatial dimensions and time dimensions. Combining the patient's clinical data, image features and clinical data characteristics are extracted through convolutional neural networks and long and short-term memory networks, and fusion classification is carried out to improve diagnostic accuracy. .

Benefits of technology

By fusing the characteristics of medical images and clinical data, the accuracy of disease diagnosis is significantly improved, and the interpretability of diagnostic results is enhanced, providing more comprehensive diagnostic information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118983077A_ABST
    Figure CN118983077A_ABST
Patent Text Reader

Abstract

The invention provides a disease diagnosis method based on an image and clinical data and a related device, and the method comprises the steps: carrying out the feature extraction processing of a medical image based on a spatial dimension and a time dimension, and obtaining the features of the medical image, the medical image being a medical image of a specific body organ of a patient; feature extraction processing is conducted on the clinical data of the patient, clinical data features are obtained, the clinical data of the patient comprise the clinical data of the patient and / or the clinical data of a patient with a specific disease, and the patient with the specific disease refers to the patient with the specific body organ suffering from the specific disease; the image features and the clinical data features are fused to obtain fused features, classification is carried out based on the fused features to obtain a classification result, and the classification result is used for representing whether the patient suffers from the specific disease or not. The method can improve the disease diagnosis accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a disease diagnosis method and related devices based on images and clinical data. Background Art

[0002] Intelligent disease diagnosis based on artificial intelligence technology is a common application scenario of artificial intelligence technology in the medical field.

[0003] Conventional intelligent disease diagnosis solutions process medical images of patients to determine whether the patient has a disease or the patient's condition. However, the above disease diagnosis solutions often make mistakes in diagnosis and the diagnosis accuracy is not high. Summary of the invention

[0004] Based on the above technical problems, the present application proposes a disease diagnosis method and related devices based on images and clinical data, which can improve the accuracy of disease diagnosis.

[0005] The first aspect of the present application proposes a disease diagnosis method based on images and clinical data, comprising:

[0006] Performing feature extraction processing on the medical image based on the spatial dimension and the temporal dimension to obtain medical image features, wherein the medical image is a medical image of a specific body organ of a patient;

[0007] Performing feature extraction processing on the patient's clinical data to obtain clinical data features, wherein the patient's clinical data includes the patient's clinical data and / or clinical data of patients with specific diseases, wherein the patients with specific diseases refer to patients with the specific body organ suffering from the specific disease;

[0008] The image features and the clinical data features are fused to obtain fused features, and classification is performed based on the fused features to obtain classification results, wherein the classification results are used to indicate whether the patient suffers from the specific disease.

[0009] In some implementations, the medical image includes a medical image sequence, and the step of performing feature extraction processing on the medical image based on a spatial dimension and a temporal dimension to obtain medical image features includes:

[0010] Performing feature extraction processing based on spatial dimensions on each medical image frame in the medical image sequence to obtain image features of each medical image frame;

[0011] Arranging image features of each medical image frame according to the order of each medical image frame in the medical image sequence to obtain an image feature sequence;

[0012] The image feature sequence is subjected to feature extraction processing based on the time dimension to obtain medical image features.

[0013] In some implementations, each medical image frame in the medical image sequence is subjected to feature extraction processing based on the spatial dimension to obtain image features of each medical image frame, including:

[0014] Each medical image frame in the medical image sequence is processed separately by a convolutional neural network to obtain image features of each medical image frame;

[0015] The convolutional neural network is used to obtain a high-dimensional feature vector of an image by extracting spatial features at different levels of the image.

[0016] In some implementations, performing feature extraction processing based on the time dimension on the image feature sequence to obtain medical image features includes:

[0017] Processing the image feature sequence through a long short-term memory network to obtain medical image features;

[0018] The long short-term memory network is used to capture features that change over time and to capture dependencies between features.

[0019] In some implementations, performing feature extraction processing based on spatial dimensions and time dimensions on a medical image to obtain medical image features, performing feature extraction processing on patient clinical data to obtain clinical data features, and fusing the image features and the clinical data features to obtain fused features, and classifying the medical image based on the fused features to obtain a classification result, including:

[0020] The medical image and the patient's clinical data are input into a pre-trained diagnostic model, so that the diagnostic model performs feature extraction processing on the medical image based on the spatial dimension and the temporal dimension to obtain medical image features, performs feature extraction processing on the patient's clinical data to obtain clinical data features, and fuses the image features and the clinical data features to obtain fused features, and classifies the medical image based on the fused features to obtain a classification result.

[0021] In some implementations, the method further includes:

[0022] Obtaining a diagnosis result corresponding to the medical image;

[0023] Based on the diagnosis result and the classification result, parameters of the diagnosis model are optimized.

[0024] The second aspect of the present application provides a disease diagnosis device based on images and clinical data, comprising:

[0025] A first processing unit is used to perform feature extraction processing on the medical image based on the spatial dimension and the temporal dimension to obtain medical image features, wherein the medical image is a medical image of a specific body organ of a patient;

[0026] A second processing unit is used to perform feature extraction processing on the patient clinical data to obtain clinical data features, wherein the patient clinical data includes the clinical data of the patient and / or clinical data of patients with specific diseases, and the patients with specific diseases refer to patients with the specific body organ suffering from the specific disease;

[0027] The classification processing unit is used to fuse the image features and the clinical data features to obtain fused features, and to perform classification based on the fused features to obtain classification results, wherein the classification results are used to indicate whether the patient suffers from the specific disease.

[0028] The third aspect of the present application provides a disease diagnosis device, comprising:

[0029] A data acquisition module, and a processor connected to the data acquisition module;

[0030] The data acquisition module is used to acquire medical images and patient clinical data;

[0031] The processor is used to execute the above-mentioned disease diagnosis method based on images and clinical data.

[0032] A fourth aspect of the present application provides an electronic device, comprising:

[0033] Memory and processor;

[0034] Wherein, the memory is connected to the processor and is used to store programs;

[0035] The processor is used to implement the above-mentioned disease diagnosis method based on images and clinical data by running the program in the memory.

[0036] A fifth aspect of the present application proposes a computer program product, comprising computer program instructions, which, when executed by a processor, enable the processor to execute the above-mentioned disease diagnosis method based on images and clinical data.

[0037] The disease diagnosis method based on images and clinical data proposed in the embodiment of the present application performs feature extraction processing on the patient's medical image based on the spatial dimension and the temporal dimension, so that more detailed and comprehensive image features can be extracted from the medical image. On this basis, the embodiment of the present application also fuses the patient's clinical data with the patient's medical image features, and then classifies specific diseases based on the fused features. The above-mentioned fused features contain richer and more detailed data information for diagnosing specific diseases, so that the classification results for specific diseases are more accurate and the diagnostic accuracy of specific diseases is improved.

[0038] In addition, since the disease diagnosis method based on images and clinical data proposed in the embodiment of the present application applies clinical data to disease diagnosis, when the disease diagnosis result (classification result) is obtained, the diagnosis result can be interpreted based on the clinical data used to obtain the diagnosis result, thereby improving the interpretability of the diagnosis result (classification result). BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0040] Figure 1 A flowchart of a disease diagnosis method based on images and clinical data provided in an embodiment of the present application.

[0041] Figure 2 A schematic diagram of the architecture of a diagnostic model provided in an embodiment of the present application.

[0042] Figure 3 A schematic diagram of the structure of a disease diagnosis device based on images and clinical data provided in an embodiment of the present application.

[0043] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The technical solution of the embodiment of the present application is applicable to disease diagnosis application scenarios based on medical images. The technical solution of the embodiment of the present application can improve the accuracy of disease diagnosis.

[0045] Disease diagnosis based on medical images usually involves processing and analyzing medical images of specific body organs through image processing technology, thereby identifying tissue abnormalities or lesions in the image, and then determining whether the body organ has a lesion and the corresponding degree of the disease.

[0046] However, due to the low quality of medical images and unreasonable image processing methods, the accuracy of disease diagnosis based on medical images is not high and diagnostic errors often occur.

[0047] In response to the above technical problems, an embodiment of the present application proposes a disease diagnosis method based on images and clinical data. On the one hand, the method optimizes the processing of medical images, so that it can more fully mine information useful for disease diagnosis from medical images for disease diagnosis. On the other hand, the method combines medical images and clinical data for disease diagnosis, thereby performing disease diagnosis through more diverse data information, which can significantly improve the accuracy of disease diagnosis.

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] The embodiments of the present application propose a disease diagnosis method based on images and clinical data, which can be applied to devices or equipment with automatic diagnosis functions, such as automatic diagnosis and treatment equipment, computers, servers, intelligent terminal devices, etc. Exemplarily, the disease diagnosis method based on images and clinical data proposed in the embodiments of the present application can be in the form of a software program running on the above-mentioned device or equipment, so that the above-mentioned device or equipment executes the disease diagnosis method based on images and clinical data proposed in the embodiments of the present application by running the software program. The embodiments of the present application mainly introduce the specific processing process of the disease diagnosis method based on images and clinical data, and do not limit the application scenarios, implementation methods, etc. of the method.

[0050] In the subsequent embodiments, the diagnosis of hepatocellular carcinoma will be taken as an example to introduce the disease diagnosis method based on images and clinical data proposed in this application. Those skilled in the art can understand that the disease diagnosis method based on images and clinical data proposed in this application is not limited to the diagnosis of a certain disease, but can be applied to the diagnosis of diseases of any body organ or tissue.

[0051] See also Figure 1 As shown, the disease diagnosis method based on images and clinical data proposed in the embodiment of the present application includes:

[0052] S101. Perform feature extraction processing on the medical image based on the spatial dimension and the temporal dimension to obtain medical image features.

[0053] The medical image is a medical image of a specific body organ of a patient, for example, in the diagnosis of hepatocellular carcinoma, the medical image is a medical image of the patient's liver. The medical image can be any type of medical image, for example, an ultrasound image, a CT image, etc. Moreover, the medical image can be one or more medical images. When there are multiple medical images, they can be multiple medical images of the same type or multiple medical images of different types.

[0054] In some embodiments, after acquiring the above-mentioned medical images, the acquired medical images may be first standardized, including adjusting the image size and grayscale value, and performing data enhancement operations, such as rotation, flipping, scaling, etc. Alternatively, a region of interest may be extracted through a segmentation algorithm, such as extracting a liver region, to ensure that disease diagnosis can be performed based on consistent and high-quality image data, thereby improving the accuracy of subsequent disease diagnosis.

[0055] The above-mentioned feature extraction processing of medical images based on spatial dimensions refers to feature extraction of medical images from spatial dimensions, specifically extracting high-dimensional spatial features, converting complex medical images into meaningful feature representations, and retaining important spatial information. Feature extraction processing of medical images based on spatial dimensions can capture key information in different spatial positions in the medical image, such as the texture, color, shape, edge, etc. of the image, and can capture the correlation between these key information, so as to comprehensively obtain the key information in the medical image.

[0056] Exemplarily, feature extraction is performed on the image content at different positions of the medical image, such as extracting texture features, color features, shape features, etc. of the image content at different positions, and then correlation analysis and fusion are performed on the image features of the image content at different positions, thereby realizing feature extraction processing based on the spatial dimension.

[0057] Alternatively, medical images can be processed with the help of a neural network with a feature extraction function based on spatial dimensions to obtain image features extracted based on spatial dimensions. For example, through convolutional neural networks CNN, graph neural networks GNN, recurrent neural networks RNN, Transformer models, residual networks ResNet, etc., medical images can be processed based on spatial dimension feature extraction to obtain image features.

[0058] The above-mentioned feature extraction processing of medical images based on the time dimension refers to extracting features of medical images from the time dimension, so as to capture the changes of image features over time. The feature extraction processing of medical images based on the time dimension can capture the changes of key information of medical images over time, such as the changes of image texture, color, shape, edge, etc. over time, and can combine image features at different times, so as to comprehensively obtain key information in medical images.

[0059] Exemplarily, by extracting features from medical images taken at different times with respect to the same part, and then fusing the image features extracted at different times, feature extraction processing based on the time dimension of the medical image is achieved.

[0060] Alternatively, medical images can be processed with the help of a neural network with a time-based feature extraction function to obtain image features extracted based on the time dimension. For example, through a 3D convolutional neural network 3D-CNN, a long short-term memory network LSTM, a Transformer model, etc., medical images can be processed for feature extraction based on the time dimension to obtain image features.

[0061] In some embodiments, the feature extraction process based on the spatial dimension and the time dimension is performed on the medical image, and the feature extraction process based on the spatial dimension is first performed on the medical image, and then the extracted features are subjected to the feature extraction process based on the time dimension to obtain the final medical image features. Alternatively, the feature extraction process based on the time dimension is first performed on the medical image, and then the extracted features are subjected to the feature extraction process based on the spatial dimension to obtain the final medical image features. Alternatively, the feature extraction process based on the spatial dimension and the feature extraction process based on the time dimension are performed on the medical image at the same time, and then the extracted spatial dimension features and time dimension features are subjected to the fusion process to obtain the final medical image features.

[0062] S102: Perform feature extraction processing on the patient's clinical data to obtain clinical data features.

[0063] The above-mentioned patient clinical data include the clinical data of the patient and / or the clinical data of patients with specific diseases, and the patients with specific diseases refer to patients suffering from specific diseases in the specific body organs.

[0064] Specifically, clinical data of patients corresponding to the above-mentioned medical images and / or clinical data of patients with specific diseases can be obtained through the medical institution's HIS (Hospital Information System), EMR (Electronic Medical Record), LIS (Laboratory Information System) and other systems.

[0065] The above-mentioned patient with a specific disease refers to a patient whose specific body organ suffers from a specific disease. For example, when the above-mentioned medical image is a medical image of the liver of patient A, the above-mentioned patient with a specific disease may be patient B who is diagnosed with hepatocellular carcinoma.

[0066] Since the clinical data of a patient may include multiple data, and some data are useless for diagnosing a specific disease, in other embodiments, the clinical data may specifically be clinical data of the patient corresponding to a specific examination item, wherein the specific examination item is a clinical examination item used to determine a specific disease.

[0067] For example, assuming that the technical solution of the embodiment of the present application is used to diagnose hepatocellular carcinoma, when a patient suffers from hepatocellular carcinoma, clinically it is usually manifested in certain physical symptoms, signs, and laboratory body fluid examination items.

[0068] Among them, in terms of physical symptoms, early hepatocellular carcinoma may have no obvious symptoms and is often discovered during physical examinations or accidental medical imaging examinations. When the tumor grows, the patient may feel discomfort or pain in the right upper abdomen, especially when the tumor is located on the surface or edge of the liver; fatigue and weight loss are common in the late stage of liver cancer; loss of appetite, nausea and vomiting are related to the impact on the digestive system.

[0069] In terms of physical signs, patients with hepatocellular carcinoma usually show symptoms such as abdominal distension, liver enlargement, jaundice, ascites, and spleen enlargement.

[0070] In laboratory body fluid examination items, the clinical test data of liver function, tumor markers, hepatitis B and C virus tests and other items of hepatocellular carcinoma patients will be abnormal. For example, in liver function tests, AST (aspartate aminotransferase) and ALT (alanine aminotransferase) levels are elevated, and bilirubin is increased; in tumor marker examinations, elevated AFP (alpha-fetoprotein) levels are a typical manifestation of hepatocellular carcinoma, but not all hepatocellular carcinoma patients have elevated AFP; in hepatitis B and C virus testing items, many hepatocellular carcinoma cases are related to hepatitis B and C virus infection.

[0071] Therefore, the data of the above-mentioned somatosensory symptoms, signs, and laboratory experience test items are useful for diagnosing hepatocellular carcinoma. When obtaining the patient's clinical data, the clinical data of the above-mentioned somatosensory symptoms, signs, and laboratory experience test items can be obtained. Other clinical data that is useless for diagnosing hepatocellular carcinoma may not be obtained, thereby avoiding redundant and useless clinical data affecting the diagnostic efficiency and accuracy of hepatocellular carcinoma.

[0072] After obtaining the above-mentioned patient clinical data, the obtained patient clinical data is processed through feature engineering to achieve characterization of the patient clinical data and obtain clinical data features.

[0073] S103: Fusing the image features and the clinical data features to obtain fused features, and performing classification based on the fused features to obtain classification results.

[0074] Specifically, after obtaining the medical image features and the clinical data features respectively by executing the processing of steps S101 and S102, the two features are fused, for example, the medical image features and the clinical data features are spliced ​​to obtain fused features.

[0075] Then, classification processing is performed based on the fused features, specifically predicting the probability of the patient suffering from a specific type of disease or not suffering from a specific type of disease, and obtaining a classification result. The classification result is used to indicate whether the patient corresponding to the above medical image suffers from the above specific disease. For example, the patient's liver image features are fused with the patient's clinical data features, and then the classification result is obtained based on the fused features to indicate whether the patient suffers from hepatocellular carcinoma or not. The classification result is used to indicate whether the patient suffers from hepatocellular carcinoma.

[0076] In some embodiments, when the above-mentioned patient clinical data only includes the clinical data of the patient, the fusion feature obtained through the above-mentioned feature fusion processing includes not only the patient's medical image features, but also the patient's clinical data features, that is, the fusion feature contains richer reference information for diagnosing specific diseases, thereby having higher diagnostic accuracy than diagnosing specific diseases only through the patient's medical image features.

[0077] In the case that the above-mentioned patient clinical data only includes clinical data of patients with specific diseases, the fusion features obtained through the above-mentioned feature fusion processing include not only the medical image features of the patient, but also the clinical data information of patients with specific diseases. Therefore, when classification is performed based on the fusion features, there are clear diagnostic criteria or diagnostic samples for specific diseases to determine whether the patient suffers from the specific disease, thereby improving the diagnostic accuracy of the specific disease.

[0078] In the case where the above-mentioned patient clinical data includes the clinical data of the patient and the clinical data of patients with specific diseases, the fusion features obtained through the above-mentioned feature fusion processing include not only the patient's medical image features, but also the patient's clinical data features, as well as the clinical data of patients with specific diseases. On the one hand, it can enrich the reference information for diagnosing whether the patient has a specific disease, and on the other hand, it can carry the diagnostic criteria or diagnostic samples of the specific disease in the fusion features, so that based on the fusion features, it can be more accurately judged whether the patient has a specific disease.

[0079] From the above introduction, it can be seen that the disease diagnosis method based on images and clinical data proposed in the embodiment of the present application performs feature extraction processing on the patient's medical image based on the spatial dimension and the time dimension, so that more detailed and comprehensive image features can be extracted from the medical image. On this basis, the embodiment of the present application also fuses the patient's clinical data with the patient's medical image features, and then classifies specific diseases based on the fused features. The above-mentioned fused features contain richer and more detailed data information for diagnosing specific diseases, so that the classification results for specific diseases are more accurate and the diagnostic accuracy of specific diseases is improved.

[0080] In addition, since the disease diagnosis method based on images and clinical data proposed in the embodiment of the present application applies clinical data to disease diagnosis, when the disease diagnosis result (classification result) is obtained, the diagnosis result can be interpreted based on the clinical data used to obtain the diagnosis result, thereby improving the interpretability of the diagnosis result (classification result).

[0081] In another embodiment, the above-mentioned medical images include a medical image sequence of a patient, such as CT images and MRI images, which not only contain clearer and richer image information in a single image, but also include a sequence of multiple images, so their image information is richer.

[0082] For example, the patient's body images are scanned by CT or MRI equipment. The scanning sequence includes the non-contrast agent enhancement period (providing baseline images), arterial phase (scanning after 20-40 seconds to capture hepatic artery blood flow), portal vein phase (scanning after 60-70 seconds to display the portal vein and its branches), delayed phase or equilibrium phase (scanning after 3-5 minutes to evaluate contrast agent elution) to obtain CT or MRI images at different time points and form time series data.

[0083] For the above-mentioned medical image sequence, when extracting features, the image features can be extracted by the following steps A1-A3:

[0084] A1. Perform feature extraction processing based on spatial dimensions on each medical image frame in the medical image sequence to obtain image features of each medical image frame.

[0085] Specifically, for each image frame in the above medical image sequence, feature extraction processing based on the spatial dimension is performed respectively. The spatial feature extraction method introduced in the above embodiment can be used to extract image features to obtain image features of each medical image frame.

[0086] As a preferred implementation, the embodiment of the present application processes each medical image frame in the medical image sequence separately through a convolutional neural network to obtain image features of each medical image frame.

[0087] Specifically, the convolutional neural network is used to obtain a high-dimensional feature vector of an image by extracting spatial features at different levels of the image.

[0088] After the above medical image sequence is standardized, such as resizing and grayscale normalization, the medical image frames in the medical image sequence are input into the pre-trained convolutional neural network ResNet, so that the convolutional neural network performs convolution operations on the medical image frames and extracts spatial features at various levels. These convolutional layers can detect image features of different scales and complexities, such as edges, textures, and shapes.

[0089] Then after convolution and pooling operations, a multi-level feature map is obtained, which represents the spatial features of the image at different levels.

[0090] Finally, the extracted features are passed through a fully connected layer to generate a high-dimensional feature vector, that is, the image features of the medical image frame are obtained.

[0091] For each medical image frame in the medical image sequence, its image features can be obtained through the above-mentioned convolutional neural network.

[0092] A2. Arrange the image features of each medical image frame according to the order of each medical image frame in the medical image sequence to obtain an image feature sequence.

[0093] Specifically, according to the order of each medical image frame in the medical image sequence, the image features of each medical image frame are arranged to obtain an image feature sequence.

[0094] A3. Perform feature extraction processing based on the time dimension on the image feature sequence to obtain medical image features.

[0095] Specifically, the above-mentioned image feature sequence is subjected to feature extraction processing based on the time dimension, that is, each image feature frame in the image feature sequence is subjected to context-related feature extraction processing to obtain medical image features corresponding to the image feature sequence. The time feature extraction method introduced in the above embodiment can be used to extract image features to obtain medical image features.

[0096] As a preferred implementation, the embodiment of the present application processes the image feature sequence through a long short-term memory network LSTM to obtain medical image features.

[0097] Among them, the above-mentioned long short-term memory network is used to capture features that change over time and capture the dependencies between features, so that more useful feature components can be extracted from the features of medical image frames at different times.

[0098] The above image feature sequence is input into the LSTM network. The LSTM network analyzes the image feature sequence through its memory unit and gate mechanism, captures the long-term dependencies between features, extracts important patterns and trends that change over time, and obtains the final medical image features.

[0099] In other embodiments, the disease diagnosis method based on images and clinical data proposed in the embodiments of the present application can be implemented by training a diagnosis model.

[0100] The diagnostic model may include the above-mentioned convolutional neural network ResNet and LSTM network. Figure 2 A schematic diagram of the architecture of the diagnostic model is shown.

[0101] See also Figure 2 As shown, after obtaining the medical image and clinical data of the patient, the medical image and clinical data of the patient are respectively input into the diagnosis model.

[0102] The convolutional neural network in the diagnosis model first performs feature extraction processing on the medical image frames based on the spatial dimension to obtain the image features of each medical image frame. Then, the LSTM network performs feature extraction processing on the image feature sequence composed of the image features of each medical image frame based on the time dimension to obtain the medical image features.

[0103] On the other hand, the characterization processing module in the diagnosis model performs characterization processing on the patient's clinical data to obtain clinical data features.

[0104] Then, the medical image features and clinical data features output by the LSTM network are fused through a fully connected layer to obtain fused features.

[0105] After feature fusion, one or more fully connected layers are used for further feature processing and transformation. These fully connected layers can be used for dimensionality reduction and complex nonlinear transformations to extract key information for classification.

[0106] After the last fully connected layer, the softmax layer is used for classification prediction, and the predicted probability of each category (having a specific disease and not having a specific disease) is output, that is, the probability distribution of each category is obtained, where the probability of each category represents the model's prediction confidence for that category. The category with the highest probability is selected as the final classification result.

[0107] When training the above-mentioned diagnostic model, the patient's medical images and clinical data are used as training samples, and the patient's diagnostic results are used as labels. The diagnostic model is trained for disease diagnosis through loss functions such as cross entropy, and the model parameters are optimized until the model's diagnostic accuracy meets the requirements and the training is stopped.

[0108] When the above-mentioned diagnostic model is actually applied to diagnose a disease, the diagnostic model can be used as an aid for doctors to diagnose the disease, and the diagnostic model can be continuously optimized during the application process.

[0109] For example, after the medical image and the patient's clinical data are input into the above-mentioned diagnostic model to obtain the classification result, the diagnostic result corresponding to the above-mentioned medical image is obtained, and the diagnostic result may be the diagnostic result obtained by the doctor for the disease of the patient corresponding to the above-mentioned medical image.

[0110] Then, based on the classification results output by the diagnosis model and the above-mentioned diagnosis results, the parameters of the diagnosis model are optimized.

[0111] Specifically, the classification result output by the diagnostic model is compared with the above diagnostic result to determine whether the two are consistent. If the two are consistent, it means that the classification result output by the diagnostic model is correct, and the diagnostic model can be optimized at this time; if the two are inconsistent, it means that the classification result output by the diagnostic model is wrong, and the parameters of the diagnostic model can be optimized by calculating the loss function between the classification result output by the diagnostic model and the above diagnostic result.

[0112] Through the above-mentioned model optimization operation, the above-mentioned diagnostic model can be continuously optimized during the application process, so that the diagnostic accuracy of the diagnostic model is increasingly higher.

[0113] Corresponding to the above method, the present application embodiment also provides a disease diagnosis device based on images and clinical data, see Figure 3 As shown, the device comprises:

[0114] The first processing unit 100 is used to perform feature extraction processing on the medical image based on the spatial dimension and the temporal dimension to obtain medical image features, wherein the medical image is a medical image of a specific body organ of a patient;

[0115] The second processing unit 110 is used to perform feature extraction processing on the patient clinical data to obtain clinical data features, wherein the patient clinical data includes the clinical data of the patient and / or clinical data of patients with specific diseases, and the patients with specific diseases refer to patients with specific diseases in the specific body organs;

[0116] The classification processing unit 120 is used to fuse the image features and the clinical data features to obtain fused features, and perform classification based on the fused features to obtain classification results, wherein the classification results are used to indicate whether the patient suffers from the specific disease.

[0117] In some implementations, the medical image includes a medical image sequence, and the step of performing feature extraction processing on the medical image based on a spatial dimension and a temporal dimension to obtain medical image features includes:

[0118] Performing feature extraction processing based on spatial dimensions on each medical image frame in the medical image sequence to obtain image features of each medical image frame;

[0119] Arranging image features of each medical image frame according to the order of each medical image frame in the medical image sequence to obtain an image feature sequence;

[0120] The image feature sequence is subjected to feature extraction processing based on the time dimension to obtain medical image features.

[0121] In some implementations, each medical image frame in the medical image sequence is subjected to feature extraction processing based on the spatial dimension to obtain image features of each medical image frame, including:

[0122] Each medical image frame in the medical image sequence is processed separately by a convolutional neural network to obtain image features of each medical image frame;

[0123] The convolutional neural network is used to obtain a high-dimensional feature vector of an image by extracting spatial features at different levels of the image.

[0124] In some implementations, performing feature extraction processing based on the time dimension on the image feature sequence to obtain medical image features includes:

[0125] Processing the image feature sequence through a long short-term memory network to obtain medical image features;

[0126] The long short-term memory network is used to capture features that change over time and to capture dependencies between features.

[0127] In some embodiments, another disease diagnosis device based on images and clinical data is proposed, which is used to perform feature extraction processing on medical images based on spatial dimensions and temporal dimensions to obtain medical image features, wherein the medical image is a medical image of a specific body organ of a patient; perform feature extraction processing on the patient's clinical data to obtain clinical data features, wherein the patient's clinical data includes the patient's clinical data and / or clinical data of patients with specific diseases, and the specific disease patients refer to patients suffering from a specific disease in the specific body organ; fuse the image features and the clinical data features to obtain fused features, and perform classification based on the fused features to obtain a classification result, and the classification result is used to indicate whether the patient suffers from the specific disease.

[0128] In some implementations, a pre-trained diagnostic model is run in the device, and the device performs feature extraction processing on the medical image based on the spatial dimension and the temporal dimension to obtain medical image features, performs feature extraction processing on the patient's clinical data to obtain clinical data features, and fuses the image features and the clinical data features to obtain fused features, and classifies the medical image based on the fused features to obtain a classification result, including:

[0129] The medical image and the patient's clinical data are input into a pre-trained diagnostic model, so that the diagnostic model performs feature extraction processing on the medical image based on the spatial dimension and the temporal dimension to obtain medical image features, performs feature extraction processing on the patient's clinical data to obtain clinical data features, and fuses the image features and the clinical data features to obtain fused features, and classifies the medical image based on the fused features to obtain a classification result.

[0130] In some implementations, the apparatus is further configured to:

[0131] Obtaining a diagnosis result corresponding to the medical image;

[0132] Based on the diagnosis result and the classification result, parameters of the diagnosis model are optimized.

[0133] The disease diagnosis device based on images and clinical data provided in this embodiment belongs to the same application concept as the disease diagnosis method based on images and clinical data provided in the above embodiments of this application, and can execute the disease diagnosis method based on images and clinical data provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not fully described in this embodiment, please refer to the specific processing content of the disease diagnosis method based on images and clinical data provided in the above embodiments of this application, which will not be repeated here.

[0134] The functions implemented by the above-mentioned first processing unit 100, the second processing unit 110 and the classification processing unit 120 may be implemented by the same or different processors respectively, which is not limited in the embodiment of the present application.

[0135] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.

[0136] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0137] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0138] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0139] An embodiment of the present application further provides a control device, which includes a processor and an interface circuit. The processor in the control device is connected to an input-output component through the interface circuit of the control device.

[0140] The input-output component specifically refers to a functional component capable of acquiring or collecting medical images and patient clinical data.

[0141] The above-mentioned interface circuit can be any interface circuit that can realize the data communication function, for example, it can be a USB interface circuit, a Type-C interface circuit, a serial port circuit, a PCIE circuit, etc.

[0142] The processor in the control device is also a circuit with signal processing capabilities, which processes the medical images and patient clinical data acquired by the input and output components to obtain disease diagnosis results by executing any of the disease diagnosis methods based on images and clinical data introduced in the above embodiments. The specific implementation of the processor can refer to the above processor implementation, and the embodiments of this application are not strictly limited.

[0143] An embodiment of the present application provides a disease diagnosis device, which includes a data acquisition module and a processor connected to the data acquisition module.

[0144] Among them, the data acquisition module is used to acquire medical images and patient clinical data. For example, it can be an image acquisition device for acquiring medical images and a data acquisition device for acquiring patient clinical data. Alternatively, the data acquisition module can also be a data receiving port for receiving medical images and patient clinical data uploaded by users.

[0145] The processor is used to process the medical images and patient clinical data acquired by the data acquisition module to obtain a disease diagnosis result by executing any one of the disease diagnosis methods based on images and clinical data described in any of the above embodiments.

[0146] The specific processing process of the above-mentioned processor can refer to the introduction of the above-mentioned method embodiment, and the specific implementation method of the processor can also refer to the introduction of the above-mentioned embodiment.

[0147] The disease diagnosis device may specifically be a terminal device with a disease diagnosis function, such as a self-service consultation terminal, a server, a computer, an intelligent terminal, etc.

[0148] Another embodiment of the present application also provides an electronic device, see Figure 4 As shown, the device includes:

[0149] Memory 200 and processor 210;

[0150] The memory 200 is connected to the processor 210 and is used to store programs;

[0151] The processor 210 is used to implement the disease diagnosis method based on images and clinical data disclosed in any of the above embodiments by running the program stored in the memory 200.

[0152] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0153] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0154] A bus may include a pathway that transfers information between components of a computer system.

[0155] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0156] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0157] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0158] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0159] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0160] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0161] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any disease diagnosis method based on images and clinical data provided in the above embodiments of the present application.

[0162] An embodiment of the present application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in a memory through the data interface to execute the disease diagnosis method based on images and clinical data introduced in any of the above embodiments. The specific processing process and its beneficial effects can be found in the above-mentioned embodiment introduction of the disease diagnosis method based on images and clinical data.

[0163] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the disease diagnosis method based on images and clinical data described in any of the above-mentioned embodiments of this specification.

[0164] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0165] In addition, an embodiment of the present application may also be a storage medium on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the disease diagnosis method based on images and clinical data described in any of the above embodiments of this specification.

[0166] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0167] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0168] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0169] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0170] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0171] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0173] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0174] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0175] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0176] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A disease diagnosis method based on images and clinical data, characterized in that: include: Performing feature extraction processing on the medical image based on the spatial dimension and the temporal dimension to obtain medical image features, wherein the medical image is a medical image of a specific body organ of a patient; Performing feature extraction processing on the patient's clinical data to obtain clinical data features, wherein the patient's clinical data includes the patient's clinical data and / or clinical data of patients with specific diseases, wherein the patients with specific diseases refer to patients with the specific body organ suffering from the specific disease; The image features and the clinical data features are fused to obtain fused features, and classification is performed based on the fused features to obtain classification results, wherein the classification results are used to indicate whether the patient suffers from the specific disease.

2. The method according to claim 1, characterized in that The medical image includes a medical image sequence, and the feature extraction process based on the spatial dimension and the temporal dimension is performed on the medical image to obtain the medical image features, including: Performing feature extraction processing based on spatial dimensions on each medical image frame in the medical image sequence to obtain image features of each medical image frame; Arranging image features of each medical image frame according to the order of each medical image frame in the medical image sequence to obtain an image feature sequence; The image feature sequence is subjected to feature extraction processing based on the time dimension to obtain medical image features.

3. The method according to claim 2, characterized in that Each medical image frame in the medical image sequence is subjected to feature extraction processing based on the spatial dimension to obtain image features of each medical image frame, including: Each medical image frame in the medical image sequence is processed separately by a convolutional neural network to obtain image features of each medical image frame; The convolutional neural network is used to obtain a high-dimensional feature vector of an image by extracting spatial features at different levels of the image.

4. The method according to claim 2, characterized in that: Performing feature extraction processing based on the time dimension on the image feature sequence to obtain medical image features includes: Processing the image feature sequence through a long short-term memory network to obtain medical image features; The long short-term memory network is used to capture features that change over time and to capture dependencies between features.

5. The method according to any one of claims 1 to 4, characterized in that: Performing feature extraction processing based on spatial dimension and time dimension on the medical image to obtain medical image features, performing feature extraction processing on the patient clinical data to obtain clinical data features, and fusing the image features and the clinical data features to obtain fused features, and classifying the medical image based on the fused features to obtain classification results, including: The medical image and the patient's clinical data are input into a pre-trained diagnostic model, so that the diagnostic model performs feature extraction processing on the medical image based on the spatial dimension and the temporal dimension to obtain medical image features, performs feature extraction processing on the patient's clinical data to obtain clinical data features, and fuses the image features and the clinical data features to obtain fused features, and classifies the medical image based on the fused features to obtain a classification result.

6. The method according to claim 5, characterized in that The method further comprises: Obtaining a diagnosis result corresponding to the medical image; Based on the diagnosis result and the classification result, parameters of the diagnosis model are optimized.

7. A disease diagnosis device based on images and clinical data, characterized in that: include: A first processing unit is used to perform feature extraction processing on the medical image based on the spatial dimension and the temporal dimension to obtain medical image features, wherein the medical image is a medical image of a specific body organ of a patient; A second processing unit is used to perform feature extraction processing on the patient clinical data to obtain clinical data features, wherein the patient clinical data includes the clinical data of the patient and / or clinical data of patients with specific diseases, and the patients with specific diseases refer to patients with the specific body organ suffering from the specific disease; The classification processing unit is used to fuse the image features and the clinical data features to obtain fused features, and to perform classification based on the fused features to obtain classification results, wherein the classification results are used to indicate whether the patient suffers from the specific disease.

8. A disease diagnosis device, characterized in that: include: A data acquisition module, and a processor connected to the data acquisition module; The data acquisition module is used to acquire medical images and patient clinical data; The processor is used to execute the disease diagnosis method based on images and clinical data as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: include: Memory and processor; Wherein, the memory is connected to the processor and is used to store programs; The processor is used to implement the disease diagnosis method based on images and clinical data as described in any one of claims 1 to 6 by running the program in the memory.

10. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to perform the disease diagnosis method based on images and clinical data as claimed in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Schizophrenia patient screening method based on space-time attention mechanism

    CN113616209A

  • Multivariate time series classification method based on deep learning

    CN114154551A

  • Canvas-based electrocardiogram drawing method and system

    CN115670476A

  • Medical image processing method and device, medical imaging equipment and storage medium

    CN117152115A

  • Image detection model training method and device and image detection method

    CN117173469A