Diagnosis and treatment plan generation device and electronic device based on multi-source heterogeneous data
Through the diagnosis and treatment plan generation device of multi-source heterogeneous data, multiple analytical models and feature extraction technologies are used to solve the problem of poor accuracy in the formulation of diagnosis and treatment plan for single images in the existing technology, and the connection between multimodal image data is realized Established, improving the accuracy of diagnosis and treatment plans.
Patent Information
- Application Number
- CN202410935057.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-07-12
AI Technical Summary
The prior art has formulated diagnostic and treatment plans based on only a single type of imaging data in tumor treatment, and failed to fully consider the connection between multi-source heterogeneous data, resulting in poor accuracy of diagnosis and treatment plans.
Design a diagnosis and treatment plan generation device based on multi-source heterogeneous data. By obtaining multiple target medical images and their modal types and image interpretation information, use corresponding analysis models to extract semantic spatial understanding features, combining natural image spatial features and natural Language characteristics, generate comprehensive diagnosis and treatment plans.
By establishing the connection between multimodal images, the accuracy of diagnosis and treatment plans is improved, and the problem of poor accuracy caused by the formulation of a single image in the prior art is overcome.
Smart Images

Figure CN118471487B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of medical technology, and more specifically, to a device and electronic device for generating a diagnosis and treatment plan based on multi-source heterogeneous data. Background Art
[0002] A lot of medical data is often generated during tumor treatment, such as clinical data, pathological data, treatment data during radiotherapy (such as positioning images, daily images, planning documents, dose files, etc.), and follow-up data. These data play a vital role in tumor efficacy analysis and patient prognosis.
[0003] However, current mainstream methods such as deep learning models and machine learning models can only input data of a certain modality to formulate diagnosis and treatment plans, while ignoring the connection between multi-modal, multi-source heterogeneous data, which makes the analysis task difficult and leads to poor accuracy of the final diagnosis and treatment plan.
[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0005] The present application provides a device and electronic device for generating a diagnosis and treatment plan based on multi-source heterogeneous data, so as to at least solve the technical problem that the diagnosis and treatment plan has poor accuracy because the prior art only formulates the diagnosis and treatment plan based on a single type of image and does not fully consider the connection between the multi-source heterogeneous data.
[0006] According to one aspect of the present application, a diagnosis and treatment plan generation device based on multi-source heterogeneous data is provided, including: an acquisition unit, used to acquire N target medical images of a target object and the modality type and image interpretation information of each target medical image, wherein N is an integer greater than 1; an analysis unit, used to input the target medical image and the image interpretation information of the target medical image into an analysis model corresponding to the modality type of the target medical image, and determine the semantic space understanding features of the target medical image through the analysis model corresponding to the modality type of the target medical image, wherein different modality types correspond to different analysis models, and the semantic space understanding features are used to characterize the association relationship between the image interpretation information of the target medical image and the image information of the medical image; a determination unit, used to determine the target diagnosis and treatment plan of the target object according to the modality type and the semantic space understanding features of the target medical image.
[0007] Optionally, the acquisition unit in the diagnosis and treatment plan generation device based on multi-source heterogeneous data also includes: a modality type determination subunit, which is used to input the target medical image into the classification model, and determine the modality type of the target medical image through the classification model using prior knowledge learned in the model training phase, wherein the classification model is a neural network model trained based on historical medical images of known modality types.
[0008] Optionally, the diagnosis and treatment plan generation device based on multi-source heterogeneous data also includes: a training data set acquisition unit, used to acquire L training data sets, wherein L is an integer greater than 1, and each training data set includes multiple training medical images of the same modality type and image interpretation information of each training medical image; an analysis model training unit, used to obtain L analysis models based on the L training data sets, wherein the i-th analysis model among the L analysis models is obtained by training the i-th training data set among the L training data sets.
[0009] Optionally, the analysis model training unit in the diagnosis and treatment plan generation device based on multi-source heterogeneous data also includes: a first processing subunit, used to establish an association relationship between the image interpretation information of the training medical images in the i-th training data set and the training medical images, wherein the training medical images in the i-th training data set are all images of the i-th modality type; an analysis model training subunit, used to input the training medical images in the i-th training data set and the image interpretation information of the training medical images into the neural network, iteratively train the neural network to learn the association relationship between the training medical images and the image interpretation information of the training medical images, and when the neural network meets the iteration end condition, use the neural network that has finished training as the analysis model corresponding to the i-th modality type.
[0010] Optionally, the determination unit in the diagnosis and treatment plan generation device based on multi-source heterogeneous data also includes: a natural image spatial feature extraction subunit, used to extract natural image spatial features from the target medical image, wherein the natural image spatial features include at least texture features, spatial features, shape features and color features of the target medical image; a diagnosis and treatment plan determination subunit, used to determine the target diagnosis and treatment plan of the target object according to the modality type, semantic space understanding features and natural image space features of the target medical image.
[0011] Optionally, the treatment plan determination subunit in the treatment plan generation device based on multi-source heterogeneous data also includes: a natural language feature extraction module, used to extract natural language features of the target medical image from the image interpretation information of the target medical image, wherein the natural language features are used to characterize the semantic meaning of the image interpretation information of the target medical image; a treatment plan determination module, used to determine the target treatment plan for the target object according to the modality type, semantic space understanding features, natural image space features and natural language features of the target medical image.
[0012] Optionally, a treatment plan determination module in a treatment plan generation device based on multi-source heterogeneous data includes: a first processing submodule, used to take the modality type, semantic space understanding features, natural image space features and natural language features of the target medical image as model input features; a second processing submodule, used to input the model input features into a composite network model, and convert the model input features into feature vectors through the composite network model, wherein the feature vector is used to represent the features after fusion processing of the modality type, semantic space understanding features, natural image space features and natural language features of the target medical image in vector form; a third processing submodule, used to determine the target treatment plan for the target object based on the feature vector through the composite network model, wherein the composite network model is a neural network model trained using historical medical images of different modalities as training samples and historical treatment plans formulated for historical medical images as training labels.
[0013] Optionally, the diagnosis and treatment plan generation device based on multi-source heterogeneous data also includes: a first processing unit, used to use the modality type, semantic space understanding features, natural image space features and natural language features of historical medical images as model training features; a second processing unit, used to input the model training features into the initial neural network model, iteratively train the initial neural network model to learn the association between the model training features and the historical diagnosis and treatment plans formulated for the historical medical images, and when the initial neural network model meets the training end conditions, the initial neural network model that has ended the training is used as the composite network model.
[0014] Optionally, the composite network model and the classification model are models obtained by joint training, wherein the joint training represents that the data output by the classification model in the training phase is used as the input data required for the composite network model during training, and the number of iterative training of the composite network model and the number of iterative training of the classification model influence each other.
[0015] According to another aspect of the present application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, wherein when the computer program is executed, the device where the computer-readable storage medium is located controls the operation of any one of the apparatuses for generating a diagnosis and treatment plan based on multi-source heterogeneous data.
[0016] According to another aspect of the present application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors control the operation of any one of the above-mentioned devices for generating a diagnosis and treatment plan based on multi-source heterogeneous data.
[0017] In the present application, a diagnosis and treatment plan generation device based on multi-source heterogeneous data is provided, wherein the diagnosis and treatment plan generation device includes: an acquisition unit, an analysis unit and a determination unit.
[0018] An acquisition unit is used to acquire N target medical images of a target object and the modality type and image interpretation information of each target medical image, wherein N is an integer greater than 1; an analysis unit is used to input the target medical image and the image interpretation information of the target medical image into an analysis model corresponding to the modality type of the target medical image, and determine the semantic space understanding features of the target medical image through the analysis model corresponding to the modality type of the target medical image, wherein different modality types correspond to different analysis models, and the semantic space understanding features are used to characterize the association between the image interpretation information of the target medical image and the image information of the medical image; a determination unit is used to determine the target diagnosis and treatment plan of the target object according to the modality type and semantic space understanding features of the target medical image.
[0019] From the above content, it can be seen that according to the technical solution of the present application, by acquiring multiple target medical images of the target object and the modality type and image interpretation information of each target medical image, the purpose of acquiring multi-source heterogeneous data including image data, image modality information and image interpretation information is achieved.
[0020] In addition, it should be noted that the present application comprehensively determines the target diagnosis and treatment plan for the target object based on the modality type of the target medical image and the semantic space understanding features extracted from the target medical image by the analysis model corresponding to the specific modality type. Compared with the prior art that only formulates diagnosis and treatment plans based on a single type of image, the present application not only uses the modality type as a reference feature for determining the diagnosis and treatment plan, but also deploys a corresponding analysis model for each modality type. Therefore, when the target medical image is a multimodal image, for each modality of the target medical image, the present application will use the corresponding analysis model to analyze and obtain the semantic space understanding features of various modality images, that is, it achieves the purpose of obtaining the feature information of various modality images and establishing the connection between multi-source heterogeneous data of different modality images, thereby improving the accuracy of the diagnosis and treatment plan, and solving the technical problem that the accuracy of the diagnosis and treatment plan is poor due to the fact that the prior art only formulates the diagnosis and treatment plan based on a single type of image and does not fully consider the connection between multi-source heterogeneous data. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0022] Figure 1 is a schematic diagram of an optional device for generating a diagnosis and treatment plan based on multi-source heterogeneous data according to an embodiment of the present application;
[0023] Figure 2 is a schematic diagram of an optional analysis model use process according to an embodiment of the present application;
[0024] Figure 3 is a schematic diagram of an optional diagnosis and treatment plan generation process according to an embodiment of the present application;
[0025] Figure 4 is a schematic diagram of another optional process of generating a diagnosis and treatment plan based on natural image spatial features according to an embodiment of the present application;
[0026] Figure 5 It is a schematic diagram of another optional process of generating a diagnosis and treatment plan based on natural language features according to an embodiment of the present application. DETAILED DESCRIPTION
[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0029] It should also be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) collected by this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with the relevant laws, regulations and standards of the relevant regions, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation portals are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or institution.
[0030] In addition, this application collects and analyzes customer information and provides users with corresponding operation entrances for them to choose to agree or reject automated decision-making results.
[0031] According to an embodiment of the present application, an optional embodiment of a diagnosis and treatment plan generation device based on multi-source heterogeneous data is provided, wherein: Figure 1 is a schematic diagram of an optional device for generating a diagnosis and treatment plan based on multi-source heterogeneous data according to an embodiment of the present application, such as Figure 1 As shown, the diagnosis and treatment plan generating device includes: an acquisition unit 101, an analysis unit 102 and a determination unit 103.
[0032] Optionally, the acquisition unit 101 is used to acquire N target medical images of the target object and the modality type and image interpretation information of each target medical image, where N is an integer greater than 1.
[0033] Optionally, the target object is a patient who is about to receive radiotherapy. A core invention purpose of the diagnosis and treatment plan generating device provided in the embodiment of the present application is to generate a target diagnosis and treatment plan for the target object by utilizing multi-source heterogeneous data related to the target object.
[0034] Optionally, the N target medical images of the target object include but are not limited to CT images, MR images, PET images and pathological images of the target object.
[0035] Optionally, the modality type and image interpretation information of the target medical image involved in the embodiments of the present application generally refer to different types of imaging technologies used in medical diagnosis and treatment and how to obtain useful information from these images. For example, the medical image modality types include but are not limited to the following:
[0036] 1. X-ray imaging: Usually used to view bone structure, it can also be used to view chest organs such as the heart and lungs.
[0037] 2. Computed tomography (CT): Uses X-rays and computer technology to obtain detailed images of cross-sections of the body. It is often used to examine organs, blood vessels, and tumors.
[0038] 3. Magnetic resonance imaging (MRI): Uses strong magnetic fields and radio waves to obtain detailed images of the body's soft tissues. It is often used to examine the brain, muscles, and joints.
[0039] 4. Positron emission tomography (PET): It evaluates the metabolic activity of organs and tissues by detecting the distribution of radioactive tracers in the body. It is often used in cancer diagnosis and brain disease research.
[0040] 5. Ultrasound imaging: Uses the reflection of high-frequency sound waves to generate real-time images of the body's internal structures. It is often used to examine the fetus, heart, blood vessels and other internal organs.
[0041] 6. Digital Subtraction Angiography (DSA): A special X-ray technique used to visualize blood vessels and blood flow.
[0042] Alternatively, image interpretation information refers to the data and insights obtained from medical images for diagnosis and treatment decisions. Image interpretation is usually performed by radiologists or specialized technicians for each modality of imaging, who analyze the images to look for abnormalities or lesions and then integrate this information into the patient's medical record.
[0043] Optionally, the image interpretation information may include: identification information of anatomical structures, detection information of lesion areas, description information of tissue characteristics, evaluation information of organ functions, distribution information and shape information of organs at risk, and distribution information and shape information of target areas. Image interpretation information is essential for formulating treatment plans, evaluating treatment effects, and conducting disease monitoring. With the development of artificial intelligence and machine learning technologies, the accuracy and efficiency of image interpretation are constantly improving, helping to provide more accurate diagnoses and better patient care.
[0044] In an optional embodiment, the diagnosis and treatment plan generation device also includes an analysis unit 102, wherein the analysis unit 102 is used to input the target medical image and the image interpretation information of the target medical image into an analysis model corresponding to the modality type of the target medical image, and determine the semantic space understanding features of the target medical image through the analysis model corresponding to the modality type of the target medical image, wherein different modality types correspond to different analysis models, and the semantic space understanding features are used to characterize the correlation between the image interpretation information of the target medical image and the image information of the medical image.
[0045] Optionally, Figure 2 is a schematic diagram of an optional analysis model use process according to an embodiment of the present application, such as Figure 2 As shown, the N target medical images corresponding to the target object include: a target PET image, a target CT image (1), a target CT image (2), and a target MR image. In addition, each target medical image corresponds to its own image interpretation information, wherein the image interpretation information corresponding to the target PET image is established by a doctor who is specifically responsible for interpreting PET images; the image interpretation information corresponding to the target CT image (1) and the image interpretation information corresponding to the target CT image (2) are established by a doctor who is specifically responsible for interpreting CT images; and the image interpretation information corresponding to the target MR image is established by a doctor who is specifically responsible for interpreting MR images.
[0046] Alternatively, if Figure 2As shown, the diagnosis and treatment plan generation device inputs the target PET image and the image interpretation information corresponding to the target PET image into the analysis model A, wherein the analysis model A is specifically used to analyze images of the PET modality, and the analysis model A can determine the semantic space understanding features of the target PET image based on the target PET image and the image interpretation information corresponding to the target PET image. Similarly, the diagnosis and treatment plan generation device inputs the target CT image (1), the target CT image (2), the image interpretation information corresponding to the target CT image (1), and the image interpretation information corresponding to the target CT image (2) into the analysis model B, wherein the analysis model B is specifically used to analyze images of the CT modality, and the analysis model B can determine the semantic space understanding features of the target CT image (1) based on the target CT image (1) and the image interpretation information corresponding to the target CT image (1), and determine the semantic space understanding features of the target CT image (2) based on the target CT image (2) and the image interpretation information corresponding to the target CT image (2). The diagnosis and treatment plan generating device can also input the target MR image and the image interpretation information corresponding to the target MR image into the analysis model C, wherein the analysis model C is specifically used to analyze the images of the MR modality, and the analysis model C can determine the semantic space understanding characteristics of the target MR image based on the target MR image and the image interpretation information corresponding to the target MR image.
[0047] It should be noted that after a patient develops a tumor, a professional doctor is needed to perform imaging diagnosis and report interpretation, which requires a lot of manpower. The medical images used in the diagnosis process may include multimodal images such as CT images, MR images, PET images, and pathological images. All report diagnostic interpretations point to the same clinical goal, that is, the diagnosis and progression of the same patient's condition, which makes it possible to convert all clinical data and images into common modal data features and automatically generate reports. The technical solution of the present application deploys multiple analysis models to process each modality of medical images and corresponding image interpretation information separately, thereby realizing universal processing of multi-source heterogeneous data of patients, thereby overcoming the problem of poor accuracy of diagnosis and treatment plans in the prior art, which is based on a single type of image to formulate diagnosis and treatment plans without fully considering the connection between multi-source heterogeneous data.
[0048] It should also be noted that, in the embodiments of the present application, multi-source heterogeneous data can be understood as data with different sources and different structures. For example, multimodal medical images such as MR images and PET images corresponding to the target object are typical multi-source heterogeneous data. MR images are obtained by a special MR imaging machine, and PET images are obtained by a special PET imaging machine, which shows that the sources of the two images are different. In addition, the imaging principles of MR images and PET images are also different, so the structures of the two images are also different.
[0049] In an optional embodiment, the diagnosis and treatment plan generating device further includes a determination unit 103, wherein the determination unit 103 is used to determine a target diagnosis and treatment plan for a target object according to a modality type and semantic space understanding features of a target medical image.
[0050] Optionally, the determination unit 103 may utilize a pre-trained neural network model with the modality type and semantic space understanding features of the target medical image as model input features, and then predict a target diagnosis and treatment plan for the target object.
[0051] Optionally, the target diagnosis and treatment plan includes, but is not limited to, a surgical plan, a chemotherapy plan, and a radiotherapy plan for the target subject.
[0052] In an optional embodiment, a treatment report model may also be deployed in the treatment plan generating device. By inputting the target treatment plan into the treatment report model, the treatment report model may output a treatment report for the target object based on the target treatment plan, wherein the treatment report is used to describe the expected treatment effect after treating the target object according to the target treatment plan.
[0053] In summary, in the present application, a diagnosis and treatment plan generation device based on multi-source heterogeneous data is provided, wherein the diagnosis and treatment plan generation device includes: an acquisition unit 101, an analysis unit 102, and a determination unit 103. The acquisition unit 101 is used to acquire N target medical images of the target object and the modality type and image interpretation information of each target medical image, wherein N is an integer greater than 1; the analysis unit 102 is used to input the target medical image and the image interpretation information of the target medical image into the analysis model corresponding to the modality type of the target medical image, and determine the semantic space understanding feature of the target medical image through the analysis model corresponding to the modality type of the target medical image, wherein different modality types correspond to different analysis models, and the semantic space understanding feature is used to characterize the association between the image interpretation information of the target medical image and the image information of the medical image; the determination unit 103 is used to determine the target diagnosis and treatment plan of the target object according to the modality type and semantic space understanding feature of the target medical image.
[0054] From the above content, it can be seen that according to the technical solution of the present application, by acquiring multiple target medical images of the target object and the modality type and image interpretation information of each target medical image, the purpose of acquiring multi-source heterogeneous data including image data, image modality information and image interpretation information is achieved.
[0055] In addition, it should be noted that the present application comprehensively determines the target diagnosis and treatment plan for the target object based on the modality type of the target medical image and the semantic space understanding features extracted from the target medical image by the analysis model corresponding to the specific modality type. Compared with the prior art that only formulates diagnosis and treatment plans based on a single type of image, the present application not only uses the modality type as a reference feature for determining the diagnosis and treatment plan, but also deploys a corresponding analysis model for each modality type. Therefore, when the target medical image is a multimodal image, for each modality of the target medical image, the present application will use the corresponding analysis model to analyze and obtain the semantic space understanding features of various modality images, that is, it achieves the purpose of obtaining the feature information of various modality images and establishing the connection between multi-source heterogeneous data of different modality images, thereby improving the accuracy of the diagnosis and treatment plan, and solving the technical problem that the accuracy of the diagnosis and treatment plan is poor due to the fact that the prior art only formulates the diagnosis and treatment plan based on a single type of image and does not fully consider the connection between multi-source heterogeneous data.
[0056] In an optional embodiment, the acquisition unit 101 further includes: a modality type determination subunit. The modality type determination subunit is used to input the target medical image into the classification model, and determine the modality type of the target medical image by using the classification model using the prior knowledge learned in the model training phase, wherein the classification model is a neural network model trained based on historical medical images of known modality types.
[0057] Optionally, Figure 3 is a schematic diagram of an optional diagnosis and treatment plan generation process according to an embodiment of the present application, such as Figure 3 As shown, first, N target medical images of the target object are obtained from the database, for example, the target CT image, the target MR image, the target PET image, and the target pathological image of the target object. The diagnosis and treatment generation device inputs the N medical images of the target object into the classification model, and the classification model determines the modality type (which can be recorded as F1) of each target medical image.
[0058] Alternatively, if Figure 3 As shown, the diagnosis and treatment plan generating device can also determine the semantic space understanding characteristics of the target medical image (which can be recorded as F2) based on each target medical image and the image interpretation information of the target medical image. Then, the diagnosis and treatment plan generating device inputs the modality type and semantic space understanding characteristics of the target medical image into the composite network model, and the composite network model outputs the target diagnosis and treatment plan of the target object.
[0059] Optionally, the present application does not specifically limit the neural network structure of the classification model. For example, the neural network structure of the classification model can be selected from various types of structures such as convolutional neural networks, graph neural networks, LSTM neural networks, and attention mechanism neural networks. In addition, during the training process of the classification model, medical images are used as sample data for model training, and the modality type of medical images is used as the training label for model training. The output of any structural layer can be specified as the classification feature F1 (i.e., the modality type output by the classification model) or a fully connected layer can be added to the last layer of any network structure, and the output of the fully connected layer can be used as the classification feature F1.
[0060] It should be noted that the imaging principles of images of different modality types are different, and the image presentation forms for the same organ area may also be different, for example, differences in resolution, clarity, shooting angles, etc. On this basis, by distinguishing the modality types of the target medical images, more accurate reference information can be provided when subsequently formulating diagnosis and treatment plans, thereby achieving the technical effect of improving the accuracy of diagnosis and treatment plan formulation.
[0061] In addition, in an optional embodiment, when subsequently formulating a target diagnosis and treatment plan for the target object, different weights can be assigned to target medical images of different modality types. For example, since the resolution of PET images is higher than that of MR images, the credibility of PET images is higher than that of MR images. On this basis, a higher weight can be assigned to PET images than to MR images, so that more attention can be paid to the information of PET images when generating a target diagnosis and treatment plan.
[0062] In an optional embodiment, the diagnosis and treatment plan generating device further includes: a training data set acquisition unit and an analysis model training unit.
[0063] Among them, the training data set acquisition unit is used to acquire L training data sets, wherein L is an integer greater than 1, and each training data set includes multiple training medical images of the same modality type and image interpretation information of each training medical image; the analysis model training unit is used to obtain L analysis models according to the L training data sets, wherein the i-th analysis model among the L analysis models is obtained by training the i-th training data set among the L training data sets.
[0064] Optionally, the diagnosis and treatment plan generating device can obtain L analysis models through training of L training data sets, wherein each analysis model is used to specifically process medical images of one modality type and corresponding image interpretation information.
[0065] For example, the L training data sets include an MR image set, a CT image set, and a PET image set, wherein the MR image set includes multiple MR images and image interpretation information of each MR image; the CT image set includes multiple CT images and image interpretation information of each CT image; and the PET image set includes multiple PET images and image interpretation information of each PET image.
[0066] Optionally, the analysis model training unit can obtain an analysis model specifically used to analyze the correlation between MR images and MR image interpretation information based on the MR image set training; obtain an analysis model specifically used to analyze the correlation between CT images and CT image interpretation information based on the CT image set training; obtain an analysis model specifically used to analyze the correlation between PET images and PET image interpretation information based on the PRT image set training.
[0067] It should be noted that the technical solution of the present application realizes the universal processing of multi-source heterogeneous data of patients by deploying multiple analysis models to process each modality of medical images and the corresponding image interpretation information respectively, thereby overcoming the problem of poor accuracy of treatment plans caused by the prior art of formulating diagnosis and treatment plans based on only a single type of image without fully considering the connection between multi-source heterogeneous data.
[0068] In an optional embodiment, the above-mentioned analysis model training unit includes: a first processing subunit and an analysis model training subunit.
[0069] Optionally, the first processing subunit is used to establish an association relationship between the image interpretation information of the training medical images in the i-th training data set and the training medical images, wherein the training medical images in the i-th training data set are all images of the i-th modality type; the analysis model training subunit is used to input the training medical images in the i-th training data set and the image interpretation information of the training medical images into the neural network, iteratively train the neural network to learn the association relationship between the training medical images and the image interpretation information of the training medical images, and when the neural network meets the iteration end condition, the neural network that has finished training is used as the analysis model corresponding to the i-th modality type.
[0070] For example, assuming that the i-th training data set mentioned above is an MR image set, the first processing subunit can first establish an association relationship between each MR image in the MR image set and the corresponding image interpretation information, wherein each MR image in the MR image set is a training medical image in the MR image set.
[0071] Optionally, the analysis model training subunit can input each MR image in the MR image set and the corresponding image interpretation information into the neural network, and then iteratively train the neural network to learn the association between each MR image and the corresponding image interpretation information, that is, train the neural network to extract the semantic spatial understanding features of the MR images. When the neural network meets the iteration end condition, the neural network that has finished training is used as an analysis model specifically for extracting the semantic spatial understanding features of MR images.
[0072] Optionally, similar to the way of processing MR image sets, the i-th training data set can also be a set consisting of other modality types of images and corresponding image interpretation information, so as to obtain an analysis model for extracting semantic spatial understanding features of other modality types of images based on these training data sets.
[0073] Optionally, the iteration end condition may be that the number of iterative training reaches the maximum number of training times, or that the loss function values output by the neural network for K consecutive times are all less than a preset threshold, where K is an integer greater than or equal to 1.
[0074] In an optional embodiment, the determination unit 103 further includes: a natural image spatial feature extraction subunit and a diagnosis and treatment plan determination subunit.
[0075] Optionally, a natural image spatial feature extraction subunit is used to extract natural image spatial features from the target medical image, wherein the natural image spatial features include at least texture features, spatial features, shape features and color features of the target medical image; a treatment plan determination subunit is used to determine a target treatment plan for the target object based on the modality type, semantic spatial understanding features and natural image spatial features of the target medical image.
[0076] Optionally, Figure 4 is a schematic diagram of another optional process of generating a diagnosis and treatment plan based on natural image spatial features according to an embodiment of the present application, such as Figure 4 As shown, the diagnosis and treatment plan generating device extracts natural image space features (which can be recorded as F3) from the target medical image, and then inputs the modality type, semantic space understanding features and natural image space features of the target medical image into the composite network model. The composite network model comprehensively determines the target diagnosis and treatment plan of the target object based on the modality type, semantic space understanding features and natural image space features.
[0077] It should be noted that the natural image spatial features at least include the texture features, spatial features, shape features and color features of the target medical image. Unlike the semantic spatial understanding features, the natural image spatial features can be understood as the features that a "person" without any medical knowledge can extract after seeing the target medical image, such as texture features, spatial features, shape features and color features, etc., while the semantic spatial understanding features represent the correlation between the image interpretation information of the target medical image and the image information of the medical image. Therefore, the semantic spatial understanding features can be understood as the features that a "doctor" with medical knowledge can extract after seeing the target medical image. It can be seen that the natural image spatial features and the semantic spatial understanding features are equivalent to the analysis results of the target medical image from different angles by different roles. The target diagnosis and treatment plan of the target object can be comprehensively determined based on the modality type, semantic spatial understanding features and natural image spatial features, which can further improve the accuracy of the target diagnosis and treatment plan.
[0078] In an optional embodiment, the diagnosis and treatment plan determination subunit also includes: a natural language feature extraction module and a diagnosis and treatment plan determination module.
[0079] Optionally, a natural language feature extraction module is used to extract natural language features of the target medical image from the image interpretation information of the target medical image, wherein the natural language features are used to characterize the semantic meaning of the image interpretation information of the target medical image; and a treatment plan determination module is used to determine the target treatment plan of the target object based on the modality type, semantic space understanding features, natural image space features and natural language features of the target medical image.
[0080] Optionally, Figure 5 is a schematic diagram of another optional process of generating a diagnosis and treatment plan based on natural language features according to an embodiment of the present application, such as Figure 5 As shown, the diagnosis and treatment plan generating device can also extract natural language features (which can be recorded as F4) from the image interpretation information of the target medical image. Then, the diagnosis and treatment generating device inputs the modality type, semantic space understanding features, natural image space features and natural language features of the target medical image into the composite network model, and the composite network model comprehensively determines the target diagnosis and treatment plan of the target object based on these features.
[0081] In an optional embodiment, if Figure 5 As shown, a treatment report model can also be deployed in the treatment plan generating device. By inputting the target treatment plan into the treatment report model, the treatment report model can output a treatment report for the target object based on the target treatment plan, wherein the treatment report is used to describe the expected treatment effect after treating the target object according to the target treatment plan.
[0082] It should be noted that, unlike the semantic space understanding features, the natural language features actually no longer focus on the image data on the target medical image, but mainly focus on the semantic meaning of the image interpretation information, for example, by extracting keywords from the image interpretation information to determine the key semantic information. When the image interpretation information contains too much content, extracting the natural language features of the image interpretation information can reduce the difficulty of analyzing the image interpretation information and extract sufficiently concise semantic information; when the image interpretation information contains inaccurate descriptions such as inconsistencies, extracting the natural language features of the image interpretation information can effectively identify the semantic content of the incorrect description, or correct the semantics by itself, thereby achieving the purpose of extracting sufficiently accurate semantic information.
[0083] In summary, determining the target diagnosis and treatment plan for the target object based on the modality type, semantic space understanding characteristics, natural image space characteristics, and natural language characteristics of the target medical image can also further improve the accuracy of the target diagnosis and treatment plan for the target object.
[0084] In an optional embodiment, the diagnosis and treatment plan determination module includes: a first processing submodule, a second processing submodule and a third processing submodule.
[0085] Among them, the first processing submodule is used to use the modality type, semantic space understanding features, natural image space features and natural language features of the target medical image as model input features; the second processing submodule is used to input the model input features into the composite network model, and convert the model input features into feature vectors through the composite network model, wherein the feature vector is used to represent the features after the fusion processing of the modality type, semantic space understanding features, natural image space features and natural language features of the target medical image in vector form; the third processing submodule is used to determine the target diagnosis and treatment plan of the target object according to the feature vector through the composite network model, wherein the composite network model is a neural network model trained with historical medical images of different modalities as training samples and historical diagnosis and treatment plans formulated for historical medical images as training labels.
[0086] Optionally, the composite network model can perform feature fusion processing on the modality type, semantic space understanding features, natural image space features, and natural language features of the target medical image, for example, by setting different weights for different features and then performing weighted calculations on the features to achieve feature fusion and calculate feature vectors.
[0087] From the above content, it can be seen that the composite network model in the present application, as the final processing model of all features, can comprehensively consider the information of all features to determine the target diagnosis and treatment plan for the target object, thereby achieving the technical effect of comprehensively considering the multi-source heterogeneous data related to the target object to determine the target diagnosis and treatment plan, improving the comprehensiveness of feature considerations, and thus solving the technical problem of poor accuracy of treatment plans in the prior art, which is to formulate diagnosis and treatment plans based on a single type of image without fully considering the connection between multi-source heterogeneous data.
[0088] In an optional embodiment, the diagnosis and treatment plan generation device also includes: a first processing unit and a second processing unit. The first processing unit is used to use the modality type, semantic space understanding features, natural image space features and natural language features of the historical medical images as model training features; the second processing unit is used to input the model training features into the initial neural network model, iteratively train the initial neural network model to learn the relationship between the model training features and the historical diagnosis and treatment plans formulated for the historical medical images, and when the initial neural network model meets the training end conditions, the initial neural network model that has finished training is used as the composite network model.
[0089] Optionally, during the training process of the composite network model, the modality type, semantic space understanding features, natural image space features, and natural language features of historical medical images can be used as model training features, and the historical diagnosis and treatment plans formulated for historical medical images can be used as model training labels. The initial neural network model is iteratively trained based on the model training features and model training labels to learn the relationship between the model training features and the model training labels. In other words, the initial neural network model is trained to have the ability to predict the content of the diagnosis and treatment plan based on the modality type, semantic space understanding features, natural image space features, and natural language features of medical images.
[0090] Optionally, the training termination condition of the initial neural network model may be that the training times of the initial neural network model reaches the maximum training times, or that the loss function value outputted by the initial neural network model at least once in a row is less than a preset function value.
[0091] In an optional embodiment, the composite network model and the classification model are models obtained by joint training, wherein the joint training represents that the data output by the classification model in the training phase is used as the input data required for the composite network model during training, and the number of iterative training times of the composite network model and the number of iterative training times of the classification model influence each other.
[0092] Optionally, in an embodiment of the present application, the modality type features of the training medical images output during the classification model training phase can be used as the input data required for the composite network model during training. In addition, only when both the classification model and the composite network model have reached the conditions for the end of iterative training can the training of the two models be ended, that is, the number of iterative training times of the composite network model and the number of iterative training times of the classification model affect each other.
[0093] It should be noted that the purpose of joint training is to establish a strong correlation between the composite network model and the classification model during the model training stage. During the training process, the two models also use each other's training results as training constraints. Therefore, overall, a classification model and a composite network model with better prediction results can be obtained.
[0094] From the above content, it can be seen that according to the technical solution of the present application, by acquiring multiple target medical images of the target object and the modality type and image interpretation information of each target medical image, the purpose of acquiring multi-source heterogeneous data including image data, image modality information and image interpretation information is achieved.
[0095] In addition, it should be noted that the present application comprehensively determines the target diagnosis and treatment plan for the target object based on the modality type of the target medical image and the semantic space understanding features extracted from the target medical image by the analysis model corresponding to the specific modality type. Compared with the prior art that only formulates diagnosis and treatment plans based on a single type of image, the present application not only uses the modality type as a reference feature for determining the diagnosis and treatment plan, but also deploys a corresponding analysis model for each modality type. Therefore, when the target medical image is a multimodal image, for each modality of the target medical image, the present application will use the corresponding analysis model to analyze and obtain the semantic space understanding features of various modality images, that is, it achieves the purpose of obtaining the feature information of various modality images and establishing the connection between multi-source heterogeneous data of different modality images, thereby improving the accuracy of the diagnosis and treatment plan, and solving the technical problem that the accuracy of the diagnosis and treatment plan is poor due to the fact that the prior art only formulates diagnosis and treatment plans based on a single type of image and does not fully consider the connection between multi-source heterogeneous data.
[0096] According to another aspect of the present application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, wherein when the computer program is executed, the device where the computer-readable storage medium is located controls the operation of any one of the apparatuses for generating a diagnosis and treatment plan based on multi-source heterogeneous data.
[0097] According to another aspect of the present application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory, and the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors control the operation of the diagnosis and treatment plan generation device based on multi-source heterogeneous data as described above.
[0098] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0099] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0100] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0101] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0102] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0103] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program code.
[0104] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A device for generating diagnosis and treatment plans based on multi-source heterogeneous data, It is characterized in that include: an acquisition unit, configured to acquire N target medical images of a target object and a modality type and image interpretation information of each target medical image, wherein N is an integer greater than 1, and the N target medical images are images of different sources and structures; an analysis unit, configured to input the target medical image and the image interpretation information of the target medical image into an analysis model corresponding to the modality type of the target medical image, and determine the semantic space understanding features of the target medical image through the analysis model corresponding to the modality type of the target medical image, wherein different modality types correspond to different analysis models, and the semantic space understanding features are used to characterize the association relationship between the image interpretation information of the target medical image and the image information of the target medical image; A determination unit is used to determine a target diagnosis and treatment plan for the target object based on the modality type of the target medical image and the semantic space understanding characteristics, wherein when the determination unit determines the target diagnosis and treatment plan for the target object based on the modality type of the target medical image and the semantic space understanding characteristics, it includes: the determination unit establishes a connection between the semantic space understanding characteristics of target medical images of different modality types, and determines the target diagnosis and treatment plan for the target object based on the connection between the semantic space understanding characteristics of the target medical images of different modalities through a composite network model.
2. The device for generating a diagnosis and treatment plan based on multi-source heterogeneous data according to claim 1, It is characterized in that The acquisition unit comprises: The modality type determination subunit is used to input the target medical image into the classification model, and determine the modality type of the target medical image through the classification model using the prior knowledge learned in the model training stage, wherein the classification model is a neural network model trained based on historical medical images of known modality types.
3. The device for generating a diagnosis and treatment plan based on multi-source heterogeneous data according to claim 2, It is characterized in that The diagnosis and treatment plan generation device based on multi-source heterogeneous data also includes: A training data set acquisition unit, used to acquire L training data sets, where L is an integer greater than 1, and each training data set includes a plurality of training medical images of the same modality type and image interpretation information of each training medical image; The analysis model training unit is used to obtain L analysis models based on the L training data sets, wherein the i-th analysis model among the L analysis models is obtained by training the i-th training data set among the L training data sets.
4. The device for generating a diagnosis and treatment plan based on multi-source heterogeneous data according to claim 3, It is characterized in that The analysis model training unit comprises: a first processing subunit, configured to establish an association relationship between the image interpretation information of the training medical images in the i-th training data set and the training medical images, wherein the training medical images in the i-th training data set are all images of the i-th modality type; The analysis model training subunit is used to input the training medical images in the i-th training data set and the image interpretation information of the training medical images into the neural network, iteratively train the neural network to learn the association between the training medical images and the image interpretation information of the training medical images, and when the neural network meets the iteration end condition, use the neural network that has finished training as the analysis model corresponding to the i-th modality type.
5. The device for generating a diagnosis and treatment plan based on multi-source heterogeneous data according to claim 1, It is characterized in that The determining unit comprises: A natural image spatial feature extraction subunit, configured to extract natural image spatial features from the target medical image, wherein the natural image spatial features at least include texture features, spatial features, shape features, and color features of the target medical image; The diagnosis and treatment plan determination subunit is used to determine the target diagnosis and treatment plan of the target object according to the modality type, semantic space understanding characteristics and natural image space characteristics of the target medical image.
6. The device for generating a diagnosis and treatment plan based on multi-source heterogeneous data according to claim 5, It is characterized in that The diagnosis and treatment plan determination subunit includes: A natural language feature extraction module, used to extract the natural language features of the target medical image from the image interpretation information of the target medical image, wherein the natural language features are used to represent the semantic meaning of the image interpretation information of the target medical image; A diagnosis and treatment plan determination module is used to determine the target diagnosis and treatment plan of the target object based on the modality type, semantic space understanding characteristics, natural image space characteristics and natural language characteristics of the target medical image.
7. The device for generating a diagnosis and treatment plan based on multi-source heterogeneous data according to claim 6, It is characterized in that The diagnosis and treatment plan determination module includes: A first processing submodule is used to use the modality type, semantic space understanding features, natural image space features and natural language features of the target medical image as model input features; A second processing submodule is used to input the model input features into a composite network model, and convert the model input features into feature vectors through the composite network model, wherein the feature vectors are used to represent the features after the modality type, semantic space understanding features, natural image space features and natural language features of the target medical image are fused in a vector form; The third processing submodule is used to determine the target diagnosis and treatment plan for the target object according to the feature vector through the composite network model, wherein the composite network model is a neural network model trained using historical medical images of different modalities as training samples and historical diagnosis and treatment plans formulated for the historical medical images as training labels.
8. The device for generating a diagnosis and treatment plan based on multi-source heterogeneous data according to claim 7, It is characterized in that The diagnosis and treatment plan generation device based on multi-source heterogeneous data also includes: A first processing unit is used to use the modality type, semantic space understanding features, natural image space features, and natural language features of the historical medical images as model training features; The second processing unit is used to input the model training features into the initial neural network model, iteratively train the initial neural network model to learn the correlation between the model training features and the historical diagnosis and treatment plans formulated for the historical medical images, and when the initial neural network model meets the training end condition, the initial neural network model that has finished training is used as the composite network model.
9. The device for generating a diagnosis and treatment plan based on multi-source heterogeneous data according to claim 8, It is characterized in that The composite network model and the classification model are models obtained through joint training, wherein the joint training represents that the data output by the classification model during the training phase is used as the input data required by the composite network model during training, and the number of iterative training times of the composite network model and the number of iterative training times of the classification model influence each other.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program runs, the device where the computer-readable storage medium is located controls the operation of the diagnosis and treatment plan generation device based on multi-source heterogeneous data as described in any one of claims 1 to 9.
11. An electronic device, It is characterized in that It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors control the operation of the diagnosis and treatment plan generation device based on multi-source heterogeneous data as described in any one of claims 1 to 9.
