Effect prediction method, device, storage medium and electronic device

Abstract

The embodiments of the present invention disclose an effect prediction method, device, storage medium, and electronic device. The method includes: obtaining multimodal information of the current object, inputting the multimodal information into an effect prediction system, wherein the effect prediction system includes a feature extraction module, a feature fusion module, and an effect prediction module for each modal information; and obtaining the predicted effect output by the effect prediction system. In the above technical solution, the multimodal information of the current object is input into the effect prediction system, and feature extraction, information fusion, and effect prediction of the multimodal information are realized. Specifically, by extracting and fusion of the multimodal information, the fusion of multiple modal information is realized, thereby improving the accuracy of the prediction effect.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of information processing technology, and in particular to an effect prediction method, device, storage medium, and electronic device. Background Art

[0002] Radiotherapy is one of the main methods of treating cancer, using radiation to kill cancer cells. During radiotherapy, a physicist designs a radiotherapy plan based on the doctor's dose prescription (adjusting the gantry angle, field of view, multi-page raster shape, and radiation dose, etc.). This maximizes the protection of normal organs while ensuring that cancerous tissues receive a sufficient dose of radiation to achieve the goal of curing the cancer.

[0003] In current radiotherapy treatment plans, the selection and verification of different radiotherapy plans often rely on the clinical experience of physicists and doctors. However, human body information is complex and changeable, and existing technologies are difficult to meet the needs of high-accuracy prediction effects. Summary of the Invention

[0004] Embodiments of the present invention provide an effect prediction method, device, storage medium, and electronic device to improve the accuracy of the predicted effect.

[0005] In a first aspect, an embodiment of the present invention provides an effect prediction method, comprising:

[0006] Acquire multimodal information of the current object, and input the multimodal information into an effect prediction system, wherein the effect prediction system includes a feature extraction module for each modal information, a feature fusion module, and an effect prediction module;

[0007] Obtain the predicted effect output by the effect prediction system.

[0008] In a second aspect, an embodiment of the present invention further provides an effect prediction device, comprising:

[0009] An information input module, configured to obtain multimodal information of the current object and input the multimodal information into an effect prediction system, wherein the effect prediction system includes a feature extraction module for each modal information, a feature fusion module, and an effect prediction module;

[0010] The effect acquisition module is used to obtain the predicted effect output by the effect prediction system.

[0011] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising:

[0012] one or more processors;

[0013] a storage device for storing one or more programs,

[0014] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the effect prediction methods described in the embodiments of the present invention.

[0015] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to execute any of the effect prediction methods described in the embodiments of the present invention.

[0016] The present invention obtains multimodal information of the current object and inputs the multimodal information into an effect prediction system, wherein the effect prediction system includes a feature extraction module, a feature fusion module, and an effect prediction module for each modal information; and obtains the predicted effect output by the effect prediction system. In the above technical solution, the multimodal information of the current object is input into the effect prediction system, which realizes feature extraction, information fusion, and effect prediction of the multimodal information. The feature extraction and information fusion of the multimodal information achieves the fusion of multiple modal information, thereby improving the accuracy of the predicted effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings introduced here only illustrate some of the embodiments to be described by the present invention, and are not exhaustive. A person skilled in the art can derive other drawings based on these drawings without inventive effort.

[0018] Figure 1 This is a flow chart of an effect prediction method provided by an embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram of feature extraction provided by an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of an effect prediction system provided by an embodiment of the present invention;

[0021] Figure 4 This is a flow chart of an effect prediction method provided by an embodiment of the present invention;

[0022] Figure 5 This is a flow chart of an effect prediction method provided by an embodiment of the present invention;

[0023] Figure 6 1 is a schematic structural diagram of an effect prediction device provided by an embodiment of the present invention;

[0024] Figure 7 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention.

[0026] It should also be noted that, for ease of description, only the part relevant to the present invention, rather than all of the content, is shown in the accompanying drawings. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processing or methods depicted as flow charts. Although flow charts describe various operations (or steps) as sequential processing, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of various operations can be rearranged. When its operation is completed, the processing can be terminated, but can also have additional steps not included in the accompanying drawings. The processing can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0027] Figure 1 This is a flowchart of an effect prediction method provided by one embodiment of the present invention. This embodiment is applicable to situations where the effect produced by object information is automatically predicted based on the object information. The method can be performed by the effect prediction device provided by the embodiment of the present invention. The device can be implemented by software and / or hardware and can be configured on an electronic computing device, such as a terminal and / or server. Specifically, the method includes the following steps:

[0028] S110 , obtaining multimodal information of the current object, and inputting the multimodal information into an effect prediction system, wherein the effect prediction system includes a feature extraction module for each modal information, a feature fusion module, and an effect prediction module.

[0029] The current object can be a human or an animal, and the multimodal information of the current object can be a combination of information in multiple different modes or types, for example, a combination of text + picture, text + picture + video, etc. In some embodiments, it can also be a combination of information in different formats, such as a combination of TXT format and DOC format, TXT format and JPG format. The multimodal information of the current object is information of multiple categories of the current object, which is used to characterize the characteristics of the current object in different dimensions, and may include but is not limited to the basic information of the current object, the treatment data to be implemented, and the medical examination data. Basic information may include age, gender, eating habits, etc. The treatment data to be implemented includes but is not limited to the treatment means to be implemented and the treatment parameters. For example, the treatment means to be implemented may be but is not limited to radiotherapy, and correspondingly, the treatment parameters include but are not limited to the radiotherapy dose. Medical examination data includes but is not limited to medical record data, medical imaging data, and test result data.

[0030] In an embodiment of the present invention, a method for acquiring multimodal information includes: acquiring multimodal information of a current subject in real time using an information acquisition device. For example, the multimodal information may include multiple vital signs, including but not limited to heart rate, blood pressure, and pulse data. The information acquisition device may be a wearable device or a medical data acquisition device, for example, acquiring the current subject's heart rate in real time using a wearable device. For example, the multimodal information may include multiple medical images, such as CT images and ultrasound images. The information acquisition device may be a medical image acquisition device. In some embodiments, the method for acquiring multimodal information may further include: acquiring information from multiple databases based on an identifier of the current subject, and importing the matched information. The databases may include but are not limited to medical imaging databases, case databases, and vital sign databases. The present invention does not limit the method for acquiring multimodal information. It should be noted that the multimodal information is provided by the current subject or authorized by the current subject.

[0031] The effect prediction system can be used to predict treatment outcomes based on the input multimodal information. The predicted outcome can be a prediction of treatment efficacy, such as the size of a region of interest. Before treatment is performed on the current subject, the effect prediction system is invoked to perform an effect prediction, facilitating pre-treatment evaluation of treatment efficacy and providing a reference for the user.

[0032] The effect prediction system includes a feature extraction module for each modal information, a feature fusion module, and an effect prediction module. Each feature extraction module is used to extract features from the corresponding modal information, the feature fusion module is used to fuse the feature information extracted by each feature extraction module, and the effect prediction module is used to predict the effect based on the fused feature information.

[0033] In other embodiments, each feature extraction module is used to perform multi-level feature extraction on the information of the corresponding modality, and the feature fusion module is used to perform multi-level fusion on the multi-level feature information extracted by each feature extraction module. Accordingly, the effect prediction module is used to perform effect prediction based on the multi-level fused feature information.

[0034] In some optional embodiments, each modality of information in the multimodal information may have a corresponding feature extraction module, such as Figure 2As shown, a feature extraction schematic diagram provided by an embodiment of the present invention is used to extract features of information of each modality separately, and then feature fusion is performed on the extracted feature vectors. The advantage of this arrangement is that features of information of each modality can be extracted separately, and corresponding feature extraction modules can be provided for information of different modalities, so that the extracted feature information can more accurately reflect the content of multimodal information. Optionally, the modality of multimodal information can be pre-set, for example, any type of information can determine a modal information, or the modality to which each information belongs can be determined according to a preset classification method. In some optional embodiments, a feature extraction module can be used to jointly extract features of multiple information, which can reduce the structural complexity of the effect prediction system. The feature extraction module can use a neural network, a decoder, a full attention mechanism module, etc. to perform feature extraction.

[0035] The feature fusion module can be used to fuse the feature information extracted by the feature extraction module. The feature fusion module can complete the fusion of feature information in the form of weight superposition, or the feature fusion module is a random forest model with feature fusion function, or the feature fusion module is a residual neural network with feature fusion function.

[0036] The effect prediction module can be used to perform predictive processing on the fusion information generated by the feature fusion module to obtain a predicted effect. The predicted effect can be a treatment effect. For example, the treatment effect can be in the form of an image or video, such as a medical image. By examining the changes between the region of interest and the normal region in the image or video, the predicted treatment effect can be used as a reference to assist in actual treatment.

[0037] S120: Obtain the predicted effect output by the effect prediction system.

[0038] In an embodiment of the present invention, multimodal information is input into an effect prediction system to obtain a predicted effect. The predicted effect output by the effect prediction system can then be obtained by a terminal or server. In some optional embodiments, the predicted effect output by the effect prediction system can be obtained in real time, making it convenient for users to view and obtain real-time prediction results. In another optional embodiment, the predicted effect output by the effect prediction system can be obtained cumulatively, enabling unified viewing of multiple predicted effects.

[0039] Based on the above embodiment, the multimodal information includes structured information and unstructured information, wherein the effect prediction system includes a feature extraction module for structured information and a feature extraction module for unstructured information.

[0040] Structured information refers to multimodal information that, after analysis, can be broken down into multiple interconnected components, each with a clear hierarchical structure. For example, a factory's records of production, operations, transactions, and customer information all fall under structured information. Unstructured information, as opposed to structured information, is relatively fluid and often takes the form of files in various formats, such as electronic documents, emails, web pages, and videos.

[0041] In an embodiment of the present invention, the feature extraction module in the effect prediction system is divided into a feature extraction module for structured information and a feature extraction module for unstructured information according to the modality of the multimodal information. The feature extraction module for structured information is used to extract feature information from structured information, and the feature extraction module for unstructured information is used to extract feature information from unstructured information. The advantage of this arrangement is that it can provide corresponding feature extraction modules for information of different modalities, so that the extracted features can more accurately reflect the content of the multimodal information.

[0042] Based on the above embodiment, the structured information includes the target subject's medical images and treatment plan; the unstructured information includes the target subject's medical history and current status information; and the predicted effect includes the predicted image after treatment based on the treatment plan. Furthermore, the predicted image can be adjusted based on at least a portion of imaging-related parameters of the target subject's medical images and / or at least a portion of imaging-related parameters of the target subject's medical images after treatment, thereby ensuring that the doctor can eliminate other possible influencing factors when comparing the predicted image with the target subject's medical images and / or the target subject's medical images after treatment.

[0043] The embodiment of the present invention provides a schematic diagram of an effect prediction system, such as Figure 3 As shown, the structured information and the unstructured information are input into the feature extraction module, the feature extraction module extracts features from the structured information and the unstructured information, and then the extracted feature information is input into the feature fusion module, the feature fusion module performs multi-level fusion on the feature information, and then the fused feature information is input into the effect prediction module to obtain a predicted image, which can provide a reference for the actual treatment of the current object.

[0044] It should be noted that training samples for predicted images can include post-treatment medical images. Inputting post-treatment medical images into the effect prediction system for training can improve training accuracy and reduce training difficulty. Furthermore, the predicted effect can include multiple predicted images, each corresponding to a different treatment plan or treatment period. Predicted images from different treatment plans or treatment periods can be compared to more intuitively and accurately demonstrate treatment effects.

[0045] In some embodiments, predicted images for different treatment plans or treatment periods may correspond to different effect prediction modules to ensure the accuracy of the predicted images. Feature extraction modules and feature fusion modules may be reused to avoid model bloat.

[0046] In an embodiment of the present invention, medical images of the target subject may include, but are not limited to, computed tomography (CT), magnetic resonance (MR), ultrasound, and positron emission computed tomography (PET). A treatment plan may be a plan for radiation therapy, with different treatment plans corresponding to different cumulative radiation dose distribution images. Medical history information may include the current subject's medical history, including but not limited to medical records, examination and test results, doctor's orders, surgical records, and nursing records. Current status information includes, but is not limited to, laboratory test results, cancer staging, and hormone levels. Laboratory test results may include, but are not limited to, blood test results, urine test results, and liver function tests. Cancer staging may be divided into early, middle, and late stages, and the tumor node metastasis (TNM) staging system may also be used. Hormone levels may refer to the amount of hormones in the body.

[0047] In an embodiment of the present invention, the predicted effect includes predicted images after treatment based on the treatment plan. The predicted images can be images after the expected treatment, and can include but are not limited to CT images, MR images, and PET images, etc., which can be used to characterize the cumulative effect of the radiation dose in the target area and normal organs after one or more treatments, such as the reduction of the target area, etc. The predicted images can be compared with the imaging data of the patient after actual treatment, and the efficacy can be evaluated more accurately. The predicted effect can also include survival rate, which can be understood as the long-term efficacy evaluation result of the current subject, for example, the current subject's 1-5 year survival expectation, recurrence warning, etc., to provide a reference for the actual treatment of the current subject and improve the quality of actual treatment.

[0048] On the basis of the above embodiment, the treatment plan includes the dose distribution of the preset number of treatments; the predicted effect is the corresponding predicted effect after the preset number of treatments.

[0049] The preset number of treatments is a simulated number of treatments, which can be one or more. When the preset number of treatments is multiple, the treatment plan can be the sum of the doses used in multiple treatments. The predicted effect is the predicted effect after the preset number of treatments, that is, the predicted effect is the final result of multiple treatments.

[0050] In an embodiment of the present invention, in some embodiments, the predicted effect of each simulated treatment can be displayed in sequence, so that the user can view the predicted effects of different treatment stages. In some embodiments, the predicted effect corresponding to the final preset number of treatments can be displayed, so that the user can view the final predicted results.

[0051] An embodiment of the present invention provides an effect prediction method, which obtains multimodal information of a current object and inputs the multimodal information into an effect prediction system, wherein the effect prediction system includes a feature extraction module, a feature fusion module, and an effect prediction module for each modal information; and obtains a predicted effect output by the effect prediction system. In the above technical solution, the multimodal information of the current object is input into the effect prediction system, which implements feature extraction, information fusion, and effect prediction of the multimodal information. The feature extraction and information fusion of the multimodal information achieves the fusion of multiple modal information, thereby improving the accuracy of the predicted effect.

[0052] Figure 4 This is a flow chart of an effect prediction method provided by an embodiment of the present invention, which can be combined with the various optional solutions in the above embodiments. In the embodiment of the present invention, optionally, the feature fusion module is used to perform weighted fusion on the feature information extracted by the feature extraction module.

[0053] like Figure 4 As shown, the method of the embodiment of the present invention specifically includes the following steps:

[0054] S210. Obtain multimodal information of the current object and input the multimodal information into an effect prediction system, wherein the effect prediction system includes a feature extraction module, a feature fusion module and an effect prediction module for each modal information; the feature fusion module is used to perform weighted fusion on the feature information extracted by the feature extraction module.

[0055] In this embodiment, the feature fusion module fuses the feature information extracted by each feature extraction module through weighted processing. Optionally, each feature information is weighted based on the weight of each feature information. The weight corresponding to each feature information is the weight corresponding to the multimodal information to which the feature information belongs. Different information in the multimodal information corresponds to different weights. By setting different weights, the weight of effective information is increased and the weight of unnecessary information is reduced, thereby improving the accuracy of the fused features and further improving the prediction accuracy.

[0056] Among them, if any modal information is empty, the weight of the feature information corresponding to the modal information is set to a preset weight, and the preset weight is smaller than the corresponding weight when the modal information is not empty.

[0057] Exemplarily, weight fusion of the feature information extracted by the feature extraction module can be performed by assigning weights to multiple feature information. If the modality of the feature information is empty, the weight corresponding to the feature information is set to a preset weight, for example, the preset weight can be set to 0. The advantage of such a setting is that when one or more pieces of information are missing, the impact of the missing information on the prediction accuracy can be reduced, while still achieving normal training, parameter adjustment, and prediction of the entire effect prediction system.

[0058] S220: Obtain the predicted effect output by the effect prediction system.

[0059] An embodiment of the present invention provides an effect prediction method, which obtains multimodal information of the current object and inputs the multimodal information into an effect prediction system, wherein the effect prediction system includes a feature extraction module, a feature fusion module and an effect prediction module for each modal information; the feature fusion module is used to perform weighted fusion on the feature information extracted by the feature extraction module, so as to achieve normal training, parameter adjustment and prediction of the entire effect prediction system when one or more pieces of information are missing.

[0060] Figure 5 The flowchart of an effect prediction method provided by one embodiment of the present invention is a schematic diagram. The embodiment of the present invention can be combined with the various optional solutions in the above embodiments. In the embodiment of the present invention, optionally, the feature fusion module is used to fuse the multi-level features corresponding to the multimodal information.

[0061] like Figure 5 As shown, the method of the embodiment of the present invention specifically includes the following steps:

[0062] S310. Obtain multimodal information of the current object and input the multimodal information into an effect prediction system, wherein the effect prediction system includes a feature extraction module, a feature fusion module and an effect prediction module for each modal information, and the feature fusion module is used to fuse the multi-level features corresponding to the multimodal information.

[0063] In the embodiments of the present invention, in some embodiments, the multi-level feature fusion processing can be the fusion of feature information at different levels. In some embodiments, the multi-level feature fusion processing can also be the fusion of feature information at the same level without fusion between multiple levels. In some embodiments, the multi-level feature fusion processing can also be the fusion of feature information at the same level with further fusion between multiple levels. The embodiments of the present invention do not impose any restrictions on this.

[0064] Multi-level feature fusion processing is a local and global modeling idea. Multi-level features can include but are not limited to low-level features and high-level features. Among them, low-level features are more suitable for representing objects with simple appearance and concrete features, such as contour features; high-level features are suitable for objects with complex appearance and abstract features, such as semantic features. The feature fusion module can flexibly configure the weights of different levels of features and different modal information in multimodal information, which helps to integrate structured information and unstructured information in multimodal information and improve the accuracy of the prediction effect of the effect prediction system. In addition, multi-level feature fusion facilitates the decoupling between different levels of features and different modal information, so that when some secondary modal information is missing, the effect prediction system can still maintain a high prediction accuracy, making the predicted image closer to the actual treatment effect image.

[0065] On the basis of the above embodiment, the feature fusion module is used to fuse the feature information of the same level corresponding to the multimodal information, and input the fused feature information of each level into the effect prediction module.

[0066] Exemplarily, the feature information extracted by the feature extraction module may include feature information of multiple levels. Exemplarily, the feature extraction module may include multiple feature extraction layers, each feature extraction layer outputs feature information, and inputs the output feature information to the next feature extraction layer, and different feature extraction layers output feature information of different levels. The feature fusion module fuses the feature information of the same level, specifically by weight fusion, or by using machine learning methods such as random forests for fusion, or by using residual neural networks for fusion. The embodiment of the present invention does not limit this, and inputs the fused feature information of each level into the effect prediction module to achieve multi-level fusion of multimodal information, and use multiple information as the basis for prediction, so as to make the prediction effect more accurate.

[0067] Based on the above embodiment, the feature fusion module is used to fuse the feature information of the same level corresponding to the multimodal information, and perform multi-level fusion on the fused feature information of each level to obtain target feature information and input it into the effect prediction module.

[0068] Exemplarily, the feature fusion module fuses the feature information of the same level, and performs multi-level fusion on the fused feature information of each level to obtain target feature information, and inputs the target feature information into the effect prediction module. Specifically, it can be fused by weight, or by using machine learning methods such as random forest, or by using residual neural network, which is not limited in the embodiment of the present invention. This embodiment realizes the multi-level fusion of multimodal information and uses multiple types of information as the basis for prediction, thereby making the prediction effect more accurate.

[0069] S320: Obtain the predicted effect output by the effect prediction system.

[0070] An embodiment of the present invention provides an effect prediction method, which obtains multimodal information of the current object and inputs the multimodal information into an effect prediction system, wherein the effect prediction system includes a feature extraction module, a feature fusion module and an effect prediction module for each modal information. The feature fusion module is used to fuse the multi-level features corresponding to the multimodal information, thereby realizing multi-level fusion of the multimodal information. Furthermore, the feature fusion module can flexibly configure the weights of different-level features and multimodal information, which is helpful for integrating multimodal information and improving the accuracy of the prediction effect of the effect prediction system. In addition, multi-level feature fusion facilitates the decoupling between different-level features and multimodal information, so that when some secondary modal information is missing, the effect prediction system can still maintain a high prediction accuracy.

[0071] Figure 6 This is a schematic diagram of the structure of an effect prediction device according to one embodiment of the present invention. The effect prediction device provided in this embodiment can be implemented using software and / or hardware and can be configured in a terminal and / or server to implement the effect prediction method according to the embodiment of the present invention. Specifically, the device may include an information input module 410 and an effect acquisition module 420.

[0072] Among them, the information input module 410 is used to obtain multimodal information of the current object and input the multimodal information into the effect prediction system, wherein the effect prediction system includes a feature extraction module, a feature fusion module and an effect prediction module for each modal information; the effect acquisition module 420 is used to obtain the predicted effect output by the effect prediction system.

[0073] An embodiment of the present invention provides an effect prediction device that obtains multimodal information of a current object and inputs the multimodal information into an effect prediction system, wherein the effect prediction system includes a feature extraction module, a feature fusion module, and an effect prediction module for each modal information; and obtains a predicted effect output by the effect prediction system. In the above technical solution, the multimodal information of the current object is input into the effect prediction system, which implements feature extraction, information fusion, and effect prediction of the multimodal information. The feature extraction and information fusion of the multimodal information achieves the fusion of multiple modal information, thereby improving the accuracy of the predicted effect.

[0074] Based on any optional technical solution in the embodiments of the present invention, optionally, the multimodal information includes structured information and unstructured information, wherein the effect prediction system includes a feature extraction module for structured information and a feature extraction module for unstructured information.

[0075] Based on any optional technical solution in the embodiment of the present invention, optionally, the structured information includes medical images and treatment plans of the target object; the unstructured information includes medical history information and current status information of the target object;

[0076] The predicted effect includes a predicted image after treatment based on the treatment plan.

[0077] Based on any optional technical solution in the embodiments of the present invention, optionally, the treatment plan includes a dose distribution for a preset number of treatments; and the predicted effect is a corresponding predicted effect after the preset number of treatments.

[0078] Based on any optional technical solution in the embodiment of the present invention, optionally, the feature fusion module is used to perform weight fusion on the feature information extracted by the feature extraction module;

[0079] Among them, if any modal information is empty, the weight of the feature information corresponding to the modal information is set to a preset weight, wherein the preset weight is smaller than the corresponding weight when the modal information is not empty.

[0080] Based on any optional technical solution in the embodiments of the present invention, optionally, the feature fusion module is used to perform fusion processing on the multi-level features corresponding to the multimodal information.

[0081] Based on any optional technical solution in the embodiment of the present invention, optionally, the feature fusion module is used to fuse feature information of the same level corresponding to the multimodal information, and input the fused feature information of each level into the effect prediction module;

[0082] or,

[0083] The feature fusion module is used to fuse the feature information of the same level corresponding to the multimodal information, and to perform multi-level fusion on the fused feature information of each level to obtain target feature information which is input into the effect prediction module.

[0084] The effect prediction device provided in the embodiment of the present invention can execute the effect prediction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0085] Figure 7 The present invention provides a schematic structural diagram of an electronic device according to an embodiment of the present invention. Figure 7 A block diagram of an exemplary electronic device 12 suitable for implementing embodiments of the present invention is shown. Figure 7 The electronic device 12 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present invention.

[0086] like Figure 7 As shown, electronic device 12 is implemented as a general-purpose computing device. Components of electronic device 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that connects various system components (including system memory 28 and processing unit 16).

[0087] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0088] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0089] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 7 Not shown, often called a "hard drive"). Although Figure 7 Not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0090] A program / utility 36 having a set (at least one) of program modules 26 may be stored, for example, in system memory 28. Such program modules 26 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 26 generally perform the functions and / or methods of the embodiments described herein.

[0091] The electronic device 12 may also communicate with one or more external devices 14 (e.g., a keyboard, a pointing device, a display 24, etc.), one or more devices that enable a user to interact with the electronic device 12, and / or any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed through an input / output (I / O) interface 22. Furthermore, the electronic device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 20. Figure 7 As shown, the network adapter 20 communicates with other modules of the electronic device 12 via the bus 18. Figure 7 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0092] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing an effect prediction method provided by an embodiment of the present invention.

[0093] An embodiment of the present invention further provides a storage medium containing computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are used to perform an effect prediction method, the method comprising:

[0094] Acquire multimodal information of the current object, and input the multimodal information into an effect prediction system, wherein the effect prediction system includes a feature extraction module for each modal information, a feature fusion module, and an effect prediction module;

[0095] Obtain the predicted effect output by the effect prediction system.

[0096] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0097] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0098] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0099] The computer program code for performing the operations of the embodiments of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0100] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for predicting an effect, characterized in that: include: Acquire multimodal information of the current object, and input the multimodal information into an effect prediction system, wherein the effect prediction system includes a feature extraction module for each modal information, a feature fusion module, and an effect prediction module; Obtaining the predicted effect output by the effect prediction system; The multimodal information includes structured information and unstructured information, wherein the effect prediction system includes a feature extraction module for structured information and a feature extraction module for unstructured information; the structured information includes medical images and a treatment plan of the target subject, wherein the treatment plan is a plan for treatment via radiation; the unstructured information includes medical history information and current status information of the target subject, wherein the current status information includes laboratory test results, cancer stage, and hormone levels; the predicted effect includes predicted images and survival rate after treatment based on the treatment plan; Each modality of information in the multimodal information has a corresponding feature extraction module, each feature extraction module is used to perform multi-level feature extraction on the information of the corresponding modality separately, the feature fusion module is used to perform multi-level fusion on the multi-level feature information extracted separately by each feature extraction module, and the effect prediction module is used to perform effect prediction based on the multi-level fused feature information; The feature fusion module is further used to perform weight fusion on the feature information extracted by the feature extraction module, assign weights to multiple feature information, and if the modality of any feature information is empty, set the weight corresponding to the feature information to a preset weight.

2. The method according to claim 1, characterized in that The treatment plan includes a dose distribution for a preset number of treatments; and the predicted effect is the corresponding predicted effect after the preset number of treatments.

3. The method according to claim 1, characterized in that The feature fusion module is used to fuse the multi-level features corresponding to the multi-modal information.

4. The method according to claim 3, characterized in that The feature fusion module is used to fuse the feature information of the same level corresponding to the multimodal information, and input the fused feature information of each level into the effect prediction module; or, The feature fusion module is used to fuse the feature information of the same level corresponding to the multimodal information, and to perform multi-level fusion on the fused feature information of each level to obtain target feature information which is input into the effect prediction module.

5. An effect prediction device, characterized in that: include: An information input module, configured to obtain multimodal information of the current object and input the multimodal information into an effect prediction system, wherein the effect prediction system includes a feature extraction module for each modal information, a feature fusion module, and an effect prediction module; An effect acquisition module, used to obtain the predicted effect output by the effect prediction system; The multimodal information includes structured information and unstructured information, wherein the effect prediction system includes a feature extraction module for structured information and a feature extraction module for unstructured information; the structured information includes medical images and a treatment plan of the target subject, wherein the treatment plan is a plan for treatment via radiation; the unstructured information includes medical history information and current status information of the target subject, wherein the current status information includes laboratory test results, cancer stage, and hormone levels; the predicted effect includes predicted images and survival rate after treatment based on the treatment plan; Each modality of information in the multimodal information has a corresponding feature extraction module, each feature extraction module is used to perform multi-level feature extraction on the information of the corresponding modality separately, the feature fusion module is used to perform multi-level fusion on the multi-level feature information extracted separately by each feature extraction module, and the effect prediction module is used to perform effect prediction based on the multi-level fused feature information; The feature fusion module is further used to perform weight fusion on the feature information extracted by the feature extraction module, assign weights to multiple feature information, and if the modality of any feature information is empty, set the weight corresponding to the feature information to a preset weight.

6. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the effect prediction method according to any one of claims 1 to 4.

7. A storage medium containing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to perform the effect prediction method according to any one of claims 1 to 4.

Images