Method and System for Evaluating Tumor Treatment Efficacy Based on Medical Images

Abstract

The present invention relates to a method and system for evaluating the efficacy of tumors based on medical images. The method includes: acquiring first PET-CT images and second PET-CT images of a hematological tumor patient before and after treatment, and determining lesion change information of the lesion area based on the first PET-CT images and the second PET-CT images; performing maximum intensity projection on the first PET-CT images and / or the second PET-CT images along different directions to generate maximum intensity projection images from multiple perspectives; determining global image information of the lesion area based on the maximum intensity projection images from multiple perspectives; determining a Deauville score corresponding to the efficacy of the tumor patient based on the global image information; and fusing the lesion change information and the Deauville score to generate tumor efficacy evaluation information.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of assisted medical technology, and particularly to a method and system for evaluating the efficacy of tumors based on medical images. Background Art

[0002] Currently, lymphoma is one of the most common malignant tumors in the hematological system. During the clinical treatment of tumors such as lymphoma, regular evaluation of the treatment efficacy is of great significance. It can not only timely evaluate the effectiveness of the current treatment plan, but also help predict the future development trend of the tumor, providing a basis for improving the subsequent treatment plan.

[0003] With the development and popularization of medical imaging, the method for evaluating the efficacy based on PET / CT images has become the main imaging evaluation method for lymphoma. By observing PET / CT images, radiologists can carefully compare and analyze the location, size, shape of lymphoma lesions before and after treatment and their relationship with surrounding tissues, and thus evaluate the treatment effect of patients. However, the manual evaluation of the efficacy of lymphoma by radiologists has a large workload, takes a long time, and the evaluation consistency among different radiologists is poor, reducing the accuracy and efficiency of lymphoma efficacy evaluation.

[0004] In related technologies, in order to improve the accuracy and efficiency of lymphoma efficacy evaluation, artificial intelligence technology has been introduced into the field of medical imaging, and the method for intelligent and automated efficacy evaluation of medical image analysis through technologies such as deep learning has gradually become the mainstream. However, the development of artificial intelligence technology in the field of automatic lymphoma efficacy evaluation is still slow. For example, most related technologies are limited to single links such as lesion segmentation and property identification in PET / CT images, resulting in the need to further improve the accuracy of efficacy evaluation; in addition, the methods based on deep learning often lack good interpretability and cannot provide clear and specific evaluation information for clinicians (different from radiologists) and patients, with low clinical practicability, restricting their application in clinical diagnosis and treatment. Summary of the Invention

[0005] To solve the above technical problems or at least partially solve the above technical problems, embodiments of the present invention provide a method and system for evaluating the efficacy of tumors based on medical images.

[0006] In a first aspect, embodiments of the present invention provide a method for evaluating the efficacy of tumors based on medical images, including:

[0007] Obtain the first PET-CT image and the second PET-CT image of a patient with a hematological system tumor before and after treatment, and determine the lesion change information of the lesion area based on the first PET-CT image and the second PET-CT image;

[0008] Performing maximum intensity projection on the first PET-CT image and / or the second PET-CT image along different directions to generate maximum intensity projection images of multiple perspectives; determining global image information of the lesion area based on the maximum intensity projection images of the multiple perspectives;

[0009] Determining the Deauville score corresponding to the treatment effect of the tumor patient based on the global image information; fusing the lesion change information and the Deauville score to generate tumor treatment effect evaluation information.

[0010] In one embodiment, the determining global image information of the lesion area based on the maximum intensity projection images of the multiple perspectives includes:

[0011] For the maximum intensity projection image of each perspective, extracting image feature information by using the corresponding feature extraction model; wherein, the feature extraction models corresponding to the maximum intensity projection images of each perspective are different and are pre-trained from the original machine learning model based on the sample maximum intensity projection images;

[0012] Fusing the image feature information corresponding to the maximum intensity projection images of each perspective to obtain the fused global image information.

[0013] In one embodiment, the different directions include at least two of the coronal direction, the sagittal direction, the right anterior lateral direction, and the left anterior lateral direction.

[0014] In one embodiment, the determining the Deauville score corresponding to the treatment effect of the tumor patient based on the global image information includes:

[0015] Inputting the global image information into the target regression prediction model to obtain the corresponding Deauville score; wherein, the target regression prediction model is pre-trained from the original regression prediction model based on the sample global image information.

[0016] In one embodiment, the determining the lesion change information of the lesion area based on the first PET-CT image and the second PET-CT image includes:

[0017] Respectively segmenting and determining the first lesion area image and the second lesion area image from the first PET-CT image and the second PET-CT image;

[0018] Performing image registration on the first lesion area image and the second lesion area image, and determining the lesion change information of the lesion area based on the registered first lesion area image and the second lesion area image, where the lesion change information includes the change information of the position distribution, quantity size, morphological boundary, and maximum standardized uptake value SUVmax of the lesion.

[0019] In one embodiment, the fusion of the lesion change information and the Deauville score to generate tumor treatment efficacy evaluation information includes:

[0020] Combining the lesion change information and the Deauville score to generate input text information; performing feature encoding on the input text information to obtain a semantic vector;

[0021] Based on the semantic vector, retrieving from an expert knowledge vector library domain knowledge texts whose semantic similarity to the semantic vector is greater than a preset threshold; wherein, the expert knowledge vector library is a pre-constructed knowledge vector library related to the evaluation of the efficacy of hematological malignancies, and contains a plurality of documents related to the evaluation of the efficacy of hematological malignancies that are pre-encoded for text indexing;

[0022] Fusing the domain knowledge texts with the input text information to obtain target text information; inputting the target text information into a large language model to obtain tumor treatment efficacy evaluation information.

[0023] In one embodiment, the target text information is used to provide professional prompt information for the large language model, so that the large language model outputs accurate and clinically understandable treatment efficacy evaluation information.

[0024] In a second aspect, an embodiment of the present invention provides a tumor treatment efficacy evaluation system based on medical images, including:

[0025] A lesion change determination module, configured to obtain a first PET-CT image and a second PET-CT image of a patient with hematological malignancies before and after treatment, and determine lesion change information of the lesion area based on the first PET-CT image and the second PET-CT image;

[0026] A global information determination module, configured to perform maximum intensity projection on the first PET-CT image and / or the second PET-CT image along different directions to generate maximum intensity projection images of multiple perspectives; determine global image information of the lesion area based on the maximum intensity projection images of multiple perspectives;

[0027] A treatment efficacy evaluation generation module, configured to determine the Deauville score corresponding to the treatment efficacy of the tumor patient based on the global image information; fuse the lesion change information and the Deauville score to generate tumor treatment efficacy evaluation information.

[0028] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the tumor treatment efficacy evaluation method based on medical images in any of the above embodiments.

[0029] In a fourth aspect, an embodiment of the present invention provides an electronic device, including:

[0030] a processor; and

[0031] a memory for storing a computer program;

[0032] wherein, the processor is configured to execute the tumor treatment efficacy evaluation method based on medical images according to any one of the above embodiments by executing the computer program.

[0033] The technical solution provided by the embodiment of the present invention has the following advantages compared with the prior art:

[0034] The tumor treatment efficacy evaluation method and system based on medical images provided by the embodiment of the present invention obtain the first PET-CT image and the second PET-CT image of a hematological system tumor patient before and after treatment, and determine the lesion change information of the lesion area based on the first PET-CT image and the second PET-CT image; perform maximum intensity projection (MIP) on the first PET-CT image and / or the second PET-CT image along different directions to generate maximum intensity projection images of multiple perspectives; determine the global image information of the lesion area based on the maximum intensity projection images of multiple perspectives; determine the Deauville score corresponding to the treatment efficacy of the tumor patient based on the global image information; fuse the lesion change information and the Deauville score to generate tumor treatment efficacy evaluation information. In this way, the solution of this embodiment accurately evaluates the Deauville score by mining the global lesion information related to the lesion in the PET-CT image based on the multi-perspective MIP map, and at the same time fuses the lesion change information obtained from the pre- and post-PET-CT images, that is, the local lesion information, so as to more comprehensively and accurately analyze the PET-CT image features, thereby improving the accuracy of the treatment efficacy evaluation of hematological system tumor patients such as lymphoma patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present invention and used together with the specification to explain the principles of the present invention.

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0037] Figure 1 is a flowchart of the tumor treatment efficacy evaluation method based on medical images according to the embodiment of the present invention;

[0038] Figure 2 is a schematic diagram of different directions of the maximum intensity projection of the image according to the embodiment of the present invention;

[0039] Figure 3 Flowchart of the global image information acquisition method according to an embodiment of the present invention;

[0040] Figure 4 Flowchart of the tumor treatment efficacy evaluation method based on medical images according to another embodiment of the present invention;

[0041] Figure 5 Flowchart of the tumor treatment efficacy evaluation method based on medical images according to still another embodiment of the present invention;

[0042] Figure 6 Schematic diagram of the tumor treatment efficacy evaluation system based on medical images according to an embodiment of the present invention. Detailed implementation manners

[0043] In order to more clearly understand the above objects, features and advantages of the present invention, the solutions of the present invention will be further described below. It should be noted that, without conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0044] In the following description, many specific details are set forth in order to fully understand the present invention, but the present invention can also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only a part of the embodiments of the present invention, rather than all the embodiments.

[0045] It should be understood that in the following text, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally means that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or its similar expression means any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0046] In the related art, first, quantitative analysis such as automatic localization, segmentation, and property identification of lesions in PET / CT images is required, and the Deauville Score (DS) is evaluated based on the changes in the lesions before and after treatment. Specifically, for example, the efficacy evaluation of lymphoma is performed according to the set rules. However, this method has a high complexity. In particular, the accuracy of the Deauville Score evaluation highly depends on the accuracy of the single result of the quantitative analysis of the lesion area, resulting in poor feasibility and stability of this method, thus leading to poor accuracy of the Deauville Score evaluation, and further making the accuracy of the efficacy evaluation to be further improved.

[0047] Figure 1 The following is a flowchart of a tumor efficacy evaluation method based on medical images according to an embodiment of the present invention. This method can be executed by a computing device such as a computer or a mobile terminal, and specifically may include the following steps:

[0048] Step S101: Obtain the first PET-CT image and the second PET-CT image of a hematological tumor patient before and after treatment, and determine the lesion change information of the lesion area based on the first PET-CT image and the second PET-CT image.

[0049] Exemplarily, the hematological tumor may be lymphoma, but is not limited thereto. The first PET-CT image and the second PET-CT image may be the PET-CT images of the patient at two adjacent times before and after treatment, such as the previous examination and the reexamination, but are not limited thereto. These images can be stored in the hospital information system for retrieval and analysis when needed.

[0050] In one embodiment, in step S101, determining the lesion change information of the lesion area based on the first PET-CT image and the second PET-CT image includes: respectively segmenting and determining the first lesion area image and the second lesion area image from the first PET-CT image and the second PET-CT image; performing image registration on the first lesion area image and the second lesion area image, and determining the lesion change information of the lesion area based on the registered first lesion area image and the second lesion area image. The lesion change information includes the change information of the position distribution, quantity, size, morphological boundary, and maximum standardized uptake value SUVmax of the lesion.

[0051] As an example, in order to obtain local lesion information related to the efficacy evaluation of lymphoma, i.e., lesion change information, in the pre- and post-treatment PET-CT images, in this embodiment, the Lugano criteria for lymphoma staging and efficacy evaluation, which are currently internationally recognized, can be referred to for measuring the lesion changes before and after treatment. First, the lesion regions can be segmented from the PET-CT images, and the locations, benign or malignant natures of different lesions can be further determined, so as to obtain information such as the location distribution, quantity, size, morphological boundary, maximum standardized uptake value (SUVmax) of lymphoma lesions. Specifically, methods based on feature comparison, deep learning, etc. can be used to achieve automatic segmentation, positioning, and qualitative determination of the above-mentioned lesions. Then, in order to accurately obtain the lesion changes in the pre- and post-treatment PET-CT images, image registration can be further performed on the images (i.e., local images) segmented from the pre- and post-treatment PET-CT images. Specifically, methods based on key point detection, deep learning, etc. can be used to achieve the above image registration process. Finally, by matching the corresponding lesions on the registered PET / CT images, the changes in information such as the location, quantity, size, morphological boundary, SUVmax, etc. of different lesions before and after treatment are statistically obtained, so as to accurately measure the changes in different lesions before and after treatment, i.e., lesion change information, which is beneficial to improving the accuracy of the final efficacy evaluation.

[0052] Step S102: Perform maximum intensity projection on the first PET-CT image and / or the second PET-CT image along different directions to generate maximum intensity projection (MIP) images from multiple perspectives; determine the global image information of the lesion region based on the maximum intensity projection images from the multiple perspectives.

[0053] Exemplarily, in one embodiment, the different directions include at least two of the coronal direction, sagittal direction, right anterolateral direction, and left anterolateral direction. Refer to Figure 2 As shown, in a specific example, the different directions may include, but are not limited to, a total of 4 directions: the coronal direction, sagittal direction, right anterolateral direction, and left anterolateral direction. In this embodiment, multiple MIP images from different perspectives are generated from the pre- and post-treatment PET images, and the subsequent automatic evaluation of Deauville scores is realized by combining the information from different perspectives, so as to enhance the stability of the evaluation. At the same time, the errors caused by links such as lesion quantitative analysis and background metabolism calculation in the related art are avoided, and the accuracy of the evaluation results is improved.

[0054] Specifically, the three-dimensional PET image is converted into an MIP image by means of maximum intensity projection. As Figure 2As shown, in addition to the conventional coronal and sagittal projections, the PET images are also subjected to maximum intensity projection in directions such as the right anterior lateral position and the left anterior lateral position, so as to obtain multi-view MIP images. In this way, the lymphoma lesions can be presented from different perspectives, so as to more comprehensively display the global distribution of the lesions in the patient before and after treatment, so that the global image information of the determined lesion area is more accurate.

[0055] Step S103: Determine the Deauville score corresponding to the treatment effect of the tumor patient based on the global image information; fuse the lesion change information and the Deauville score to generate tumor treatment effect evaluation information.

[0056] Exemplarily, in this embodiment, the evaluation of the lymphoma treatment effect not only depends on the analysis of the changes in the local lesions before and after, but also depends on the analysis of the global PET-CT images. It is necessary to fuse the local image information, that is, the lesion change information and the global image information to effectively evaluate the treatment effect. Specifically, based on the multi-view MIP images, the accurate global image information related to the lesions in the PET images can be mined, so as to realize the accurate evaluation of the Deauville score, that is, the Deauville score can be accurately evaluated based on the multi-view MIP images. Finally, fuse the local information and the global information. For example, by extracting and quantitatively analyzing the lymphoma lesions in the PET-CT images to obtain the local information related to the treatment effect, and by using the multi-view MIP images to mine the global information in the PET images to accurately evaluate the Deauville score. In this way, fusing the local and global information can more comprehensively and accurately analyze the PET-CT image features, and enhance the accuracy and stability of the lymphoma treatment effect evaluation.

[0057] The solution of this embodiment is based on the multi-view MIP maps to mine the global lesion information related to the lesions in the PET-CT images to realize the accurate evaluation of the Deauville score, and at the same time fuse the lesion change information, that is, the lesion local information, obtained from the previous and subsequent PET-CT images. In this way, the PET-CT image features can be analyzed more comprehensively and accurately, so as to improve the accuracy of the treatment effect evaluation of patients with hematological malignancies such as lymphoma patients.

[0058] In one embodiment, determining the global image information of the lesion area based on the maximum intensity projection images of the multiple perspectives in step S102 includes: for the maximum intensity projection image of each perspective, using the corresponding feature extraction model to extract the image feature information; wherein, the feature extraction models corresponding to the maximum intensity projection images of each perspective are different and are pre-trained from the original machine learning model based on the sample maximum intensity projection images; fuse the image feature information corresponding to the maximum intensity projection images of each perspective to obtain the fused global image information.

[0059] Specifically, as an example, refer to Figure 3As shown, the feature extraction model can be an image encoder. Specifically, the image encoder can be obtained by training the original machine learning model with the sample maximum density projection image. The original machine learning model can at least include a convolutional neural network model or a Transformer model, but is not limited thereto. Exemplarily, the present invention extracts global image feature information from multi-view MIP images and fuses the image feature information of different-view MIP images, so as to achieve automatic and accurate prediction of subsequent Deauville scores. As Figure 3 shown, first, global image features related to the Deauville score are extracted from the MIP images of different views by each image encoder. Then, the image features of different views are fused together, and this fusion process can be achieved by means such as addition and merging along the channel direction. The global image information obtained in this way is more accurate, making the final efficacy evaluation more accurate.

[0060] Based on the above embodiments, in one embodiment, determining the Deauville score corresponding to the efficacy of a tumor patient based on the global image information includes: inputting the global image information into a target regression prediction model to obtain the corresponding Deauville score; wherein, the target regression prediction model is obtained by pre-training the original regression prediction model based on the sample global image information.

[0061] Exemplarily, regression prediction of the Deauville score is performed using the above-obtained multi-view fusion global image information. Specifically, the original regression prediction model can adopt a support vector machine, a multi-layer perceptron, etc. It can be trained in an end-to-end manner to obtain the target regression prediction model, and in the application stage, the automatic evaluation of the Deauville score based on the model is realized according to the fused global image information corresponding to the multi-view MIP images. In this way, a more accurate Deauville score can be obtained, making the final efficacy evaluation more accurate.

[0062] Based on any of the above embodiments, in one embodiment, with reference to Figure 4 shown, the step of fusing the lesion change information and the Deauville score in step S103 to generate tumor efficacy evaluation information can specifically include the following steps:

[0063] Step S301: Combine the lesion change information and the Deauville score to generate input text information; perform feature encoding on the input text information to obtain a semantic vector;

[0064] Step S302: Based on the semantic vector, retrieve from the expert knowledge vector library the domain knowledge texts whose semantic similarity to the semantic vector is greater than a preset threshold; wherein, the expert knowledge vector library is a pre-constructed knowledge vector library related to the efficacy evaluation of hematological malignancies, and contains a plurality of documents related to the efficacy evaluation of hematological malignancies that are pre-encoded for text indexing.

[0065] Step S303: Integrate the domain knowledge text with the input text information to obtain target text information; input the target text information into a large language model to obtain tumor efficacy evaluation information.

[0066] In one embodiment, the target text information is used to provide professional prompt information for the large language model, so that the large language model outputs accurate and clinically understandable efficacy evaluation information. The large language model can be large language models provided by various manufacturers on the market, such as ChatGPT.

[0067] Exemplarily, as shown in Figure 5 the present invention further proposes an efficacy evaluation generation method based on RAG (Retrieval-augmented Generation) and a large language model to enhance, for example, the accuracy, professionalism, and interpretability of lymphoma efficacy evaluation, so that the efficacy evaluation has good interpretability and can provide clear and specific evaluation information for clinicians (different from radiologists) and patients, facilitating its wide application in clinical diagnosis and treatment.

[0068] Specifically, the present invention first combines the obtained local and global information to obtain the original input text information, which includes descriptions of the number, size, location, morphology, SUVmax, etc. of local lesions before and after treatment, as well as the Deauville score of the patient. Next, in order to enable the large language model to better understand and analyze the treatment effect of lymphoma, the present invention constructs a professional knowledge base for lymphoma treatment effect evaluation. Specifically, a large number of domain documents related to lymphoma treatment effect evaluation are first collected, such as relevant typical cases, international standards, diagnosis and treatment guidelines, etc., and these document information are encoded and indexed through a text encoder, so as to construct an expert knowledge vector library related to lymphoma treatment effect evaluation. Then, after the original input text information is feature-encoded to obtain a semantic vector, domain knowledge related to treatment effect evaluation is retrieved in the expert knowledge vector library according to semantic similarity, and it is merged with the original input text information to obtain the model prompt text information integrated with professional knowledge, that is, the target text information. Specifically, the prompt text information not only includes the local lesions and global imaging information of the current patient, but also incorporates professional information such as case records and relevant diagnosis and treatment standards similar to the patient's situation, so as to provide more comprehensive and professional prompt information for the large language model. Finally, the model prompt text information integrated with professional knowledge is input into the large language model, and the lymphoma treatment effect evaluation information can be generated. As Figure 5 shown, the lymphoma treatment effect evaluation information not only includes the final evaluation result, but also analyzes and interprets information such as the changes in local lesions and Deauville score before and after treatment according to the input patient situation and professional knowledge, so as to provide treatment effect evaluation content that is easily understood clinically. In addition, by retrieving relevant domain knowledge from the professional knowledge base, it is not only possible to effectively enhance the professionalism, authenticity and stability of the content generated by the large language model, but also facilitate the private domain management of various sensitive data in the medical field, ensuring the security and privacy protection of medical data.

[0069] In this embodiment, the large language model is used to summarize and analyze the extracted local and global information, and professional domain knowledge is introduced through retrieval-augmented generation to reduce the generation of irrelevant information to enhance data accuracy and security, and output accurate, professional and highly interpretable lymphoma treatment effect evaluation results.

[0070] The large language models in the prior art can analyze and answer medical-related questions to assist clinical decision-making and improve clinical diagnosis and treatment efficiency. However, in specific professional fields such as lymphoma treatment effect evaluation, it is still prone to generate outputs that are irrelevant or untrue to the questions, restricting its application in clinical diagnosis and treatment. The above-mentioned solution in this embodiment can output accurate, professional and highly interpretable lymphoma treatment effect evaluation results, so this solution can be widely applied in clinical diagnosis and treatment.

[0071] In a specific example, the embodiments of the present invention aim to achieve automatic efficacy evaluation of, for example, lymphoma based on PET / CT images. To this end, the present invention proposes an automatic efficacy evaluation method that integrates local information and global information of lesions. After obtaining the PET / CT images before and after the treatment of lymphoma patients, the method first extracts and calculates the local information of the lymphoma lesion area from the PET / CT images to obtain the measurement of the lesion change information before and after treatment. Then, multi-view MIP images are generated using the PET images, and global image information related to efficacy evaluation is mined from them to achieve an automatic and accurate evaluation of the Deauville score. Finally, the Deauville score that integrates the above local information and global information is used to automatically generate lymphoma efficacy evaluation information through a large language model based on RAG.

[0072] It should be noted that although the steps of the method in the present invention are described in a specific order in the drawings, this does not require or imply that these steps must be executed in this specific order, or that all the steps shown must be executed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution, etc. Also, it is easy to understand that these steps may be executed synchronously or asynchronously, for example, in multiple modules / processes / threads.

[0073] As Figure 6 shown, the embodiments of the present invention provide a tumor efficacy evaluation system based on medical images, including:

[0074] A lesion change determination module 401, configured to obtain the first PET-CT image and the second PET-CT image of a hematological tumor patient before and after treatment, and determine the lesion change information of the lesion area based on the first PET-CT image and the second PET-CT image;

[0075] A global information determination module 402, configured to perform maximum intensity projection on the first PET-CT image and / or the second PET-CT image along different directions to generate maximum intensity projection images of multiple views; determine the global image information of the lesion area based on the maximum intensity projection images of multiple views;

[0076] An efficacy evaluation generation module 403, configured to determine the Deauville score corresponding to the efficacy of the tumor patient based on the global image information; integrate the lesion change information and the Deauville score to generate tumor efficacy evaluation information.

[0077] In one embodiment, the global information determination module determines the global image information of the lesion area based on the maximum intensity projection images of multiple views, including:

[0078] For the maximum intensity projection images of each perspective, image feature information is extracted using the corresponding feature extraction model; among them, the feature extraction models corresponding to the maximum intensity projection images of each perspective are different and are pre-trained from the original machine learning model based on sample maximum intensity projection images;

[0079] Fuse the image feature information corresponding to the maximum intensity projection images of each perspective to obtain the fused global image information.

[0080] In one embodiment, the different directions include at least two of the coronal direction, sagittal direction, right anterolateral direction, and left anterolateral direction.

[0081] In one embodiment, the efficacy evaluation generation module determines the Deauville score corresponding to the efficacy of the tumor patient based on the global image information, including:

[0082] Input the global image information into the target regression prediction model to obtain the corresponding Deauville score; among them, the target regression prediction model is pre-trained from the original regression prediction model based on sample global image information.

[0083] In one embodiment, the lesion change determination module determines the lesion change information of the lesion area based on the first PET-CT image and the second PET-CT image, including:

[0084] Respectively segment and determine the first lesion area image and the second lesion area image from the first PET-CT image and the second PET-CT image;

[0085] Perform image registration on the first lesion area image and the second lesion area image, and determine the lesion change information of the lesion area based on the registered first lesion area image and the second lesion area image. The lesion change information includes the change information of the position distribution, quantity, size, morphological boundary, and maximum standardized uptake value SUVmax of the lesion.

[0086] In one embodiment, the efficacy evaluation generation module fuses the lesion change information and the Deauville score to generate tumor efficacy evaluation information, including:

[0087] Combine the lesion change information and the Deauville score to generate input text information; perform feature encoding on the input text information to obtain a semantic vector;

[0088] Based on the semantic vector, retrieve and determine domain knowledge texts from the expert knowledge vector library whose semantic similarity to the semantic vector is greater than a preset threshold; wherein, the expert knowledge vector library is a pre-constructed knowledge vector library related to the efficacy evaluation of hematological malignancies, and contains multiple documents related to the efficacy evaluation of hematological malignancies that are indexed by pre-performing text encoding.

[0089] Fuse the domain knowledge text with the input text information to obtain target text information; input the target text information into a large language model to obtain tumor efficacy evaluation information.

[0090] In one embodiment, the target text information is used to provide professional prompt information for the large language model, so that the large language model outputs accurate and clinically understandable efficacy evaluation information.

[0091] Regarding the system in the above embodiment, the specific ways in which each module performs operations and the corresponding technical effects have been described in detail in the corresponding method embodiments, and will not be elaborated here.

[0092] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present invention, the features and functions of the two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units. The components shown as modules or units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. Those of ordinary skill in the art can understand and implement it without creative work.

[0093] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the method for evaluating tumor efficacy based on medical images described in any one of the above embodiments.

[0094] Exemplarily, the readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0095] The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, in which the readable program code is carried. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than the readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination of the above.

[0096] An embodiment of the present invention further provides an electronic device, including a processor and a memory, where the memory is used to store a computer program. Among them, the processor is configured to execute the tumor treatment effect evaluation method based on medical images in any one of the above embodiments by executing the computer program.

[0097] It should be noted that in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including", or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article, or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article, or device including the element.

[0098] The above are only specific embodiments of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments described herein, but rather will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for evaluating tumor efficacy based on medical imaging, characterized in that: The tumor is a lymphoma, including: Acquire a first PET-CT image and a second PET-CT image twice before and after the treatment of a patient with a hematological tumor, and determine lesion change information of the lesion area based on the first PET-CT image and the second PET-CT image; Performing maximum density projection on the PET image in the first PET-CT image and / or the second PET-CT image along different directions to generate maximum density projection images of multiple viewing angles; determining global image information of the lesion area based on the maximum density projection images of the multiple viewing angles; the different directions include at least two of the coronal direction, the sagittal direction, the right anterior lateral direction, and the left anterior lateral direction; Determine the Deauville score corresponding to the therapeutic effect of the tumor patient based on the global image information; fuse the lesion change information and the Deauville score to generate tumor therapeutic effect evaluation information; Wherein, determining the global image information of the lesion area based on the maximum density projection images of the multiple viewing angles includes: For the maximum density projection image of each viewing angle, the corresponding feature extraction model is used to extract image feature information; wherein the feature extraction model corresponding to the maximum density projection image of each viewing angle is different and is obtained by pre-training the original machine learning model based on the sample maximum density projection image; The image feature information corresponding to the maximum density projection image of each viewing angle is fused to obtain the fused global image information.

2. The method according to claim 1, characterized in that Determining the Deauville score corresponding to the therapeutic effect of the tumor patient based on the global image information includes: The global image information is input into a target regression prediction model to obtain a corresponding Deauville score; wherein the target regression prediction model is obtained by pre-training an original regression prediction model based on the sample global image information.

3. The method according to claim 1, characterized in that The determining of lesion change information of the lesion area based on the first PET-CT image and the second PET-CT image includes: Segmenting and determining a first lesion region image and a second lesion region image from the first PET-CT image and the second PET-CT image respectively; The first lesion area image and the second lesion area image are image registered, and lesion change information of the lesion area is determined based on the registered first lesion area image and the second lesion area image, wherein the lesion change information includes the location distribution, quantity, morphological boundaries, and change information of the maximum standardized uptake ratio SUVmax of the lesions.

4. The method according to any one of claims 1 to 3, characterized in that The fusing of the lesion change information and the Deauville score to generate tumor efficacy evaluation information includes: Combining the lesion change information and the Deauville score to generate input text information; performing feature encoding on the input text information to obtain a semantic vector; Based on the semantic vector, domain knowledge texts whose semantic similarity with the semantic vector is greater than a preset threshold are retrieved from an expert knowledge vector library; wherein the expert knowledge vector library is a pre-constructed knowledge vector library related to the evaluation of therapeutic efficacy of hematological tumors, and contains a plurality of documents related to the evaluation of therapeutic efficacy of hematological tumors that are pre-text-encoded and indexed; The domain knowledge text is fused with the input text information to obtain target text information; the target text information is input into a large language model to obtain tumor efficacy evaluation information.

5. The method according to claim 4, characterized in that The target text information is used to provide professional prompt information for the large language model, so that the large language model outputs accurate and clinically understandable efficacy evaluation information.

6. A tumor efficacy evaluation system based on medical imaging, characterized in that: The tumor is a lymphoma, including: A lesion change determination module, used to obtain a first PET-CT image and a second PET-CT image twice before and after the treatment of a patient with a hematological tumor, and determine lesion change information of a lesion area based on the first PET-CT image and the second PET-CT image; a global information determination module, configured to perform maximum density projection along different directions on the PET image in the first PET-CT image and / or the second PET-CT image to generate maximum density projection images of multiple viewing angles; and determine global image information of the lesion area based on the maximum density projection images of the multiple viewing angles; the different directions include at least two of the coronal direction, the sagittal direction, the right anterior lateral direction, and the left anterior lateral direction; A therapeutic effect evaluation generation module is used to determine the Deauville score corresponding to the therapeutic effect of the tumor patient based on the global image information; and to fuse the lesion change information and the Deauville score to generate tumor therapeutic effect evaluation information; Among them, the global information determination module is specifically used to: extract image feature information using the corresponding feature extraction model for the maximum density projection image of each viewing angle; wherein the feature extraction model corresponding to the maximum density projection image of each viewing angle is different and is pre-trained based on the original machine learning model of the sample maximum density projection image; fuse the image feature information corresponding to the maximum density projection image of each viewing angle to obtain the fused global image information.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for evaluating tumor efficacy based on medical images according to any one of claims 1 to 5 is implemented.

8. An electronic device, characterized in that: include: processor; as well as Memory for storing computer programs; Wherein, the processor is configured to execute the tumor efficacy evaluation method based on medical imaging according to any one of claims 1 to 5 by executing the computer program.

Images