Bone metastasis functional medical image generation method, device, equipment, medium and product
By constructing a conditional generative adversarial network with multi-scale information fusion modules, the problem of lesion localization in the diagnosis of bone metastases was solved, and a dynamic gradual transition process from lesion images to healthy images was realized, improving the accuracy and efficiency of diagnosis and treatment.
Patent Information
- Application Number
- CN202410951049.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-12
AI Technical Summary
In the diagnosis of bone metastases, current technologies often present challenges. SPECT images are noisy and complex, making it difficult to accurately locate lesions. Furthermore, existing GAN models fail to effectively learn lesion characteristics, resulting in deviations in the physiological structure of the repaired images, which affects diagnosis and treatment planning.
A conditional generative adversarial network with a multi-scale information fusion module is constructed. By introducing lesion labels and semantic modulatory factors during the training process, a gradual process of lesion formation from non-existence to existence and from existence to non-existence is generated, thereby improving the accuracy of lesion area repair and the matching degree of physiological structure.
The generated images are more accurate in terms of physiological structure, better reflect the dynamic changes of lesions, improve the accuracy of diagnosis and the efficiency of treatment planning, and provide personalized medical services.
Smart Images

Figure CN118941470A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of medical imaging, and in particular to a method, device, equipment, medium and product for generating functional medical images of bone metastasis. Background Art
[0002] Medical imaging is a key component of modern clinical diagnosis and treatment, using various technologies and equipment to obtain structural and functional information inside the human body. It plays an irreplaceable role in medical diagnosis, treatment and research. Among them, nuclear medicine imaging, as an important branch of medical imaging, plays a vital role in the diagnosis, treatment, evaluation and prevention of diseases. As an important functional medical imaging technology, nuclear medicine imaging can reflect the tissue function and structural changes of organs in the body, and provide blood flow, function and metabolic information of the lesion site, which is beyond the reach of traditional structural imaging technology.
[0003] Since the 1990s, nuclear medicine imaging has been recognized as a mature and effective medical method and has become an indispensable part of modern medicine. In particular, single-photon emission computed tomography (SPECT) and positron emission tomography (PET) technologies have been widely used in the detection of primary tumors and disease metastasis. Compared with PET, SPECT technology is easy to operate, low cost and has high sensitivity, making it more widely used in clinical practice. In SPECT examination, bone imaging agents are injected into the patient's body in advance, and then the gamma rays released by radionuclides in the human body are detected in a non-invasive manner by imaging equipment to obtain a distribution map of radioactive substances in the human body. On the SPECT image, metastatic lesions appear as a large accumulation of radionuclides, which are visually manifested as spots of higher brightness, called hot spots.
[0004] In the diagnosis of bone metastasis cases, it is extremely important to determine the location and range of the lesion through SPECT images. However, since SPECT images of patients with bone metastasis often contain complex noise and interference, doctors need to spend a lot of time and energy to accurately locate the lesion, which undoubtedly increases the difficulty of clinical work. This is a challenge to the efficiency of clinical work and the accuracy of patient treatment.
[0005] To overcome this challenge, the rise of deep learning has provided a new solution. Generative Adversarial Networks (GAN), as a generative technology, has shown great potential in generating realistic images. GAN uses adversarial training of two networks, the generator and the discriminator. The generator strives to produce images that are good enough to deceive the discriminator, while the discriminator strives to distinguish the authenticity of the image. This technology has shown excellent performance in areas such as style transfer because GAN can learn and imitate the important feature distribution of images in the training data.
[0006] In the field of medical imaging, especially for SPECT lesion diagnosis and repair, existing technical solutions are usually based on the standard GAN framework. Specifically, the existing solutions first sample healthy images, remove them randomly or in a specific way, and then use the GAN network to repair and reconstruct the removed image areas. On this basis, the network is trained by comparing and analyzing the differences between the repaired images and the original healthy images. For images with lesions, the existing technology uses lesion labels and segmentation networks to identify the lesion area, removes the lesion area, and applies the trained GAN to repair it.
[0007] However, the main drawback of the existing technology is that the entire training process is mainly focused on healthy images, and the characteristics of lesion images are not effectively learned. This method does not enable the model to learn healthy images, lesion images, lesion characteristics, etc. at the semantic category level. It can only generate repaired lesion-free images and cannot show the dynamic process of lesion formation and deterioration, which limits the doctor's cognition and understanding of the dynamic evolution of lesions. In addition, the identification and repair of tissue information around the lesion is often not accurate enough, resulting in deviations in the physiological structure of the repaired image, affecting the doctor's judgment on the severity of the lesion in the area, so the rationality of the physiological structure is particularly important for diagnosis and treatment planning. Summary of the invention
[0008] The purpose of this application is to provide a method, device, equipment, medium and product for generating functional medical images of bone metastasis, which can accurately generate the dynamic gradual change process of lesion images to healthy images.
[0009] To achieve the above objectives, this application provides the following solutions:
[0010] In the first aspect, the present application provides a method for generating functional medical images of bone metastasis, including: constructing a conditional generative adversarial network with a multi-scale information fusion module as an input layer; training the conditional generative adversarial network using a non-bone metastasis patient image set to obtain a conditional generative adversarial network after one training; the non-bone metastasis patient image set includes non-bone metastasis patient images and non-bone metastasis patient images with lesion areas corresponding to lesion labels of metastatic patients removed; inputting multiple bone metastasis patient images with lesion areas corresponding to lesion labels of metastatic patients removed into the conditional generative adversarial network after one training respectively, and outputting multiple preliminary disease-free reconstructed images; extracting the lesion area corresponding to the lesion label of the metastatic patient from each preliminary disease-free reconstructed image, and adding them pixel by pixel with the corresponding bone metastasis patient images with the lesion labels of the metastatic patient removed to obtain multiple disease-free reconstructed images of the metastatic patient; establishing a semantic The health-lesion balance loss of the adjustment factor is used; multiple bone metastasis patient images corresponding to the lesion areas of the metastasis patient lesion labels are removed as input, and the semantic adjustment factor, multiple metastasis patient disease-free reconstructed images and multiple bone metastasis patient images constitute the learning target, and combined with the metastasis patient lesion labels adjusted by the semantic adjustment factor, the conditional generative adversarial network after the first training is trained twice using the health-lesion balance loss to obtain a trained conditional generative adversarial network; the metastasis patient images to be generated after the lesion area is removed and the metastasis patient lesion labels adjusted by different semantic adjustment factors are input into the trained conditional generative adversarial network to obtain a preliminary reconstructed lesion gradient image; the lesion area is extracted from the preliminary reconstructed lesion gradient image, and is added pixel by pixel to the metastasis patient image to be generated after the lesion area is removed to obtain a lesion state image of the gradual change of the lesion area of the metastasis patient.
[0011] In the second aspect, the present application provides a bone metastasis functional medical image generation device, including: a network construction module, a primary training module, a preliminary reconstruction module, a disease removal reconstruction module, a loss establishment module, a secondary training module, a network application module and a lesion gradient reconstruction module.
[0012] The network construction module is used to construct a conditional generative adversarial network with a multi-scale information fusion module as an input layer; the one-time training module is used to train the conditional generative adversarial network using a non-bone metastasis patient image set to obtain a conditional generative adversarial network after one-time training; the non-bone metastasis patient image set includes non-bone metastasis patient images and non-bone metastasis patient images with the lesion area corresponding to the lesion label of the metastasis patient removed; the preliminary reconstruction module is used to input multiple bone metastasis patient images with the lesion area corresponding to the lesion label of the metastasis patient removed into the conditional generative adversarial network after one-time training, and output multiple preliminary disease-free reconstructed images; the disease-free reconstruction module is used to extract the lesion area corresponding to the lesion label of the metastasis patient from each preliminary disease-free reconstructed image, and add it pixel by pixel with the bone metastasis patient images with the lesion area corresponding to the lesion label of the metastasis patient removed, to obtain multiple disease-free reconstructed images of the metastasis patient; the loss establishment module is used to establish a health- Lesion balance loss; the secondary training module is used to take multiple bone metastasis patient images corresponding to the lesion area of the metastasis patient lesion labels removed as input, and the semantic adjustment factor, multiple metastasis patient disease-free reconstructed images and multiple bone metastasis patient images constitute the learning target, and combined with the metastasis patient lesion labels adjusted by the semantic adjustment factor, the conditional generative adversarial network after the first training is trained twice using the health-lesion balance loss to obtain a trained conditional generative adversarial network; the network application module is used to input the metastasis patient image to be generated after the lesion area is removed and the metastasis patient lesion labels adjusted by different semantic adjustment factors into the trained conditional generative adversarial network to obtain a preliminary reconstructed lesion gradient image; the lesion gradient reconstruction module is used to extract the lesion area from the preliminary reconstructed lesion gradient image, and add it pixel by pixel with the metastasis patient image to be generated after the lesion area is removed, to obtain a lesion state image of the gradient of the lesion area of the metastasis patient.
[0013] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for generating functional medical images of bone metastasis described above.
[0014] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for generating functional medical images of bone metastasis described above.
[0015] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method for generating functional medical images of bone metastasis described above.
[0016] According to the specific embodiments provided in this application, this application discloses the following technical effects:
[0017] The present application provides a method, apparatus, device, medium and product for generating functional medical images of bone metastasis. By adopting a multi-scale information fusion module, semantic and feature analysis is performed at different scales, thereby improving the matching degree between the repaired image and the medical physiological structure, and being able to better identify and integrate the tissue information and feature information around the lesion, the image finally generated is more accurate in physiological structure. By establishing a healthy-lesion balance loss containing a semantic adjustment factor, and accurately learning and adjusting the semantic information during the training process of a conditional generative adversarial network, the conditional generative adversarial network can be used to generate lesion status images with different lesion severities, thereby achieving the accurate generation of the dynamic gradual change process of the lesion from the lesion image to the healthy image. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 This is an application environment diagram of a method for generating a bone metastasis functional medical image in one embodiment of the present application;
[0020] Figure 2 A schematic diagram of a flow chart of a method for generating a bone metastasis functional medical image according to an embodiment of the present application;
[0021] Figure 3 A schematic diagram of the structure of a multi-scale information fusion module provided in an embodiment of the present application;
[0022] Figure 4 This is a schematic diagram of the overall principle of a method for generating a bone metastasis functional medical image in one embodiment of the present application;
[0023] Figure 5 A schematic diagram of the first stage lesion removal and reconstruction process provided in one embodiment of the present application;
[0024] Figure 6 A schematic diagram of a gradual change process generation flow chart provided in an embodiment of the present application;
[0025] Figure 7 A schematic diagram of the gradual process effect of lesion elimination provided in an embodiment of the present application;
[0026] Figure 8 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0028] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.
[0029] The bone metastasis functional medical image generation method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send non-bone metastasis patient images, bone metastasis patient images and metastasis patient images to be generated to the server 104. After the server 104 receives the non-bone metastasis patient images, bone metastasis patient images and metastasis patient images to be generated, the server 104 constructs a conditional generative adversarial network with a multi-scale information fusion module as an input layer; the conditional generative adversarial network is trained using a non-bone metastasis patient image set to obtain a trained conditional generative adversarial network; the non-bone metastasis patient image set includes non-bone metastasis patient images and non-bone metastasis patient images with lesion areas corresponding to metastasis patient lesion labels removed; multiple bone metastasis patient images with lesion areas corresponding to metastasis patient lesion labels removed are respectively input into the trained conditional generative adversarial network to output multiple preliminary disease-free reconstructed images; the lesion area corresponding to the metastasis patient lesion label is extracted from each preliminary disease-free reconstructed image, and the lesion area corresponding to the metastasis patient lesion label is extracted, and compared with the corresponding bone metastasis patient images with the corresponding lesion area of the metastasis patient lesion label removed. The image is added pixel by pixel to obtain multiple disease-free reconstructed images of metastatic patients; a health-lesion balance loss including a semantic adjustment factor is established; multiple bone metastasis patient images corresponding to the lesion area of the lesion label of the metastatic patient are removed as input, and the semantic adjustment factor, multiple disease-free reconstructed images of the metastatic patient and multiple bone metastasis patient images constitute a learning target, and combined with the lesion label of the metastatic patient adjusted by the semantic adjustment factor, the health-lesion balance loss is used to perform a secondary training on the conditional generative adversarial network after the first training to obtain a trained conditional generative adversarial network; the image of the metastatic patient to be generated after the lesion area is removed and the lesion label of the metastatic patient adjusted by different semantic adjustment factors are input into the trained conditional generative adversarial network to obtain a preliminary reconstructed lesion gradient image; the lesion area is extracted from the preliminary reconstructed lesion gradient image, and it is added pixel by pixel to the image of the metastatic patient to be generated after the lesion area is removed to obtain a lesion state image of the lesion area of the metastatic patient. The server 104 can feed back the obtained lesion state image of the lesion area of the metastatic patient to the terminal 102. In addition, in some embodiments, the bone metastasis functional medical image generation method can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly generate medical images for non-bone metastasis patient images, bone metastasis patient images, and metastasis patient images to be generated, or the server 104 can obtain non-bone metastasis patient images, bone metastasis patient images, and metastasis patient images to be generated from the data storage system, and generate medical images for non-bone metastasis patient images, bone metastasis patient images, and metastasis patient images to be generated.
[0030] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.
[0031] In an exemplary embodiment, Figure 2 As shown, a method for generating a bone metastasis functional medical image is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps 201 to 208. Among them:
[0032] Step 201: Construct a conditional generative adversarial network with a multi-scale information fusion module as an input layer.
[0033] Step 202: Use the non-bone metastasis patient image set to train the conditional generative adversarial network to obtain a trained conditional generative adversarial network; the non-bone metastasis patient image set includes non-bone metastasis patient images and non-bone metastasis patient images with lesion areas corresponding to metastasis patient lesion labels removed.
[0034] Step 203: Multiple images of patients with bone metastasis with lesion areas corresponding to lesion labels of metastatic patients removed are respectively input into a conditional generative adversarial network after training once, and multiple preliminary disease-free reconstructed images are output.
[0035] Step 204: extract the lesion area corresponding to the lesion label of the metastatic patient from each preliminary disease-free reconstructed image, and add them pixel by pixel with the corresponding bone metastasis patient images with the lesion area corresponding to the lesion label of the metastatic patient removed to obtain multiple disease-free reconstructed images of the metastatic patient.
[0036] Step 205: Establish a healthy-lesion balance loss including semantic adjustment factors.
[0037] Step 206: Taking multiple bone metastasis patient images with lesion areas corresponding to the lesion labels of metastatic patients removed as input, the semantic adjustment factor, multiple metastatic patient disease-free reconstructed images and multiple bone metastasis patient images constitute the learning objectives, and combined with the metastatic patient lesion labels adjusted by the semantic adjustment factor, the conditional generative adversarial network after the first training is trained twice using the health-lesion balance loss to obtain a trained conditional generative adversarial network.
[0038] Step 207: Input the to-be-generated metastatic patient image after removing the lesion area and the metastatic patient lesion label adjusted by different semantic adjustment factors into the trained conditional generative adversarial network to obtain a preliminary reconstructed lesion gradient image.
[0039] Step 208: extracting the lesion region from the initially reconstructed lesion gradient image, and adding it pixel by pixel to the to-be-generated metastasis patient image after the lesion region is removed, to obtain a lesion state image of the metastasis patient lesion region gradient.
[0040] GAN (Generative Adversarial Networks) can effectively process and convert SPECT images of patients with bone metastasis. The core of this application is to convert the area with lesions into a lesion-free state. By learning and adjusting the semantic information during the GAN training process, it is possible to control and display the gradual process of lesion disappearance. Specifically, the adjustment of semantic information enables the generation of SPECT images showing different severity states of lesions, which is of great significance in medical diagnosis.
[0041] GANs are able to recognize and understand the different severity levels represented by lesions in images through a designed loss function. By training GANs, the network not only learns to visually reconstruct lesion-free bone images, but also adjusts the severity of lesions in the generated images based on the input control conditions (i.e., semantic information). Doctors can use these generated SPECT images to quickly and accurately assess the location and size of lesions. More importantly, by analyzing the state of lesions of different severity, doctors can have an intuitive understanding of the progression of the disease, including how lesions develop from the initial stage to a more severe state. This is extremely valuable for the diagnosis process, the formulation of treatment plans, and the monitoring of disease progression. This technology can also assist doctors in monitoring changes in the condition during treatment, evaluating the effectiveness of treatment, and potentially predicting the trend of recovery. With the high flexibility of GAN technology, doctors can save and analyze this data in the format of images, GIFs, or even videos to provide tailored medical services for each patient.
[0042] In short, through the precise learning and adjustment of semantic information, this application can use GAN technology to generate SPECT images with different lesion severity, providing a new perspective and tool for medical diagnosis, treatment planning and disease monitoring. The application of this technology will undoubtedly greatly improve the accuracy of medical diagnosis and the efficiency of treatment, and provide patients with better medical services.
[0043] In another exemplary embodiment of the present application, in order to effectively improve the accuracy of the organizational structure and the semantic information representation of the image generation, four independent branches are used to capture and integrate information, such as Figure 3 As shown, the specific structure of the multi-scale information fusion module in the above step 201 can be:
[0044] The multi-scale information fusion module includes: a first convolution branch, a second convolution branch, a third convolution branch, a fourth convolution branch and an adding unit; the first convolution branch is used to input the feature map of the input image after 1×1 convolution to the adding unit; the second convolution branch is used to input the feature map of the input image after 3×3 convolution to the adding unit; the third convolution branch is used to input the feature map of the input image after 3×3 convolution, 1×3 convolution and 3×1 convolution in sequence to the adding unit; the fourth convolution branch is used to input the feature map of the input image after 3×3 convolution, 3×3 convolution and 3×3 convolution in sequence to the adding unit; the adding unit is used to add the feature map of the first convolution branch, the feature map of the second convolution branch, the feature map of the third convolution branch and the feature map of the fourth convolution branch, and output a fused feature map.
[0045] The multi-scale information fusion module is used in the model input layer, which can fuse information of multiple scales to enhance the representation ability of the model. The scale focus of the feature map of each branch is different, covering information extraction from dense to sparse, so that the model can understand the image content at different scales. This strategy of adding pixels not only promotes the fusion of information at each scale, but also ensures that each point at the pixel level is rich in comprehensive contextual information. Adding the feature maps of the four branches ensures that the information of each pixel point is combined with the feature information of the neighboring area to form an information-rich feature representation.
[0046] In another exemplary embodiment of the present application, since the spine and ribs are the most common metastatic sites of various cancers and are located in the chest area, and the chest area is mostly a high-risk area for lung cancer bone metastasis. Therefore, the chest area is extracted from the whole body image as training data. After the above step 201, the method can also include the following steps 301 to 304. Among them:
[0047] Step 301: Establishing a whole-body SPECT bone scan image dataset of a metastatic patient and a whole-body SPECT bone scan image dataset of a non-metastatic patient; the whole-body SPECT bone scan image dataset of a metastatic patient includes the whole-body SPECT bone scan image of the metastatic patient and the lesion label of the metastatic patient.
[0048] Step 302: Set the pixel points whose pixel values are less than the pixel threshold on the whole-body SPECT bone scan image of the metastatic patient and the whole-body SPECT bone scan image of the non-metastatic patient in the whole-body SPECT bone scan image set to 0, and obtain the whole-body SPECT bone scan image of the metastatic patient after denoising and the whole-body SPECT bone scan image of the non-metastatic patient after denoising.
[0049] Step 303: Along the straight line with the least number of pixels on the horizontal axis and the straight line with the least number of pixels on the vertical axis, the chest area is cropped from the denoised whole-body SPECT bone scan image of the metastatic patient and the denoised whole-body SPECT bone scan image of the non-metastatic patient, respectively, and the cropped chest area is placed at the center of the image. The image size is padded with 0 to 256×256 pixels to obtain the SPECT chest bone scan image of the metastatic patient and the SPECT chest bone scan image of the non-metastatic patient.
[0050] Step 304: The SPECT chest bone scan image of the metastatic patient is used as the bone metastasis patient image, and the SPECT chest bone scan image of the non-metastatic patient is used as the non-bone metastasis patient image.
[0051] That is, in this exemplary embodiment, the present application crops the chest area from each whole-body SPECT bone scan image by the minimum number of pixels present in the image (horizontal and vertical axes). The data preprocessing steps specifically include: data denoising, extracting valid areas, and filling the background. Data denoising: Set pixels less than a threshold of 5 to 0. Extracting valid areas: Crop the chest area from the whole-body SPECT bone scan image by the minimum number of pixels present in the image (horizontal and vertical axes). Filling the background: Place the cropped area at the center of the image and fill the image size to 256×256 pixels using 0 padding.
[0052] It can be seen from steps 202 to 206 that in order to obtain the generation of the gradient process, two training stages are required. The first stage is used to generate the lesion image into the lesion-free image, and the second stage is used to generate the gradient process between the lesion image and the lesion-free image. The overall principle diagram is shown in FIG. Figure 4 shown.
[0053] The first stage is generation: lesion removal and reconstruction.
[0054] The training model generates lesion images into lesion-free images. The process is as follows: Figure 5 shown.
[0055] Disease-removal and reconstruction image: preliminary overall image for lesion repair.
[0056] Optimization: Extract the lesion area (healthy state after the lesion is removed) in the disease-free reconstructed image through the lesion label, and add the area to the original image after the lesion is removed. Ensure that other parts outside the lesion remain unchanged.
[0057] The second stage generates:
[0058] (1) Loss Construction
[0059] This application proposes a healthy-lesion balance loss, which is used for the generator to learn different semantic information, and to learn different semantic variables by setting semantic adjustment factors based on the basic loss function. Ultimately, it is possible to generate lesion images of any severity by controlling the semantic labels, as well as to generate a lesion-no-lesion gradient process. Common losses for generative models include but are not limited to L1Loss and L2 Loss, and the loss construction takes L1 Loss as an example. L1 Loss is shown in the following formula:
[0060] Among them, x represents the metastatic patient lesion label, y represents the real image, y' represents the real image with the corresponding area removed according to the metastatic patient lesion label, and z represents the noise image. The meaning of G(x,y',z): G is the generator, x, y' and z are the inputs of the generator, and G(x,y',z) is the generated image of the generator. x,y,y',z Represents the expectation of the loss function given x, y, y', z. L1 Loss (mean absolute error) refers to the average of the absolute differences between the generator's generated image G(x, y', z) and the real image y.
[0061] The health-diseased balance loss is given by the following formula:
[0062] x′=α×x
[0063] Where L represents the health-lesion balance loss, L1 represents the loss function of the generative model, x′ represents the lesion label of the metastatic patient adjusted by the semantic adjustment factor, α represents the semantic adjustment factor, α∈[0,1]; y1 represents the image of the bone metastasis patient, y1' represents the image of the bone metastasis patient with the lesion area removed according to the lesion label of the metastatic patient, and y2 represents the disease-free reconstructed image of the metastatic patient.
[0064] (2) Gradual process generation flow chart
[0065] In order to generate a gradual process through semantic adjustment, this application designs a unique balanced training method, the flow chart is as follows Figure 6 shown.
[0066] This application has successfully created a new technology for generating a gradual process between health and lesions based on semantic information regulation. The core is to use the regulation of semantic information to accurately generate lesion images of any severity, thereby obtaining a gradual process from lesions to no lesions. Compared with traditional technologies, the most notable feature of this application is that it can accurately control the display of lesions, including the generation, evolution and recovery process of lesions, according to the regulation of semantic information. This feature enables doctors to understand the dynamic changes of lesions more accurately and intuitively, greatly improving the accuracy and efficiency of disease diagnosis and treatment planning. This application can not only generate highly realistic images of lesions with different degrees of severity, but also reflect the development and treatment process of lesions by detailed learning and simulation of the characteristics of lesions and their surrounding tissues, providing a new perspective for medical imaging diagnosis and greatly enriching the diagnosis and treatment applications of medical images. In addition, the application of this application will significantly improve the efficiency of clinical diagnosis and the accuracy of treatment planning, and provide patients with more accurate and personalized medical services.
[0067] Compared with the prior art, the method proposed in this application shows many advantages in processing lesion images. The prior art can only generate lesion images as lesion-free images, while the method proposed in this application can generate the gradual process of lesions, including the generation, evolution and recovery process of lesions. First, the method of this application is not limited to learning healthy images, but in the second stage, by adding lesion images as training samples, GAN can learn and understand the characteristics of the lesion area, so that the generated images are improved in terms of the accuracy of lesion-free and the authenticity of lesions. Secondly, compared with the prior art, the system of this application learns semantic information and can adjust the representation of semantic information, so that lesion images of different severities can be generated, so that the gradual process can reflect the development of lesions from nothing to something and the treatment process from something to nothing. This feature is very valuable for doctors in diagnosis, treatment planning and efficacy monitoring, because it provides an intuitive model of lesion development and helps doctors better understand the dynamic changes of the disease.
[0068] Furthermore, the method of this application adopts a multi-scale strategy, and by performing semantic and feature analysis at different scales, it further improves the matching degree between the repaired image and the medical physiological structure. This means that the technology of this application can better identify and integrate the tissue information and feature information around the lesion, and the final generated image is more accurate in physiological structure, providing doctors with more reliable image basis. This not only optimizes the quality of lesion repair, but also provides a more solid foundation for subsequent medical decision-making.
[0069] Compared with the existing technology, the method of this application provides a more comprehensive and dynamic solution, which not only expands the application scope of GAN in the diagnosis and repair of medical imaging lesions, but also improves the diagnostic value and clinical relevance of images, which are not achievable by existing technical solutions. Through more sophisticated lesion feature learning, dynamic lesion characterization, and accurate reconstruction of physiological structures, the technology of this application is superior to existing technologies in terms of aesthetics, accuracy, and practicality, which is of great significance and value in actual clinical applications. Based on this background, this application proposes a method and system for generating functional medical images that focuses on the gradual process of disease.
[0070] The effect diagram of the gradual process of lesion elimination is as follows Figure 7 As shown, Figure 7 The red lines in the figure indicate the location of the lesion.
[0071] This application solves the following technical problems:
[0072] (1) How to solve the problem of accuracy of generated images in terms of organizational structure and semantic categories.
[0073] This application proposes a multi-scale feature extractor to solve the problem of accuracy in tissue structure of generated images and the problem of semantic information fusion in adjacent regions. By combining image details and semantic information at different scales, the model can enhance the learning of tissue features and semantic changes around the lesion, thereby maintaining and improving the accuracy and detail expression of tissue structure and semantic categories in the generated images. This not only improves the overall quality of the image, but also ensures the accuracy and reliability of doctors when using these images for diagnosis and treatment planning.
[0074] (2) How to generate the dynamic changes of lesions from lesion images to healthy images through semantic control.
[0075] This application aims to study the dynamic changes of bone imaging lesions, and uses SPECT bone imaging as data. The semantic label is used as a regulating variable, and the healthy-lesion balance loss is constructed to enable the model to learn the semantic information of the lesion at different development stages. By adjusting the conditional variables, lesion images of any severity are generated, thereby obtaining the gradual change process from lesion to health (or health-lesion) in the lesion area.
[0076] The advantages of this application are as follows:
[0077] (1) The generative model based on deep learning can reconstruct images according to the image’s own characteristics. Compared with traditional interpolation methods, the reconstruction results are more in line with physiological structure requirements.
[0078] (2) Compared with traditional lesion repair, which can only repair lesions to lesion-free, but cannot present the gradual change from lesion to lesion-free, the present application can generate the gradual change from lesion to lesion-free, including the generation, evolution and recovery process of lesions, providing doctors with a more effective auxiliary perspective.
[0079] (3) The gradient process is completed by using the model to transform the semantic information category, rather than interpolating the images of no lesion and lesion. The model can generate accurate lesion status images based on semantic information by learning the changes in semantic information.
[0080] The following prior art is provided below to demonstrate the advantages of the method of the present application by comparing the prior art.
[0081] 1. The Chinese patent application with publication number CN117495733A, "A method, device, electronic device and storage medium for repairing lesion images", includes: obtaining a lesion image to be repaired and lesion extraction label data; using the lesion extraction label data to perform image pixel processing on the lesion image to be repaired, and obtaining a lesion extraction image corresponding to the lesion image to be repaired; inputting the lesion extraction image into a pre-trained lesion image repair model to obtain an initial lesion repair image output by the model; and determining a target lesion repair image according to the initial lesion repair image. By comparing the target lesion repair image with the lesion image to be repaired, it is beneficial for doctors to quickly and accurately locate the lesion area according to the differences between the two images, so as to provide doctors with more reliable auxiliary diagnosis basis for clinical diagnosis and treatment, and improve the doctor's diagnostic efficiency of the lesion area and the accuracy of the diagnostic results. This method can only generate a lesion-free image from a lesion image, but cannot obtain the gradual change process of the lesion-health image like the present application, and does not design a healthy-lesion balance loss for generating a gradual change process.
[0082] 2. The Chinese patent application "A method for generating pseudo-healthy images of cerebral hemorrhage based on pathological decoupling" with publication number CN116630272A specifically includes: constructing and training a pseudo-image generation model; at each training iteration: first, the pathological image as training data is input into the segmenter, and the pathological region mask is segmented from the pathological image by the segmenter; then the pathological region mask and the pathological image are input into the generator, and a pseudo-healthy image is generated by the generator; then the pseudo-healthy image and the pathological region mask are input into the reconstructor, and a reconstructed pathological image is generated by the reconstructor; finally, the pseudo-image generation model is trained based on the total loss of the loss calculation model of the segmenter, the generator and the reconstructor; the corresponding pseudo-healthy image is generated by the generator of the trained pseudo-image generation model. The present invention can ensure that the original identity of the pathological image remains unchanged when removing the pathological region, and can realize a one-to-one mapping of the pathological region in the process of generating and reconstructing the pathological image. This scheme can only generate a lesion-free image from a lesion image, and it is also impossible to obtain the gradual change process of the lesion-healthy image like the present application, and there is no healthy-lesion balance loss designed to generate the gradual change process.
[0083] 3. The Chinese patent "Image Restoration Method and Device, Electronic Device and Storage Medium" with publication number CN111553864A, the method comprises: obtaining an image to be repaired and a reference image; wherein the clarity of the reference image is greater than the clarity of the image to be repaired; performing face repair on the face area of the image to be repaired based on the face area of the reference image to obtain a repaired face area; and fusing the repaired face area with the image to be repaired to obtain a repaired image. The disclosed embodiment can improve the clarity of the image. This solution improves the clarity of the image, but cannot change the lesion image into a lesion-free image.
[0084] Based on the same inventive concept, the embodiment of the present application also provides a device for generating functional medical images of bone metastasis involved in the above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiment of a device for generating functional medical images of bone metastasis provided below can refer to the limitations of the method for generating functional medical images of bone metastasis above, and will not be repeated here.
[0085] In an exemplary embodiment, a bone metastasis functional medical image generation device includes: a network construction module, a primary training module, a preliminary reconstruction module, a disease removal reconstruction module, a loss establishment module, a secondary training module, a network application module and a lesion gradient reconstruction module.
[0086] The network building module is used to construct a conditional generative adversarial network with a multi-scale information fusion module as the input layer.
[0087] The one-time training module is used to train the conditional generative adversarial network using a non-bone metastasis patient image set to obtain a conditional generative adversarial network after one training; the non-bone metastasis patient image set includes non-bone metastasis patient images and non-bone metastasis patient images with the lesion area corresponding to the metastasis patient lesion label removed.
[0088] The preliminary reconstruction module is used to input multiple bone metastasis patient images with lesion areas corresponding to the lesion labels of the metastatic patients removed into the conditional generative adversarial network after training once, and output multiple preliminary disease-free reconstructed images.
[0089] The disease removal and reconstruction module is used to extract the lesion area corresponding to the lesion label of the metastatic patient from each preliminary disease removal and reconstruction image, and add it pixel by pixel with the corresponding bone metastasis patient image with the lesion area corresponding to the lesion label of the metastatic patient removed, so as to obtain multiple disease removal and reconstruction images of the metastatic patient.
[0090] The loss building module is used to build a healthy-lesion balance loss that includes semantic conditioning factors.
[0091] The secondary training module is used to take multiple bone metastasis patient images corresponding to the lesion areas of the metastasis patient lesion labels removed as input, and the semantic adjustment factor, multiple metastasis patient disease-free reconstructed images and multiple bone metastasis patient images constitute the learning objectives, and combine the metastasis patient lesion labels adjusted by the semantic adjustment factor, use the health-lesion balance loss, and perform secondary training on the conditional generative adversarial network after the first training to obtain a trained conditional generative adversarial network.
[0092] The network application module is used to input the metastatic patient image to be generated after the lesion area is removed and the metastatic patient lesion label adjusted by different semantic adjustment factors into the trained conditional generative adversarial network to obtain a preliminary reconstructed lesion gradient image.
[0093] The lesion gradient reconstruction module is used to extract the lesion area from the initially reconstructed lesion gradient image, and add it pixel by pixel to the image of the metastatic patient to be generated after the lesion area is removed, so as to obtain a lesion state image of the lesion area gradient of the metastatic patient.
[0094] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store bone metastasis functional medical imaging generation data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for generating bone metastasis functional medical images is implemented.
[0095] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0096] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.
[0097] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.
[0098] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.
[0099] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0100] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0101] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.
[0102] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0103] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.
Claims
1. A method for generating a functional medical image of bone metastasis, characterized in that: include: Construct a conditional generative adversarial network with a multi-scale information fusion module as the input layer; The conditional generative adversarial network is trained using a non-bone metastasis patient image set to obtain a trained conditional generative adversarial network; the non-bone metastasis patient image set includes non-bone metastasis patient images and non-bone metastasis patient images without lesion areas corresponding to lesion labels of metastasis patients; Input multiple bone metastasis patient images with lesion areas corresponding to the lesion labels of the metastatic patients into the trained conditional generative adversarial network, and output multiple preliminary disease-free reconstructed images; Extracting the lesion area corresponding to the lesion label of the metastatic patient from each preliminary disease-free reconstructed image, and adding the lesion area corresponding to the lesion label of the metastatic patient to the corresponding bone metastasis patient image after excavating the lesion area pixel by pixel, to obtain multiple disease-free reconstructed images of the metastatic patient; Establishing healthy-lesion balance loss including semantic regulators; Taking multiple bone metastasis patient images with lesion areas corresponding to the lesion labels of metastatic patients removed as input, semantic adjustment factors, multiple disease-free reconstructed images of metastatic patients and multiple bone metastasis patient images constitute learning targets, and combining the lesion labels of metastatic patients adjusted by the semantic adjustment factors, using the health-lesion balance loss, the conditional generative adversarial network after the first training is trained twice to obtain a trained conditional generative adversarial network; The metastatic patient image to be generated after the lesion area is removed and the metastatic patient lesion label adjusted by different semantic adjustment factors are input into the trained conditional generative adversarial network to obtain a preliminary reconstructed lesion gradient image; The lesion area is extracted from the initially reconstructed lesion gradient image, and is added pixel by pixel to the image of the metastatic patient to be generated after the lesion area is removed, so as to obtain a lesion state image of the lesion area gradient of the metastatic patient.
2. The method for generating bone metastasis functional medical images according to claim 1, characterized in that: The multi-scale information fusion module includes: a first convolution branch, a second convolution branch, a third convolution branch, a fourth convolution branch and an adding unit; The first convolution branch is used to input the feature map of the input image after 1×1 convolution to the addition unit; The second convolution branch is used to input the feature map of the input image after 3×3 convolution to the addition unit; The third convolution branch is used to sequentially perform 3×3 convolution, 1×3 convolution and 3×1 convolution on the input image, and input the feature map into the adding unit; The fourth convolution branch is used to perform 3×3 convolution, 3×3 convolution and 3×3 convolution on the input image in sequence, and input the feature map into the adding unit; The adding unit is used to add the feature map of the first convolution branch, the feature map of the second convolution branch, the feature map of the third convolution branch and the feature map of the fourth convolution branch, and output a fused feature map.
3. The method for generating bone metastasis functional medical images according to claim 1, characterized in that: Construct a conditional generative adversarial network with a multi-scale information fusion module as the input layer, followed by: Establishing a whole-body SPECT bone scan image dataset of patients with metastasis and a whole-body SPECT bone scan image dataset of patients without metastasis; the whole-body SPECT bone scan image dataset of patients with metastasis includes whole-body SPECT bone scan images of patients with metastasis and lesion labels of patients with metastasis; The pixel points whose pixel values are less than the pixel threshold on the whole body SPECT bone scan images of the metastatic patients and the whole body SPECT bone scan images of the non-metastatic patients in the whole body SPECT bone scan image set are set to 0, so as to obtain the whole body SPECT bone scan images of the metastatic patients after denoising and the whole body SPECT bone scan images of the non-metastatic patients after denoising; Along the straight line with the least number of pixels on the horizontal axis and the straight line with the least number of pixels on the vertical axis, the chest area is cropped from the whole-body SPECT bone scan image of the metastatic patient after denoising and the whole-body SPECT bone scan image of the non-metastatic patient after denoising, and the cropped chest area is placed at the center of the image, and the image size is padded to 256×256 pixels using 0, so as to obtain the SPECT chest bone scan image of the metastatic patient and the SPECT chest bone scan image of the non-metastatic patient; The SPECT chest bone scan images of the metastatic patients are used as the bone metastasis patient images, and the SPECT chest bone scan images of the non-metastatic patients are used as the non-bone metastasis patient images.
4. The method for generating bone metastasis functional medical images according to claim 1, characterized in that: The health-disorder balance loss is: L=αL1(x',y1,g′1,z)+(1-α)L1(x',y2,y′1,z); x′=α×x; Where L represents the health-lesion balance loss, L1 represents the loss function of the generative model, x represents the lesion label of the metastatic patient, x′ represents the lesion label of the metastatic patient adjusted by the semantic adjustment factor, α represents the semantic adjustment factor, α∈[0,1]; y1 represents the image of the bone metastasis patient, y1′ represents the image of the bone metastasis patient with the lesion area removed according to the lesion label of the metastatic patient, y2 represents the disease-free reconstructed image of the metastatic patient, and z represents the noise image.
5. The method for generating bone metastasis functional medical images according to claim 4, characterized in that: The loss function of the generation model includes: L1 Loss or L2 Loss.
6. A bone metastasis functional medical image generation device, characterized in that: include: A network construction module is used to construct a conditional generative adversarial network with a multi-scale information fusion module as the input layer; A one-time training module is used to train the conditional generative adversarial network using a non-bone metastasis patient image set to obtain a conditional generative adversarial network after one-time training; the non-bone metastasis patient image set includes non-bone metastasis patient images and non-bone metastasis patient images without the lesion area corresponding to the metastasis patient lesion label; A preliminary reconstruction module, for respectively inputting multiple bone metastasis patient images with lesion areas corresponding to the lesion labels of the metastasis patients removed into the conditional generative adversarial network after one training, and outputting multiple preliminary disease-free reconstructed images; A disease removal and reconstruction module is used to extract the lesion area corresponding to the lesion label of the metastatic patient from each preliminary disease removal and reconstruction image, and add the lesion area corresponding to the lesion label of the metastatic patient to the corresponding bone metastasis patient image pixel by pixel, so as to obtain multiple disease removal and reconstruction images of the metastatic patient; A loss building module for building a healthy-lesion balance loss that includes semantic conditioning factors; A secondary training module is used to take multiple bone metastasis patient images with lesion areas corresponding to the lesion labels of metastatic patients removed as input, a semantic adjustment factor, multiple disease-free reconstructed images of metastatic patients, and multiple bone metastasis patient images as learning targets, and combine the lesion labels of metastatic patients adjusted by the semantic adjustment factor, use the health-lesion balance loss, and perform secondary training on the conditional generative adversarial network after the primary training to obtain a trained conditional generative adversarial network; The network application module is used to input the metastatic patient image to be generated after the lesion area is removed and the metastatic patient lesion label adjusted by different semantic adjustment factors into the trained conditional generative adversarial network to obtain a preliminary reconstructed lesion gradient image; The lesion gradient reconstruction module is used to extract the lesion area from the initially reconstructed lesion gradient image, and add it pixel by pixel with the metastasis patient image to be generated after the lesion area is removed, so as to obtain the lesion state image of the metastasis patient lesion area gradient.
7. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for generating functional medical images of bone metastasis according to any one of claims 1 to 5.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating functional medical images of bone metastasis described in any one of claims 1 to 5 is implemented.
9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating functional medical images of bone metastasis described in any one of claims 1 to 5 is implemented.
Citation Information
Patent Citations
Image restoration method and device, electronic equipment and storage medium
CN111553864A
Cerebral hemorrhage pseudo-health image generation method based on pathological decoupling
CN116630272A
Focus image restoration method and device, electronic equipment and storage medium
CN117495733A
