Image processing method and device and storage medium

Abstract

The invention relates to an image processing method and device and a storage medium. The method comprises the following steps: acquiring a first modal image of a target part; the target part comprises a region of interest; and inputting the first modal image into an image processing model, and performing region-of-interest segmentation and modal conversion on the first modal image through the image processing model to obtain a second modal image including the region-of-interest. By adopting the method, the quality of the obtained image of the second modal can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to an image processing method, apparatus and storage medium. Background Technology

[0002] With the continuous development of image processing technology, the conversion between images of different modalities has become possible. For example, in the medical field, it is possible to convert between computed tomography (CT) images and positron emission tomography (PET) images.

[0003] In existing technologies, image style transfer techniques are mainly used to analyze images of the first modality and convert them into images of the second modality. However, the quality of the second modality images obtained by existing methods is poor. Summary of the Invention

[0004] Therefore, it is necessary to provide an image processing method, apparatus, and storage medium that can improve the quality of the obtained second-modality image in order to address the aforementioned technical problems.

[0005] In a first aspect, this application provides an image processing method, the method comprising:

[0006] Acquire the first modal image of the target region; the target region includes the region of interest.

[0007] The first modal image is input into the image processing model, which performs region of interest segmentation and modality transformation on the first modal image to obtain a second modal image including the region of interest.

[0008] In one embodiment, the image processing model includes a global prediction model; the image processing model is used to segment the region of interest and perform mode transformation on the first modality image to obtain a second modality image including the region of interest, including:

[0009] The global prediction image is obtained by performing region of interest segmentation and mode transformation on the first modality image using a global prediction model.

[0010] The global prediction image is used as the second modality image.

[0011] In one embodiment, the image processing model includes a local prediction model; the image processing model is used to segment the region of interest and perform mode transformation on the first modality image to obtain a second modality image including the region of interest, including:

[0012] The first modality image is cropped to obtain a local image of the first modality including the region of interest;

[0013] The local prediction model is used to segment the region of interest and perform mode transformation on the local image of the first modality to obtain the local prediction image; the local prediction image is then used as the second modality image.

[0014] In one embodiment, the image processing model includes a global prediction model, a local prediction model, and a fusion model. The image processing model performs region-of-interest (ROI) segmentation and modality transformation on the first modality image to obtain a second modality image including the RIO, comprising:

[0015] The global prediction image is obtained by performing region of interest segmentation and mode transformation on the first modality image using a global prediction model.

[0016] The first modality image is cropped to obtain a local image of the first modality including the region of interest. The local prediction model is then used to segment the region of interest and perform modality transformation on the local image of the first modality to obtain a local prediction image.

[0017] The global and local predicted images are input into the fusion model to obtain the second modality image.

[0018] In one embodiment, the global prediction image and the local prediction image are input into a fusion model to obtain a second modality image, including:

[0019] The product of the feature value of the global predicted image and the corresponding first weight is used as the global feature, and the product of the local predicted image and the corresponding second weight is used as the local feature.

[0020] A second modality image is generated based on global and local features.

[0021] In one embodiment, the process of constructing the image processing model includes:

[0022] Acquire the first modality sample image;

[0023] The first modality sample image is input into the initial generator of the initial image processing model to obtain the second modality prediction image and the segmentation result of the region of interest;

[0024] Based on the second modality prediction image and the segmentation results of the region of interest, the parameters of the initial generator are adjusted until the second modality prediction image output by the initial generator meets the preset conditions, thus obtaining the image processing model.

[0025] In one embodiment, the first modality sample image is input into the initial generator of the initial image processing model, and the second modality predicted image and the segmentation result of the region of interest are output, including:

[0026] Obtain the second modality sample image corresponding to the first modality sample image; the second modality sample image includes the region of interest label corresponding to the region of interest;

[0027] Based on the second modality sample image, the first modality sample image is segmented by the initial generator of the initial image processing model to obtain the segmentation result of the region of interest. Then, the first modality sample image is modally transformed by the initial generator to obtain the second modality prediction image.

[0028] In one embodiment, the parameters of the initial generator are adjusted based on the second modality predicted image and the segmentation result of the region of interest, including:

[0029] The second modality prediction image and the second modality sample image are input into the initial discriminator of the initial image processing model to obtain the discrimination result;

[0030] Based on the discrimination results and the segmentation results of the region of interest, the parameters of the initial image processing model are adjusted.

[0031] Secondly, this application also provides an image processing apparatus, which includes:

[0032] The image acquisition module is used to acquire a first modal image of the target region; the target region includes the region of interest.

[0033] The modality conversion module is used to input the first modality image into the image processing model, and to perform region of interest segmentation and modality conversion on the first modality image through the image processing model to obtain a second modality image including the region of interest.

[0034] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the content of any embodiment of the image processing method of the first aspect described above.

[0035] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the content of any embodiment of the image processing method of the first aspect described above.

[0036] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the content of any embodiment of the image processing method of the first aspect described above.

[0037] The aforementioned image processing method, apparatus, and storage medium acquire a first modal image of a target region, which includes a region of interest (ROI). The first modal image is input into an image processing model, which performs ROI segmentation and modality transformation on the first modal image to obtain a second modal image including the ROI. This method, while performing modality transformation on the image containing the ROI using the image processing model, also segments the image into its ROI. This segmentation process accurately distinguishes the boundaries between the ROI and other regions. By fusing the segmentation and modality transformation processes, the quality of the second modal image obtained through modality transformation is improved. Attached Figure Description

[0038] Figure 1 This is an application environment diagram of an image processing method in one embodiment;

[0039] Figure 2 This is a flowchart illustrating an image processing method in one embodiment;

[0040] Figure 3 This is a flowchart illustrating an image processing method in one embodiment;

[0041] Figure 4 This is a flowchart illustrating an image processing method in one embodiment;

[0042] Figure 5 This is a flowchart illustrating an image processing method in one embodiment;

[0043] Figure 6 This is a flowchart illustrating an image processing method in one embodiment;

[0044] Figure 7 This is a flowchart illustrating an image processing method in one embodiment;

[0045] Figure 8 This is a flowchart illustrating an image processing method in one embodiment;

[0046] Figure 9 This is a flowchart illustrating an image processing method in one embodiment;

[0047] Figure 10 This is a schematic diagram of the network framework of an image processing model in one embodiment;

[0048] Figure 11 This is a flowchart illustrating an image processing method in one embodiment;

[0049] Figure 12 This is a structural block diagram of an image processing device in one embodiment. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] Before providing a detailed introduction to the technical solution of this application, let me first briefly introduce the technical background of this application.

[0052] Imaging of target sites using 18F-fluorodeoxyglucose (FDG) positron emission tomography (PET) allows for the determination of glucose metabolic activity at the target site. Furthermore, lesions (e.g., tumors) exhibit more active glucose metabolism than normal tissue, and PCT images accurately reveal the location and extent of lesions. Therefore, compared to computed tomography (CT) and magnetic resonance imaging (MRI), PET imaging offers significant imaging advantages in identifying early, small lesions and in lesion staging.

[0053] PET images are often combined with CT images for the diagnosis, treatment, and monitoring of tumors. Specifically, CT images provide anatomical information about the target area, which can help with attenuation correction of PET images.

[0054] However, the clinical application of PET / CT equipment faces significant limitations for the following reasons: Firstly, the penetration rate of PET / CT equipment in China is low, failing to meet user demand. PET / CT equipment is only widely available in a few major cities and higher-level medical institutions, with most primary healthcare facilities lacking it. Secondly, the cost of using PET / CT equipment is high, and it poses radiation risks. A single PET / CT scan is expensive and exposes the user to radiation; the radiation dose from a single scan is approximately 10-30 times the total environmental radiation a user receives in a year. Furthermore, after the scan, the nuclear tracer in the user's body cannot be immediately eliminated, potentially acting as a mobile radiation source and causing secondary radiation to the surrounding environment and population.

[0055] To avoid radiation from PET / CT equipment, image style transfer technology can be used to analyze CT images of the target area and convert them into PET images for early detection of small lesions. The radiation from a single enhanced CT scan is only one-fifth that of a PET scan, thus reducing the radiation exposure from the PET equipment. Furthermore, the cost of a single enhanced CT scan is only about one-twentieth that of a PET scan, reducing the financial burden on users.

[0056] Image style transfer technology has been widely used in the synthesis of multimodal medical images. It achieves the transformation between modalities through image style transfer and uses adversarial loss to increase the capture of detailed tissue structures.

[0057] However, due to the insufficient expression of the contextual relationship between healthy and lesion tissues by image style transfer technology, the quality of the obtained PET images is poor.

[0058] Therefore, to address the aforementioned problems, this application provides an image processing method that integrates a segmentation process during image modality conversion, resulting in a higher quality converted modal image. Of course, the technical solutions provided in the embodiments of this application are not limited to solving only the above problems, and also possess other technical effects, which can be found in the following embodiments. The technical solutions of this application will now be described in detail.

[0059] The image processing method provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown.

[0060] The computer equipment in this application environment can be a server, personal computer, laptop, smartphone, tablet, mobile phone, etc. This computer equipment may include a processor, memory, and network interface connected via a system bus or wirelessly. The processor provides computing and control capabilities. The memory may include non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data during image processing. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements an image processing method. This computer equipment can be implemented using a standalone computer or a cluster of multiple computer devices. It should be noted that the memory of this computer equipment is not limited to the above-mentioned memory types and may also include high-speed random access memory, volatile solid-state memory, etc. Furthermore, the architecture of this computer equipment is not limited to the above-described cases; some components may be added or omitted.

[0061] In one embodiment, such as Figure 2 As shown, an image processing method is provided, which is applied to... Figure 1 Taking a computer device as an example, the explanation includes the following steps:

[0062] S201, acquire the first modal image of the target region; the target region includes the region of interest.

[0063] The target site can be any part of the human body, such as the head and neck, stomach, or abdominal and pelvic region. The region of interest can be the lesion area within the target site (e.g., the tumor area within the target site), the lesion area and surrounding areas (e.g., other tissues and / or organs around the tumor), or part or all of the corresponding area of ​​the lesion area. It should be noted that since the target site is larger than or equal to the region of interest, the target site can be used as the region of interest.

[0064] Different modal images refer to images obtained by imaging devices using different imaging principles. That is, imaging devices with different imaging principles will acquire different modal images. For example, when the medical device is a computed tomography (CT) scanner, the corresponding scan image can be a first modal image; when the medical device is a magnetic resonance imaging (MRI) scanner, the corresponding scan image can be a second modal image; and when the medical device is a PET scanner, the corresponding scan image can be a third modal image.

[0065] In this embodiment, the computer device can acquire a first modal image of the target area from a Picture Archiving and Communication Systems (PACS). Alternatively, the computer device can determine the imaging device corresponding to the modal image required for the target area. An imaging command is sent to the imaging device corresponding to the modal image, and the imaging device can acquire an image of the target area according to the imaging command and send the image to the computer device. Based on this, the computer device can acquire the first modal image of the target area. This embodiment does not limit the method of acquiring the first modal image of the target area.

[0066] S202, the first modal image is input into the image processing model, and the image processing model performs region of interest segmentation and modality transformation on the first modal image to obtain a second modal image including the region of interest.

[0067] The image processing model is a pre-trained model. During training, this model has learned the processes of region-of-interest (ROI) segmentation and mode transformation; that is, the trained model possesses the ability to perform RIO and mode transformation. The image processing model can be a Generative Adversarial Network (GAN), which may have a Convolutional Neural Network (CNN) as its backbone. As a non-limiting example, the image processing model can be an end-to-end model or a partially end-to-end model.

[0068] The applicant discovered that during training, region-of-interest (ROI) segmentation can distinguish ROIs from other background regions, and the parameters of the image processing model can be adjusted based on the segmentation results. Thus, through continuous training, the modality transformation performance of the image processing model improves. Therefore, a single model with both RIO and modality transformation capabilities (e.g., a single generative adversarial network) can achieve better modality transformation performance while ensuring a less complex training process and a simpler model structure.

[0069] In this embodiment, after acquiring the first modal image, the computer device can input the first modal image into an image processing model. The image processing model then segments the region of interest (ROI) from the first modal image. Simultaneously, based on the conversion relationship between the first and second modalities, the modality of the first modal image is converted to obtain a second modal image including the ROI. It should be noted that the second modal image can also be an image containing an outline of the ROI.

[0070] Specifically, during modality transformation, the image processing model can extract global features from the first modality image and, based on these global features and the transformation relationship, obtain the second modality image of the region of interest. Alternatively, the image processing model can extract local features from the region of interest in the first modality image and, based on these local features and the transformation relationship, obtain the second modality image of the region of interest. Furthermore, the image processing model can fuse the second modality image corresponding to the global features and the second modality image corresponding to the local features, and use the fused result as the second modality image of the region of interest.

[0071] In the aforementioned image processing method, a first modal image of the target region is acquired; the target region includes a region of interest (ROI); the first modal image is input into an image processing model, which performs ROI segmentation and modality transformation on the first modal image to obtain a second modal image including the ROI. This method, while performing modality transformation on the image containing the ROI using the image processing model, also performs ROI segmentation on the image. This segmentation process accurately distinguishes the boundaries between the ROI and other regions. By fusing the segmentation and modality transformation processes, the quality of the second modal image obtained through modality transformation is improved.

[0072] Image modality conversion primarily relies on image processing models. The modality conversion process can be categorized into three cases: conversion based on global features, conversion based on local features, and conversion by fusing global and local features.

[0073] First, the process of transformation based on global features will be described in detail. Assuming the image processing model includes a global prediction model, in one embodiment, such as... Figure 3 As shown, the process of segmenting the region of interest and modal transformation of the first modality image using an image processing model to obtain a second modality image including the region of interest is described in detail. This step includes the following:

[0074] S301, the global prediction image is obtained by performing region of interest segmentation and mode transformation on the first modality image through a global prediction model.

[0075] The global prediction model is trained based on the complete sample modal image of any part, and can predict the first modal image of the target part as a whole.

[0076] In this embodiment, the computer device can input a first modal image into a global prediction model, which can extract global feature information from the first modal image. Based on this global feature information, the region of interest (ROI) of the target area is determined and segmented to distinguish it from other areas. Simultaneously, based on the global feature information, modal transformation is performed on the first modal image to obtain a global prediction image of the ROI.

[0077] S302, use the global prediction image as the second modality image.

[0078] In this embodiment, after obtaining the global prediction image, the computer device can directly use the global prediction image as the second modality image. Furthermore, it is understood that the computer device can also fuse the segmentation result of the region of interest with the global prediction image to obtain a second modality image containing the outline of the region of interest.

[0079] In the image processing method described above, a global prediction model is used to segment the region of interest and perform mode transformation on the first modality image to obtain a globally predicted image; this globally predicted image is then used as the second modality image. This method, by performing global prediction on the first modality image, allows for a more comprehensive mode transformation from a global perspective, resulting in a second modality image with more complete information.

[0080] The process of transformation based on local features will be described in detail below. Assuming the image processing model includes a local prediction model, in one embodiment, such as... Figure 4 As shown, the process of segmenting the region of interest and modal transformation of the first modality image using an image processing model to obtain a second modality image including the region of interest will be described in detail. This step may also include the following:

[0081] S401, the first modal image is cropped to obtain a first modal local image including the region of interest.

[0082] In this embodiment, the computer device can crop the background region in the first modality image based on a cropping command triggered by the user, and use the remaining image as the first modality local image of the region of interest. Alternatively, the computer device can also identify the contour of the region of interest in the first modality image using a contour recognition model, and crop based on the contour region to obtain the first modality local image of the region of interest.

[0083] It should be noted that the first modal local image can be a first modal image that only includes the region of interest, or it can be a first modal image that includes the region of interest and the surrounding region.

[0084] S402, the local image of the first modality is segmented into regions of interest and modally transformed using a local prediction model to obtain a local prediction image; the local prediction image is then used as the second modality image.

[0085] In this embodiment, the computer device can input a first modal local image into a local prediction model, which can extract local feature information from the first modal local image. Based on this local feature information, the region of interest (ROI) of the target area is determined and segmented to distinguish it from other areas. Simultaneously, based on the local feature information, modal transformation is performed on the first modal local image to obtain a local prediction image of the ROI. The obtained local prediction image is then used as the second modal image.

[0086] In the image processing method described above, the first modality image is cropped to obtain a local image of the first modality including the region of interest; the local image of the first modality is then segmented into its region of interest and modally transformed using a local prediction model to obtain a local prediction image; this local prediction image is then used as the second modality image. This method, by cropping the first modality image and then performing modal transformation on the local image from a local perspective, makes the modal transformation and segmentation process more targeted, resulting in a more accurate second modality image.

[0087] Finally, the process of fusing global and local features for transformation is described in detail. Assuming the image processing model includes a global prediction model, a local prediction model, and a fusion model, in one embodiment, such as... Figure 4 As shown, the above steps may also include:

[0088] S501, the global prediction image is obtained by performing region of interest segmentation and mode transformation on the first modality image through a global prediction model.

[0089] In this embodiment, the computer device can input a first modal image into a global prediction model, which can extract global feature information from the first modal image. Based on this global feature information, the region of interest (ROI) of the target area is determined and segmented to distinguish it from other areas. Simultaneously, based on the global feature information, modal transformation is performed on the first modal image to obtain a global prediction image of the ROI.

[0090] S502, the first modal image is cropped to obtain a first modal local image including the region of interest, and the region of interest is segmented and modal transformed in the first modal local image through a local prediction model to obtain a local prediction image.

[0091] In this embodiment, the computer device can crop the background region in the first modality image and use the remaining image as the first modality local image of the region of interest. The first modality local image is then input into a local prediction model, which extracts local feature information from the first modality local image. Based on this local feature information, the region of interest of the target area is determined and segmented to distinguish it from other areas. Simultaneously, based on the local feature information, modality transformation is performed on the first modality local image to obtain a local prediction image of the region of interest. The obtained local prediction image is then used as the second modality image.

[0092] S503: Input the global prediction image and the local prediction image into the fusion model to obtain the second modality image.

[0093] In this embodiment, the global prediction image contains more comprehensive feature information, while the local prediction image contains richer detail information. Therefore, after acquiring the global and local prediction images, the computer device can input both images into a fusion model, which then fuses the two prediction images to obtain the second modality image.

[0094] Specifically, during the fusion process, the global product of the global weights and the feature information of the global predicted image can be calculated, as well as the local product of the local weights and the feature information of the local predicted image. The image corresponding to the sum of the global product and the local product is used as the second modality image.

[0095] In the image processing method described above, a global prediction model is used to segment the region of interest (ROI) and perform mode transformation on the first modality image to obtain a global prediction image. The first modality image is then cropped to obtain a local first modality image including the ROI. A local prediction model is then used to segment the ROI and perform mode transformation on the local first modality image to obtain a local prediction image. The global and local prediction images are then input into a fusion model to obtain a second modality image. This method first performs mode transformation on the first modality image using both global and local predictions, and then fuses the transformation results. This results in a second modality image that includes not only more comprehensive global features but also more detailed local features, leading to higher quality second modality images.

[0096] In one embodiment, such as Figure 6 As shown, one implementation of the above-mentioned "inputting the global prediction image and the local prediction image into the fusion model to obtain the second modality image" includes:

[0097] S601, the product of the feature value of the global prediction image and the corresponding first weight is used as the global feature, and the product of the local prediction image and the corresponding second weight is used as the local feature.

[0098] In the process of fusion, the fusion model fuses the global features of the global prediction image with the local features of the local prediction image, and uses the image corresponding to the fused features as the second modality image.

[0099] In this embodiment, the computer device calculates the product of the feature value of the global predicted image and the corresponding first weight, and uses the product result as the global feature. It also calculates the product of the local predicted image and the corresponding second weight, and uses the product result as the local feature.

[0100] It should be noted that the first and second weights can be set manually. As a non-restrictive example, the sum of the first and second weights can be 1.

[0101] S602 generates a second modality image based on global and local features.

[0102] In this embodiment, after obtaining global and local features, the computer device can combine the global and local features to obtain combined features. The image corresponding to the combined features is then used as the second modality image.

[0103] In the image processing method described above, the product of the feature value of the globally predicted image and its corresponding first weight is used as the global feature, and the product of the local predicted image and its corresponding second weight is used as the local feature. A second modality image can be generated based on these global and local features. This method accurately calculates the global and local features by multiplying the feature value by its corresponding weight, and then combines these features to obtain a more accurate second modality image.

[0104] The above embodiments are all explanations of the application process of image processing models. In one embodiment, such as Figure 7 As shown below, the specific details of the training process for the image processing model will be introduced next.

[0105] S701, acquire the first modality sample image.

[0106] The first modality sample image is an image obtained by scanning with an imaging device of the first modality. The first modality sample image can be a global image of a certain part or a local image of a region of interest in that part.

[0107] In this embodiment of the application, the computer device can acquire multiple images of the first modality from the PACS and use the multiple images of the first modality as first modality sample images.

[0108] S702, the first modality sample image is input into the initial generator of the initial image processing model to obtain the second modality prediction image and the segmentation result of the region of interest.

[0109] In this embodiment, the computer device can input a first modality sample image into the initial generator of an initial image processing model. The initial generator extracts features from the first modality sample image to obtain sample image features. Based on the sample image features, the region of interest (ROI) of the first modality sample image is segmented to obtain the ROI segmentation result. Simultaneously, based on the sample image features, modality transformation is performed on the first modality sample image to obtain a second modality prediction image.

[0110] As an unrestricted example, when the initial image processing model is a GAN network, a branch is built on the generator of the GAN network, a supervised segmentation task is added, and Dice loss is introduced as a weakly supervised term to assist the iteration of the generation task. After the first modality sample image is input into the segmentation network, the automatic segmentation result is output. S703, based on the second modality predicted image and the segmentation result of the region of interest, the parameters of the initial generator are adjusted until the second modality predicted image output by the initial generator meets the preset conditions, thus obtaining the image processing model.

[0111] In this embodiment, after obtaining the second modality prediction image and the segmentation result of the region of interest, the computer device can adjust the parameters of the initial generator based on the second modality prediction image and the segmentation result of the region of interest, and determine whether the output result of the initial generator meets the preset conditions after the adjustment. If not, the next adjustment is performed based on the current output result. Through continuous parameter adjustment, the second modality prediction image output by the initial generator meets the preset conditions. At this point, the initial image processing model training is completed, and the image processing model is obtained.

[0112] In the image processing method described above, a first modality sample image is acquired; this first modality sample image is input into the initial generator of the initial image processing model to obtain a second modality prediction image and a segmentation result of the region of interest; based on the second modality prediction image and the segmentation result of the region of interest, the parameters of the initial generator are adjusted until the second modality prediction image output by the initial generator meets preset conditions, thus obtaining the image processing model. This method performs region of interest segmentation and modality conversion through the initial generator in the model, and adjusts the parameters based on the output results. In this way, the parameters of the initial generator can be gradually adjusted to the optimal level during training, resulting in a more accurate image processing model.

[0113] The above embodiments describe the training process of the image processing model. In one embodiment, such as... Figure 8 As shown below, the processing procedure of the initial generator will be described in detail below.

[0114] S801, Obtain the second modality sample image corresponding to the first modality sample image; the second modality sample image includes a region of interest label corresponding to the region of interest.

[0115] In this embodiment, when there are multiple first modal sample images, each first modal sample image has different identification information. Therefore, for any given first modal sample image, the computer device can determine an image with the same identification information from the PACS system based on the identification information of the first modal sample image, and use that image as the corresponding second modal sample image. Both the second modal image and the first modal image can include a region of interest.

[0116] A region of interest (ROI) label can be any information used to mark or describe the region of interest. For example, an ROI label can be an outline that traces or circles the region of interest. Alternatively, an ROI label can be a color block that covers the region of interest and has the same size as the region of interest. Or, an ROI label can be textual information that describes the location, size, and / or shape of the region of interest. This application does not limit the type of ROI label.

[0117] S802, based on the second modality sample image, the first modality sample image is segmented by the initial generator of the initial image processing model to obtain the segmentation result of the region of interest, and the first modality sample image is modally transformed by the initial generator to obtain the second modality prediction image.

[0118] In this embodiment, the second modality sample image, including the region of interest (ROI) label, can serve as a standard. The computer device can segment the first modality sample image according to this standard, resulting in a more accurate ROI segmentation. Furthermore, the computer device can also use an initial generator to convert the first modality sample image into the second modality, obtaining a second modality prediction image.

[0119] Since the image processing model involves segmentation and mode transformation tasks during training, the initial total loss function L of the image processing model includes the loss function L1 for the mode transformation task and the loss function L2 for the segmentation task. Through continuous iteration, the total loss function L is minimized. This allows the generator's generation accuracy and the discriminator's discrimination ability to continuously improve during training, ultimately reaching a "Nash equilibrium," where the segmentation task achieves good segmentation results, and the second modality predicted image is closer to the second modality sample image.

[0120] In the above image processing method, a second modality sample image corresponding to the first modality sample image is obtained. The second modality sample image includes a region of interest (ROI) label. Based on the second modality sample image, the first modality sample image is segmented using the initial generator of the initial image processing model to obtain the ROI segmentation result. Furthermore, the first modality sample image is modally transformed using the initial generator to obtain the second modality prediction image. This method uses the second modality sample image including the ROI label as a standard during segmentation, thus requiring only comparison with the standard for segmentation, resulting in more accurate segmentation results.

[0121] In one embodiment, such as Figure 9 As shown, one implementation of the above-mentioned "adjusting the parameters of the initial generator based on the segmentation results of the second modality predicted image and the region of interest" can be:

[0122] S901, the second modality prediction image and the second modality sample image are input into the initial discriminator of the initial image processing model to obtain the discrimination result.

[0123] The initial discriminator of the initial image processing model is used to determine the authenticity of the second modality predicted image.

[0124] In this embodiment, the computer device can input the second modality predicted image and the second modality sample image into the initial discriminator of the initial image processing model. The initial discriminator calculates the error between the second modality predicted image and the second modality sample image. This error value is then compared with a preset error value. If the error value is less than the preset error value, the discrimination result is determined to be true, and no further parameter adjustment is needed; the training process is complete. If the error value is greater than or equal to the preset error value, the discrimination result is determined to be false, and the model parameters still need to be adjusted.

[0125] S902, based on the discrimination results and the segmentation results of the region of interest, adjust the parameters of the initial image processing model.

[0126] In this embodiment, if the discrimination result is not true, the computer device can adjust the parameters of the initial image processing model based on the segmentation result of the region of interest and the error value between the second modality predicted image and the second modality sample image. In the next training round, the second modality predicted image is generated again by the initial generator in the initial image processing model, and the second modality predicted image is discriminated against by the initial discriminator. If the discrimination result is not true, parameter adjustment and subsequent training continue until the discrimination result is true, at which point training stops, and the image processing model is obtained.

[0127] In the aforementioned image processing method, the second modality prediction image and the second modality sample image are input into the initial discriminator of the initial image processing model to obtain the discrimination result. Based on the discrimination result and the segmentation result of the region of interest, the parameters of the initial image processing model are adjusted. This method uses the initial discriminator to judge the second modality prediction image and the second modality sample image, which can accurately determine the discrimination result. Furthermore, the segmentation result is introduced during the parameter adjustment process. The segmentation result can distinguish the region of interest from other regions, thus making the output result of the initial image processing model more accurate, thereby making the training process of the initial image processing model more efficient, and consequently, the image quality obtained after the image processing model performs modality transformation is higher.

[0128] The training process of the image processing model was described in the above embodiments. This image processing model can adopt a cascaded network strategy, that is, cascading a coarse multi-task model (Coarse Stage) and a fine multi-task model (Fine Stage) to form the complete network framework of the image processing model. The network frameworks of the two models are exactly the same, the only difference being the input image and the output image.

[0129] Figure 10 This diagram illustrates the network framework of an image processing model, including a coarse-grained multi-task model and a fine-grained multi-task model. Both models include a generator and a discriminator. It's important to note that the sample images used in the coarse-grained multi-task model include both first-modality and second-modality sample images, while the sample images used in the fine-grained multi-task model include local images of both the first-modality and second-modality samples. Aside from this, the training processes for both models are identical.

[0130] In one embodiment, such as Figure 11 As shown, the specific steps of the above image processing method may include:

[0131] S1001, Obtain the first modal image of the target region;

[0132] S1002, the global prediction image is obtained by performing region of interest segmentation and mode transformation on the first modality image through a global prediction model;

[0133] S1003, crop the first modality image to obtain a first modality local image including the region of interest;

[0134] S1004, The local prediction image is obtained by performing region of interest segmentation and mode transformation on the local image of the first mode through the local prediction model;

[0135] S1005, the product of the feature value of the global prediction image and the corresponding first weight is used as the global feature, and the product of the local prediction image and the corresponding second weight is used as the local feature.

[0136] S1006, based on global and local features, generates a second modality image.

[0137] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0138] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0139] In one embodiment, such as Figure 12 As shown, an image processing apparatus is provided, including: an image acquisition module 11 and a mode conversion module 12, wherein:

[0140] Image acquisition module 11 is used to acquire a first modal image of the target region; the target region includes a region of interest;

[0141] The modality conversion module 12 is used to input the first modality image into the image processing model, and to perform region of interest segmentation and modality conversion on the first modality image through the image processing model to obtain a second modality image including the region of interest.

[0142] In one embodiment, the modality conversion module 12 includes: a first conversion unit and a first determination unit, wherein:

[0143] The first conversion unit is used to perform region of interest segmentation and mode conversion on the first modality image through a global prediction model to obtain a global prediction image;

[0144] The first determining unit is used to take the global prediction image as the second modality image.

[0145] In one embodiment, the modality conversion module 12 further includes: a trimming unit and a second conversion unit, wherein:

[0146] The cropping unit is used to crop the first modality image to obtain a first modality local image including the region of interest;

[0147] The second conversion unit is used to perform region of interest segmentation and mode conversion on the local image of the first modality using a local prediction model to obtain a local prediction image; and to use the local prediction image as the second modality image.

[0148] In one embodiment, the modality conversion module 12 further includes: a third conversion unit, a fourth conversion unit, and a second determination unit, wherein:

[0149] The third conversion unit is used to perform region of interest segmentation and mode conversion on the first modality image through a global prediction model to obtain a global prediction image;

[0150] The fourth conversion unit is used to crop the first modal image to obtain a first modal local image including the region of interest, and to perform region of interest segmentation and modal conversion on the first modal local image through a local prediction model to obtain a local prediction image;

[0151] The second determining unit is used to input the global prediction image and the local prediction image into the fusion model to obtain the second modality image.

[0152] In one embodiment, the second determining unit is further configured to use the product of the feature value of the global prediction image and the corresponding first weight as the global feature, and the product of the local prediction image and the corresponding second weight as the local feature; and generate a second modality image based on the global feature and the local feature.

[0153] In one embodiment, the image processing apparatus further includes: a sample image acquisition module, a segmentation result determination module, and a parameter adjustment module, wherein:

[0154] The sample image acquisition module is used to acquire the first modality sample image;

[0155] The segmentation result determination module is used to input the first modality sample image into the initial generator of the initial image processing model to obtain the second modality prediction image and the segmentation result of the region of interest;

[0156] The parameter adjustment module is used to adjust the parameters of the initial generator based on the second modality prediction image and the segmentation results of the region of interest, until the second modality prediction image output by the initial generator meets the preset conditions, thus obtaining the image processing model.

[0157] In one embodiment, the segmentation result determination module includes: a sample image acquisition unit and a segmentation unit, wherein:

[0158] The sample image acquisition unit is used to acquire a second modality sample image corresponding to the first modality sample image; the second modality sample image includes a region of interest label corresponding to the region of interest;

[0159] The segmentation unit is used to segment the first modality sample image based on the second modality sample image by using the initial generator of the initial image processing model to obtain the segmentation result of the region of interest, and to perform modality transformation on the first modality sample image by using the initial generator to obtain the second modality prediction image.

[0160] In one embodiment, the parameter adjustment module includes: a third determining unit and a parameter adjustment unit, wherein:

[0161] The third determining unit is used to input the second modality prediction image and the second modality sample image into the initial discriminator of the initial image processing model to obtain the discrimination result;

[0162] The parameter adjustment unit is used to adjust the parameters of the initial image processing model based on the discrimination results and the segmentation results of the region of interest.

[0163] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0164] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the content of any of the embodiments of the above-described image processing methods.

[0165] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the content of any one of the embodiments of the above image processing methods.

[0166] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the content of any one of the embodiments of the above-described image processing methods.

[0167] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0168] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0169] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0170] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that, include: Acquire a first modal image of the target region; the target region includes a region of interest; The first modal image is input into an image processing model, and the image processing model performs region of interest segmentation and modality transformation on the first modal image to obtain a second modal image including the region of interest.

2. The image processing method according to claim 1, characterized in that, The image processing model includes a global prediction model; the step of performing region of interest segmentation and modality transformation on the first modal image using the image processing model to obtain a second modal image including the region of interest includes: The first modality image is segmented into regions of interest and modality transformed using a global prediction model to obtain a global prediction image; The global prediction image is used as the second modality image.

3. The image processing method according to claim 1, characterized in that, The image processing model includes a local prediction model; the step of performing region-of-interest segmentation and modality transformation on the first modal image using the image processing model to obtain a second modal image including the region of interest includes: The first modal image is cropped to obtain a first modal local image including the region of interest; The local image of the first modality is segmented into regions of interest and modally transformed using a local prediction model to obtain a local prediction image; the local prediction image is then used as the second modality image.

4. The image processing method according to claim 1, characterized in that, The image processing model includes a global prediction model, a local prediction model, and a fusion model. The step of performing region-of-interest (ROI) segmentation and modality transformation on the first modal image using the image processing model to obtain a second modal image including the ROI includes: The first modality image is segmented into regions of interest and modality transformed using a global prediction model to obtain a global prediction image; The first modal image is cropped to obtain a first modal local image including the region of interest, and the first modal local image is segmented and modally transformed using a local prediction model to obtain a local prediction image; The global prediction image and the local prediction image are input into the fusion model to obtain the second modality image.

5. The image processing method according to claim 4, characterized in that, The step of inputting the global prediction image and the local prediction image into the fusion model to obtain the second modality image includes: The product of the feature value of the global predicted image and the corresponding first weight is used as the global feature, and the product of the local predicted image and the corresponding second weight is used as the local feature. The second modality image is generated based on the global features and the local features.

6. The image processing method according to claim 1, characterized in that, The process of constructing the image processing model includes: Acquire the first modality sample image; The first modality sample image is input into the initial generator of the initial image processing model to obtain the second modality prediction image and the segmentation result of the region of interest; Based on the second modality prediction image and the segmentation result of the region of interest, the parameters of the initial generator are adjusted until the second modality prediction image output by the initial generator meets the preset conditions, thereby obtaining the image processing model.

7. The image processing method according to claim 6, characterized in that, The step of inputting the first modality sample image into the initial generator of the initial image processing model and outputting the second modality prediction image and the segmentation result of the region of interest includes: Obtain a second modality sample image corresponding to the first modality sample image; the second modality sample image includes a region of interest label corresponding to the region of interest; Based on the second modality sample image, the first modality sample image is segmented by the initial generator of the initial image processing model to obtain the segmentation result of the region of interest, and the first modality sample image is modally transformed by the initial generator to obtain the second modality prediction image.

8. The image processing method according to claim 7, characterized in that, The step of adjusting the parameters of the initial generator based on the second modality prediction image and the segmentation result of the region of interest includes: The second modality prediction image and the second modality sample image are input into the initial discriminator of the initial image processing model to obtain the discrimination result; Based on the discrimination result and the segmentation result of the region of interest, the parameters of the initial image processing model are adjusted.

9. An image processing apparatus, characterized in that, include: The image acquisition module is used to acquire a first modal image of the target region; the target region includes a region of interest. The modality conversion module is used to input the first modality image into the image processing model, and to perform region of interest segmentation and modality conversion on the first modality image through the image processing model to obtain a second modality image including the region of interest.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images