Image processing method, image processing apparatus, device, and medium

Abstract

Embodiments of the present application provide an image processing method, device, equipment and medium, the method comprises: decomposing a plurality of three-dimensional component images from an original three-dimensional image, different three-dimensional component images correspond to different frequencies; from a plurality of three-dimensional component images, obtain a target three-dimensional component image adapted to a specified target object; input the target three-dimensional component image into the image generation model corresponding to the target object, to generate a target three-dimensional image based on the target three-dimensional component image; wherein the image quality of the target three-dimensional image is higher than the image quality of the original three-dimensional image, the image quality includes at least one of the following: contrast, texture definition; display the target three-dimensional image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, image processing apparatus, device and medium. Background Technology

[0002] Three-dimensional images are increasingly being used in various fields due to their rich information content. However, with the development of 3D image applications, the demand for high-quality 3D images is growing.

[0003] For example, in the medical field, high-quality medical images provide clearer information about lesions, which is more helpful for doctors in diagnosing conditions. Similarly, in object detection, high-quality 3D scene images can contain more discriminative object information, making it easier to track and detect target objects.

[0004] In related technologies, high-quality three-dimensional images are generally obtained through expensive equipment. Especially in the medical field, obtaining high-quality three-dimensional medical images not only requires expensive instruments, but also places high demands on patients, such as requiring patients to remain still for a long time and using high doses of contrast agents. This leads to the problem of high acquisition costs. Summary of the Invention

[0005] In view of the above problems, an image processing method, image processing apparatus, device and medium according to embodiments of this application are proposed to overcome the above problems or at least partially solve the above problems.

[0006] To address the aforementioned problems, a first aspect of this application discloses an image processing method, the method comprising:

[0007] Multiple three-dimensional component images are decomposed from the original three-dimensional image, and different three-dimensional component images correspond to different frequencies;

[0008] From multiple 3D component images, obtain the target 3D component image that is adapted to the specified target object;

[0009] The target 3D component image is input into the image generation model corresponding to the target object to generate a target 3D image based on the target 3D component image; wherein, the image quality of the target 3D image is higher than that of the original 3D image, and the image quality includes at least one of the following: contrast and texture sharpness;

[0010] Display the three-dimensional image of the target.

[0011] Optionally, from multiple 3D component images, a target 3D component image adapted to the specified target object is obtained, including:

[0012] From a preset association table, the target frequency range corresponding to the target object is determined; wherein, the preset association table includes the frequency range corresponding to each type of object;

[0013] From multiple three-dimensional component images, determine the target three-dimensional component image whose frequency belongs to the target frequency range.

[0014] Optionally, determining the frequency range corresponding to each object includes:

[0015] For each object, acquire multiple initial 3D images including that object;

[0016] The initial three-dimensional images are decomposed to obtain multiple first three-dimensional component images corresponding to each initial three-dimensional image;

[0017] The sharpness of the object in multiple first three-dimensional component images is determined respectively;

[0018] From a plurality of first three-dimensional component images, at least one second three-dimensional component image that meets preset conditions is determined;

[0019] Based on the frequencies corresponding to multiple second- and third-dimensional component images, the frequency range corresponding to this type of object is determined.

[0020] Optionally, the image generation module includes: a convolution module, and a feature enhancement module and an upsampling module sequentially connected after the convolution module; inputting the target 3D component image into the image generation model corresponding to the target object to generate a target 3D image based on the target 3D component image includes:

[0021] The convolution module is used to extract three-dimensional features from the target three-dimensional component image to obtain a three-dimensional feature map;

[0022] The feature enhancement module iteratively performs multi-scale 3D convolution and fusion on the 3D feature map to obtain an enhanced 3D feature map.

[0023] The target 3D image is obtained by upsampling the enhanced 3D feature map using the upsampling module.

[0024] Optionally, the feature enhancement module includes multiple concatenated feature transfer layers and at least one convolutional module connected to each of the multiple feature transfer layers, wherein a three-dimensional convolutional unit is connected between each feature transfer layer and the convolutional module; the step of iteratively performing three-dimensional convolutions at multiple scales on the three-dimensional feature map through the feature enhancement module, and fusing the convolutional feature maps obtained from the three-dimensional convolutions at multiple scales to obtain an enhanced three-dimensional feature map includes:

[0025] The three-dimensional feature map is input into the feature transfer layer. After the three-dimensional feature map input into the feature transfer layer is 3D convolved by the feature transfer layer, it is fused with all the three-dimensional feature maps before the feature transfer layer and then input into the next feature transfer layer and the three-dimensional convolution unit.

[0026] The input features are convolved at the corresponding scale by the three-dimensional convolution unit and then input into the corresponding convolution module. The convolution module then performs three-dimensional convolution on the features output by the three-dimensional convolution unit.

[0027] Optionally, the image generation model is an adversarial network model, which inputs the target 3D component image into the image generation model corresponding to the target object to generate a target 3D image based on the target 3D component image, including:

[0028] The target 3D component image is input into the generator in the image generation model to generate the target 3D image.

[0029] Optionally, the image generation model is obtained through the following steps:

[0030] Obtain a training sample set, which includes multiple three-dimensional image samples and three-dimensional component image samples corresponding to the target object decomposed from each three-dimensional image sample;

[0031] Multiple three-dimensional component image samples are input into a generator in an adversarial network, and the generator generates a three-dimensional predicted image corresponding to the three-dimensional component image samples.

[0032] The discriminator in the adversarial network outputs the quality parameter difference between each of the three-dimensional predicted images and the corresponding three-dimensional image samples.

[0033] Based on the differences in the quality parameters, the adversarial network is iteratively updated to obtain the image generation model.

[0034] Optionally, based on the differences in quality parameters, the adversarial network is iteratively updated to obtain the image generation model, including:

[0035] Based on the difference in the quality parameters, a first loss value corresponding to the generator and a second loss value corresponding to the discriminator are determined.

[0036] The parameters of the generator are updated based on the first loss value, and the parameters of the discriminator are updated based on the second loss value.

[0037] Optionally, the method further includes:

[0038] Based on the first loss value and the second loss value, determine the overall loss value;

[0039] Based on the overall loss value, the parameters of the discriminator and the generator are updated.

[0040] Optionally, the step of decomposing multiple three-dimensional component images from the original three-dimensional image includes:

[0041] Step 1: Based on adjacent windows of a preset size, determine the local maxima and local minima of the three-dimensional volume from the original three-dimensional image;

[0042] Step 2: Using the corresponding filter, construct the average envelope of the original 3D image based on the local maxima and local minima;

[0043] Step 3: Based on the average envelope, determine the corresponding three-dimensional component image from the original three-dimensional image, and repeat steps 1-3 above until the decomposed three-dimensional component image meets the first preset condition.

[0044] Obtain multiple 3D component images at the end of the iteration.

[0045] Optionally, after determining the corresponding three-dimensional component image from the original three-dimensional image based on the average envelope, the method further includes:

[0046] The current three-dimensional component image is verified to determine whether the current three-dimensional component image meets the second preset condition;

[0047] If the condition is met, then based on the original three-dimensional image after removing the three-dimensional component image of the current time, repeat steps 1-3 above until the decomposed three-dimensional component image meets the first preset condition.

[0048] If the condition is not met, then based on the corresponding original 3D image, repeat steps 1 and 3 until the current 3D component image meets the second preset condition.

[0049] Optionally, the current three-dimensional component image is verified to determine whether the current three-dimensional component image meets the second preset condition, including:

[0050] Obtain the previous three-dimensional component image that satisfies the second preset condition as determined in the previous iteration;

[0051] Based on the previous three-dimensional component image, the current three-dimensional component image, and their respective three-dimensional coordinates, the standard deviation between two consecutive iterations is determined.

[0052] Based on the standard deviation, determine whether the current three-dimensional component image meets the second preset condition.

[0053] A third aspect of this application also discloses an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the image processing method as described in the first aspect embodiment.

[0054] A fourth aspect of this application also discloses a computer-readable storage medium storing a computer program that causes a processor to perform the image processing method as described in the first aspect of this application.

[0055] In this embodiment, multiple three-dimensional component images can be decomposed from the original three-dimensional image, and a target three-dimensional component image adapted to a specified target object can be obtained from the multiple three-dimensional component images. Then, the target three-dimensional component image is input into the image generation model corresponding to the target object to generate a target three-dimensional image based on the target three-dimensional component image. The quality parameter value corresponding to the target object in the target three-dimensional image is higher than the quality parameter value corresponding to the target object in the original three-dimensional image. The quality parameter value is used to characterize at least one of the following image qualities: image contrast and image texture.

[0056] On the one hand, since high-quality target 3D images are generated using image generation models, even if low-quality original 3D images are obtained using low-performance instruments, 3D images with improved quality can still be obtained through image generation models, thereby reducing the cost of obtaining high-quality 3D images.

[0057] On the other hand, since different three-dimensional component images correspond to different frequencies, and target three-dimensional component images that are compatible with the target object are selected from multiple three-dimensional component images, the target three-dimensional component images obtained in a targeted manner can fully reflect the characteristics of the target object, thereby enhancing the adaptability between the improved three-dimensional image and the target object.

[0058] On the other hand, since the target 3D image is superior to the original 3D image in at least one of image contrast and image texture, the generated target 3D image can more clearly depict the target object, thereby providing high-quality images for subsequent business and improving the accuracy of subsequent business. Attached Figure Description

[0059] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 A flowchart illustrating the steps of the image processing method in an embodiment of this application is shown;

[0061] Figure 2 A schematic diagram illustrating the steps of decomposing multiple 3D component images from an original 3D image is shown.

[0062] Figure 3 A flowchart illustrating the steps for verifying the current 3D component image is shown.

[0063] Figure 4 A schematic diagram illustrating the steps of training the image generation model of this application is shown;

[0064] Figure 5 A network structure diagram of an image generation model in one alternative embodiment is shown;

[0065] Figure 6 A schematic diagram illustrating the process of determining each object and its corresponding frequency range is shown.

[0066] Figure 7 This is a schematic diagram of the frame structure of the image processing device in an embodiment of this application;

[0067] Figure 8 This is a schematic diagram of the frame structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0068] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0069] Currently, deep learning has been widely researched and used in the field of image processing. With the expansion of image processing, deep learning is often used in some areas to improve the image quality of 3D images. This image quality improvement generally takes the following forms:

[0070] Method 1: Prepare low-resolution and high-resolution 3D images as a sample set to train the neural network model. However, this method still requires the use of high-resolution medical images as labels, and it is difficult to collect a sufficient number of high-resolution medical images in practice.

[0071] Method 2: Improve the quality of the generated image by adding lesion information to the medical image; however, this requires obtaining the lesion information separately, and the problem of difficulty in obtaining the information still exists.

[0072] In view of this, this application proposes an image processing method that can generate a high-quality target 3D image by utilizing the 3D component image adapted to the specified target object in the original 3D image. This avoids the need for costly methods to obtain high-quality 3D images and reduces the difficulty of sample acquisition.

[0073] The image processing method of this application will be described below:

[0074] Reference Figure 1 As shown, a flowchart illustrating the steps of the image processing method of this application is presented, as follows: Figure 1 As shown, this method can be applied to terminal devices, for example, to terminal devices used by physicians, and specifically includes the following steps:

[0075] Step S101: Decompose multiple three-dimensional component images from the original three-dimensional image. Different three-dimensional component images correspond to different frequencies.

[0076] The original three-dimensional image can be obtained by acquiring images of the target object. In the medical field, the original three-dimensional image can be obtained by scanning biological tissues using CT and MRI. For example, it can be a three-dimensional CT image or a three-dimensional MRI image, and the generated tissue can be organs such as the lungs or brain.

[0077] The original 3D image can have three dimensions: length, width, and height. In practice, frequency decomposition can be performed on the original 3D image. Specifically, 3D empirical mode decomposition can be used to decompose the original 3D image into multiple 3D component images, each corresponding to a different frequency. Each 3D component image contains image information of the target object.

[0078] The resolution and other image quality of the original 3D image are not required; that is, a low-quality original 3D image is acceptable.

[0079] Step S102: Obtain the target three-dimensional component image that is adapted to the specified target object from multiple three-dimensional component images.

[0080] In general, the original 3D image includes multiple objects in the target object. The image quality of these multiple objects can vary in different 3D component images. Some objects have higher image quality in the 3D component image, while others have lower image quality.

[0081] This process involves selecting the highest-quality 3D component image from multiple 3D component images to represent the target object. This high-quality 3D component image can be referred to as the target 3D component image. Specifically, the target object exhibits higher quality in the target 3D component image but lower quality in other 3D component images. This quality can refer to aspects such as sharpness and the amount of detail information.

[0082] In one alternative example, multiple 3D component images and the identifier of the target object can be displayed on the front end, along with a prompt message that prompts the user to select a target 3D component image from the multiple 3D component images. Then, in response to the user's selection of the 3D component images, the target 3D component image that matches the target object can be determined.

[0083] For example, taking a lung MRI image as the original 3D image, this lung MRI image can be decomposed into eight 3D component images, where the target object is a lung nodule. Since the image quality of the lung nodule can vary in different 3D component images, the nodule may appear clearer in one 3D component image and less clear in others. Therefore, these eight 3D component images can be displayed on the front end along with a prompt message to guide the user to select the 3D component image with higher clarity for the lung nodule; for example, the user selects 3D component image A as the target 3D component image.

[0084] Step S103: Input the target 3D component image into the image generation model corresponding to the target object to generate a target 3D image based on the target 3D component image.

[0085] The target 3D image has a higher image quality than the original 3D image, and the image quality includes at least one of the following: contrast and texture sharpness.

[0086] Step S104: Display the target 3D image.

[0087] Although the target object in the target 3D component image has a high quality, this is relative to other 3D component images. In practice, it is still necessary to improve the quality of the target 3D component image to generate a higher quality target 3D image.

[0088] Therefore, the target 3D component image can be input into the image generation model corresponding to the target object. The image generation model can generate a high-quality target 3D image based on the target 3D component image. The high-quality target 3D image has at least one of contrast and texture sharpness that is better than the original 3D image.

[0089] In some optional examples, to improve the targeting of image quality enhancement, an image generation model can be set for each target object. That is, different target objects can correspond to different image generation models. For example, for the lungs, lesions generally include lung nodules, lung vesicles, and lung tumors. Among them, lung nodules, lung vesicles, and lung tumors can each correspond to different image generation models. Thus, the image quality of the target object can be improved in a targeted manner through the image generation model. In other words, the image generation model can focus more on improving the quality of the target object.

[0090] The image generation model is used to convolve and enhance the features of the target 3D component image to obtain the target 3D image. As mentioned above, the target 3D image can still reflect all the feature information of the target object. Among these features, the information of the image region of the target object is enhanced in a more targeted manner, such as the texture quality of lung nodules being enhanced in a targeted manner.

[0091] After obtaining the target 3D image, it can be sent to the front end for display. Of course, in some embodiments, after obtaining the target 3D image, it can also be used for downstream services, such as image recognition tasks, image segmentation tasks, etc. Image segmentation tasks can refer to segmenting the region where the target object is located in the target 3D image. Image recognition tasks can refer to identifying the type of the target object. For example, in the medical field, image recognition tasks can refer to identifying the type of lesion, such as identifying the stage of lung tumors, gene mutation types, etc.

[0092] The image processing method in this embodiment generates a high-quality target 3D image using an image generation model. Therefore, even if a low-quality original 3D image is acquired using low-performance instruments, an improved 3D image can still be obtained through the image generation model, thereby reducing the cost of acquiring high-quality 3D images. Furthermore, target 3D component images adapted to the target object are selected from multiple 3D component images. This allows the specifically acquired target 3D component images to fully reflect the characteristics of the target object, thus enhancing the adaptability between the improved 3D image and the target object.

[0093] In some embodiments, the process of decomposing multiple three-dimensional component images from an original three-dimensional image can be accomplished using three-dimensional empirical mode decomposition.

[0094] Reference Figure 2 As shown, a schematic diagram illustrates the steps involved in decomposing a raw 3D image into multiple 3D component images. Figure 2 As shown, the specific steps may include:

[0095] Step 1: Based on adjacent windows of a preset size, determine the local maxima and local minima of the 3D volume from the original 3D image;

[0096] Step 2: Using the current filter, construct the average envelope of the original 3D image based on local maxima and local minima;

[0097] Step 3: Based on the average envelope, determine the corresponding three-dimensional component image from the original three-dimensional image, and repeat steps 1-3 above until the decomposed three-dimensional component image meets the first preset condition.

[0098] Step 4: Obtain multiple 3D component images at the end of the iteration.

[0099] In this embodiment, adjacent windows can be used to browse the original 3D image, thereby obtaining a mapping of local maxima and minima. Specifically, adjacent windows can be 3×3×3 windows, where local maxima and local minima can be represented by M respectively. max (m, n, b) and M min (m, n, b), where m, n, and b are the length, width, and height of the three-dimensional volume in the three-dimensional image; it should be noted that within each adjacent sliding window, there is a local maximum and a local minimum, and thus, the local maxima and local minimums of multiple sliding windows constitute the mapping described above.

[0100] This process involves iteratively decomposing the original 3D image multiple times using a corresponding filter based on the mapping between local maxima and local minima. Each iteration decomposes one 3D component image from the original 3D image. During each iteration to generate the 3D component image, the average envelope of the original 3D image can be constructed based on the local maxima and local minima using the currently applied filter. In other words, the filter removes some frequencies while retaining the rest. Specifically, the upper envelope E of each local maxima can be generated using a Max filter. max (m, n, b), the lower envelope E of each local minimum is generated by the Min filter. min (m, n, b), and then, the average envelope M(m, n, b) can be obtained from the upper envelope and the lower envelope;

[0101] in,

[0102] Next, the average envelope M(m, n, b) can be smoothed using an average filter to obtain the smoothed average envelope line, denoted as M. s (m, n, b); then, based on the average envelope, the corresponding 3D component image can be determined from the original 3D image. Specifically, the extracted 3D component image can be updated by removing the smoothed average envelope, and this 3D component image can be denoted as h. i，j (m, n, b), where h i，j (m, n, b) = h i，j-1 (m, n, b)-M s (m, n, b); where h i，j-1 (m, n, b) is the previously determined three-dimensional component image.

[0103] Specifically, for the first determined three-dimensional component image, this first three-dimensional component image is the image obtained by subtracting the average envelope from the original three-dimensional image.

[0104] Following the above process, the iteration is repeated multiple times, that is, steps 1-3 are repeated multiple times, until the first preset condition is met. The first preset condition is the end condition of the iteration. The end condition of the iteration can be that the number of extreme values ​​in a sliding window after sliding through adjacent windows is less than three. At this point, the iteration can be ended, and the remaining image in the original 3D image is used as the final residual.

[0105] After the above process, multiple three-dimensional component images can be obtained, which may not include the final residual image.

[0106] In some optional examples, after obtaining the corresponding 3D component image for the current time, the 3D component image can be verified to determine whether the 3D component image is qualified. Based on the verification result, it is determined whether the 3D component image for the current time should be discarded. If it should be discarded, the 3D component image for the current time is discarded, and then the filter is adjusted to redetermine the 3D component image for the current time until the 3D component image for the current time is determined to be qualified.

[0107] In practice, the current 3D component image can be verified to determine whether it meets the second preset condition. If it does, steps 1-3 above are repeated based on the original image after removing the current 3D component image, until the decomposed 3D component image meets the first preset condition; if it does not, the corresponding filter size is updated, and steps 2 and 3 are repeated until the current 3D component image meets the second preset condition.

[0108] In one alternative example, the second preset condition may refer to: in the three-dimensional component image, the number of extrema and the number of zero crossings are equal or differ by no more than one; and the average value of the upper envelope formed by local maxima and the lower envelope formed by local minima is zero, that is, the upper and lower envelopes are locally symmetrical with respect to the time axis.

[0109] If the three-dimensional component image meets the above conditions, the verification is successful; otherwise, it is unsuccessful. Specifically, when the three-dimensional component image meets the above conditions, it can be used as the component image obtained in this iteration. Then, using the original three-dimensional image after removing the three-dimensional component image as a reference, steps 1-3 are executed to continue decomposing the next three-dimensional component image from the three-dimensional image after removing the three-dimensional component image.

[0110] If the 3D component image does not meet the above conditions, the 3D component image can be discarded, and steps 1 and 3 can be repeated. Specifically, the corresponding original 3D image is still used as the basis, and the 3D component image is decomposed again from the original 3D image. In this decomposition, the size of the adjacent windows can be changed based on the Sané component image. For details, please refer to relevant technologies, which will not be elaborated here.

[0111] By verifying the component images as described above, the accuracy of the decomposed 3D component images can be improved, thus ensuring that each 3D component image contains useful information and improving the quality of the subsequent target 3D image.

[0112] In some alternative examples, the current three-dimensional component image is verified by verifying the difference between two consecutively determined three-dimensional component images. If the difference is large, the verification fails; if the difference is small, the verification passes. It should be noted that the two consecutively determined three-dimensional component images include the previously verified three-dimensional component image and the current three-dimensional component image.

[0113] In specific implementation, refer to Figure 3 As shown, a flowchart illustrating the steps for verifying the current 3D component image is presented. Figure 3 As shown, the specific steps may include:

[0114] Step S301: Obtain the previous three-dimensional component image that satisfies the second preset condition as determined in the previous iteration.

[0115] Step S302: Based on the previous 3D component image, the current 3D component image, and their respective 3D coordinates, determine the standard deviation between two consecutive iterations.

[0116] Step S303: Based on the standard deviation, determine whether the current three-dimensional component image meets the second preset condition.

[0117] In this embodiment, the previous three-dimensional component image refers to the previously verified qualified three-dimensional component image. Since each three-dimensional component image includes a three-dimensional volume, the standard deviation of the three-dimensional volume between the previous three-dimensional component image and the current three-dimensional component image can be determined based on their respective volumes.

[0118] Specifically, the standard deviation can be calculated using the following formula:

[0119]

[0120] Where (x, y, z) are the coordinates of each pixel in the 3D volume, and m, n, b are the length, width, and height of the 3D volume. SD ij It represents the standard deviation.

[0121] The standard deviation characterizes the difference between the current 3D component image and the previously validated 3D component image. A large difference indicates a discrepancy in the decomposition of the current 3D component image, while a small difference indicates no discrepancy. This validation method ensures that the differences between the multiple decomposed 3D component images are minimal, meaning that a single 3D component image can fully represent the information of the target object. This is because in consecutive decompositions, each decomposed image needs to reflect the information of the target object; therefore, the differences in information contained in multiple 3D component images should be limited to a certain range.

[0122] In practice, if the standard deviation is less than or equal to the preset standard deviation, it means that the current three-dimensional component image meets the second preset condition; if the standard deviation is greater than the preset standard deviation, it means that the current three-dimensional component image does not meet the second preset condition.

[0123] By adopting this implementation scheme, it can be ensured that each individual 3D component image can fully reflect the information of the target object, thereby improving the uniformity of the target object reflected by multiple 3D component images.

[0124] The following section introduces how to generate a target 3D image using an image generation model.

[0125] First, an image generation model can be trained. In some embodiments, different image generation models can be trained for different target objects. Taking the medical field as an example, the target object can refer to a lesion, such as a lung lesion, which generally includes nodules, vesicles, tuberculosis, tumors, etc. Therefore, different image generation models can be trained for different lesions to specifically improve the quality of 3D images.

[0126] In some embodiments, the image generation model may be an adversarial network model, which may include a generator and a discriminator, wherein the generator is used to generate a target 3D image, and the discriminator is used to identify the differences between the target 3D image and the original 3D image to determine whether the quality has been improved.

[0127] Accordingly, the target 3D component image can be input into the generator in the image generation model to generate the target 3D image.

[0128] In this process, when training the image generation model corresponding to the target object, the parameters of the adversarial network are updated using the difference between the 3D image output by the generator and the original 3D image, as determined by the discriminator. This eliminates the need for high-quality 3D image samples as labels, thus reducing the difficulty and cost of sample acquisition during model training.

[0129] In some embodiments, the original 3D image samples can be input into the generator of the adversarial network to obtain a high-quality target 3D image of the original 3D image samples. Of course, in this application, in order to ensure that the generated high-quality target 3D image can be fully adapted to the target object, for example, in the medical field, in order to ensure that the generated high-quality target 3D image can clearly reflect the lesion, that is, to make the image quality of the lesion area in the target 3D image more significantly improved, the 3D component image corresponding to the target object can be used as a training sample for input.

[0130] In specific implementation, refer to Figure 4 As shown, a flowchart illustrating the steps of training the image generation model of this application is presented. Figure 4 As shown, the specific steps may include:

[0131] Step S401: Obtain a training sample set, which includes multiple 3D image samples and 3D component image samples corresponding to the target object decomposed in each 3D image sample.

[0132] In this embodiment, the multiple 3D image samples in the training sample set can be low-quality 3D image samples, such as low-resolution 3D image samples, or low-definition, low-contrast 3D image samples. Each 3D image sample can be decomposed into 3D component image samples corresponding to the target object. The specific decomposition process can be referred to the above embodiment and will not be repeated here.

[0133] In this context, a 3D image sample and the 3D component image sample corresponding to the target object in the 3D image sample can form a training sample pair to train the adversarial network.

[0134] Step S402: Input multiple 3D component image samples into the generator in the adversarial network, and generate 3D prediction images corresponding to the 3D component image samples through the generator.

[0135] In this embodiment, multiple three-dimensional component image samples of the target object can be input into the generator in the adversarial network, and the generator can generate a three-dimensional predicted image corresponding to the three-dimensional component image samples.

[0136] In practice, the 3D component image samples from multiple training sample pairs can be input into the generator, and the 3D image samples from the training sample pairs can be input into the discriminator. The generator can generate a 3D prediction image corresponding to each 3D component image sample. Then, multiple 3D prediction images can be input into the discriminator, which can identify the quality differences between the corresponding 3D prediction images and 3D image samples, and thus output the corresponding quality parameter differences.

[0137] Step S403: The discriminator in the adversarial network outputs the quality parameter difference between each 3D predicted image and the corresponding 3D image sample.

[0138] In this embodiment, the quality parameter difference may include at least one of contrast difference and texture difference. The contrast difference may be the difference between the gray values ​​of each three-dimensional volume in the three-dimensional prediction image and the gray values ​​of each three-dimensional volume in the three-dimensional image sample. The texture difference may refer to the difference between the texture values ​​(e.g., pixel values ​​or three-dimensional coordinate values) of each three-dimensional volume in the three-dimensional prediction image and the texture values ​​(e.g., pixel values ​​or three-dimensional coordinate values) of each three-dimensional volume in the three-dimensional image sample.

[0139] In some optional examples, the contrast of the 3D predicted image can be determined based on the gray values ​​of each 3D volume in the 3D predicted image, the contrast of the 3D image sample can be determined based on the gray values ​​of each 3D volume in the 3D image sample, and thus the contrast difference between the 3D image samples can be determined.

[0140] In some optional examples, the texture sharpness of the 3D predicted image can be determined based on the texture values ​​of each 3D volume in the 3D predicted image, and the texture sharpness of the 3D image sample can be determined based on the texture values ​​of each 3D volume in the 3D image sample, thereby determining the difference in texture sharpness between 3D image samples.

[0141] The training objective can be that the quality parameter difference characterizes the 3D predicted image as being superior to the 3D image sample, and the quality parameter difference is greater than or equal to the preset difference.

[0142] Step S404: Based on the differences in quality parameters, iteratively update the adversarial network to obtain the image generation model.

[0143] In this embodiment, based on the difference in quality parameters, it can be determined whether the quality of the 3D predicted image generated by the generator in the adversarial network has been enhanced. If it has not been significantly enhanced, the parameters of the adversarial network can be adjusted so that the generator of the adversarial network can evolve in a direction that can enhance the image quality.

[0144] In practice, the loss value of the adversarial network can be determined based on the difference in quality parameters. Then, the parameters of the generator and discriminator in the adversarial network can be updated according to the loss value.

[0145] In some examples, the loss values ​​for the generator and discriminator can be determined separately based on the differences in quality parameters, and then the parameters can be updated based on their respective loss values. In specific implementations, a first loss value corresponding to the generator and a second loss value corresponding to the discriminator can be determined based on the differences in quality parameters; and the parameters of the generator can be updated based on the first loss value, and the parameters of the discriminator can be updated based on the second loss value.

[0146] When determining the first loss value corresponding to the generator based on the difference in quality parameters, the first loss value can be determined according to the following formula (I):

[0147]

[0148]

[0149] In Formula (1), L G Let x represent the first loss value. i and x f Let C(x) represent the 3D predicted image and the 3D image sample, respectively. C(x) represents the output of the discriminator, where C(x) characterizes the image quality corresponding to the 3D predicted image and the 3D image sample output by the discriminator. D(x) i x f This can be represented as a difference in quality parameters.

[0150] When determining the second loss value corresponding to the discriminator based on the difference in quality parameters, the second loss value can be determined according to the following formula (II):

[0151]

[0152] In formula (ii), L D This represents the second loss value.

[0153] In this optional example, the network parameters of the generator and discriminator can be updated separately using their respective loss values, thereby improving the targeting of parameter updates for adversarial networks.

[0154] In some other examples, the overall loss value of the adversarial network can be determined based on the differences in quality parameters, and then the parameters can be updated based on the overall loss value. Furthermore, after updating the parameters of the generator and discriminator based on their respective loss values, an update can then be performed based on the overall loss value. In specific implementations, the overall loss value can be determined based on the first and second loss values, and the updated parameters of the discriminator and generator can be updated based on the overall loss value.

[0155] The average loss value between the first and second loss values ​​can be used as the overall loss value to fine-tune the updated parameters of the adversarial network. This ensures the accuracy of the parameter updates and helps the network converge as quickly as possible.

[0156] The structure of the image generation model is explained. In some examples, if the image generation model is an adversarial network, then the following model structure belongs to the generator structure.

[0157] Reference Figure 5 As shown, a schematic diagram of the image generation model is illustrated, wherein, as Figure 5 As shown, the image generation model may include a convolution module, and a feature enhancement module and an upsampling module connected in sequence after the convolution module.

[0158] Accordingly, when the target 3D component image is input into the image generation model corresponding to the target object to generate the target 3D image based on the target 3D component image, the 3D feature of the target 3D component image can be extracted by the convolution module to obtain the 3D feature map; the 3D feature map can be iteratively convolved and fused at multiple scales by the feature enhancement module to obtain the enhanced 3D feature map; and the enhanced 3D feature map can be upsampled by the upsampling module to obtain the target 3D image.

[0159] In this optional example, the convolution module can be a 3D convolution module, used to perform 3D convolution on the input target 3D component image. The size and parameters of the 3D convolution kernel used have been fixed during the training of the adversarial network. This convolution module can be used to extract 3D features from the target 3D component image, thereby obtaining a 3D feature map.

[0160] Specifically, the feature enhancement module is mainly used to iteratively perform 3D convolutions at multiple scales on 3D feature maps, and then fuse the convolutional feature maps obtained from these multiple scales at different scales. Here, scale can refer to the receptive field, and different scales correspond to different receptive fields. For example, fusing feature maps with large receptive fields and feature maps with small receptive fields preserves image detail while improving global image contrast. For instance,... Figure 5As shown, the convolutional feature map obtained by 3D convolution at each scale needs to be fused with the convolutional feature map obtained by 3D convolution at the corresponding scale. In this way, the details of the convolutional feature maps at different scales can be preserved during the convolution process, thereby avoiding the loss of details and enabling a lightweight network architecture for the generator.

[0161] In one alternative example, the feature enhancement module may include multiple residual branches, each of which can be used to avoid loss of detail during image processing, i.e., to avoid loss of detail in the previously extracted 3D feature map.

[0162] Specifically, the feature enhancement module may include multiple concatenated feature transfer layers and at least one convolutional module connected to each of the multiple feature transfer layers, wherein a three-dimensional convolutional unit is connected between each feature transfer layer and the convolutional module.

[0163] Accordingly, when performing multi-scale 3D convolutions on the 3D feature map through the feature enhancement module and fusing the convolutional feature maps obtained from the multi-scale 3D convolutions to obtain the enhanced 3D feature map, the 3D feature map can be input into the feature transfer layer. After the 3D feature map input to the feature transfer layer is 3D convolved by the feature transfer layer, it is fused with all the 3D feature maps before the feature transfer layer and then input into the next feature transfer layer and the 3D convolution unit. After the 3D convolution unit performs 3D convolution on the input features at the corresponding scale, it is input into the corresponding convolution module, and the convolution module performs 3D convolution on the features output by the 3D convolution unit.

[0164] like Figure 5 As shown, the three-dimensional convolutional unit with the smallest receptive field is connected to the input of the second convolutional module with a larger receptive field, and the three-dimensional convolutional unit with a larger receptive field is connected to the input of the first convolutional module with a smaller receptive field, so that feature fusion can be performed hierarchically according to the receptive field.

[0165] like Figure 5 As shown, the feature transfer layer includes multiple concatenated residual branches. The number of residual branches can be determined according to the number of convolutional units in the convolutional module of the network layer. That is, the output of each convolution can be input to the feature transfer layer through the residual branches. The feature transfer layer can be regarded as a convolutional layer. The convolutional layer can convolve the feature map brought by the current residual branch, fuse it with the feature maps brought by other residual branches, and fuse it with the three-dimensional feature map obtained by the first convolution. After fusion, the feature maps are input to the next feature transfer layer and the three-dimensional convolutional unit respectively.

[0166] The three-dimensional convolutional unit is used to convolve the feature map at the corresponding scale, and the convolutional data is then input into the corresponding first and second convolutional modules.

[0167] in, Figure 5 The residual branches in the feature transfer layer are only for illustrative purposes. In practice, the connection of the residual branches can be referenced from the ResNet network structure.

[0168] In this embodiment, each feature transfer layer performs a 3D convolution operation of the corresponding scale on the input 3D feature map to obtain a convolutional feature map. Each feature transfer layer is also connected to a 3D convolutional unit, which can be further used to perform another 3D convolution on the convolutional feature map. This allows for the preservation of image details while further enhancing the global features of the 3D image, thereby improving the detail and global contrast of the 3D image, contributing to enhanced contrast and clearer texture details.

[0169] The aforementioned adversarial network training scheme offers several advantages. First, it eliminates the need for high-resolution original 3D images as labels, allowing for model training directly with low-quality original 3D images. This reduces the difficulty of sample collection by eliminating the need to acquire high-resolution original 3D images. Second, by using 3D component images that enhance lesion information (target object) alongside the original 3D images for model training, the trained model benefits from components that enhance specific lesion information, thus solving the problem of finding low-quality-high-quality training image pairs.

[0170] In this way, we can obtain image generation models corresponding to various objects. That is, there is an image generation model for different objects. When faced with an original 3D image that needs to be improved, we can call the image generation model corresponding to the target object from multiple image generation models according to the specified target object, and input the target 3D component image corresponding to the target object into the image generation model to obtain a high-quality target 3D image.

[0171] As described above, for each object, a corresponding three-dimensional component image is decomposed from the three-dimensional image. The image region of that object in the three-dimensional component image has higher quality, for example, clearer details and obvious contrast. However, the image quality of the same object will be different in different three-dimensional component images. For example, the image quality of a lung nodule is not high in the three-dimensional component image at frequency 1, but is higher in the three-dimensional component image at frequency 2.

[0172] Therefore, the appropriate frequency range for each object can be determined, where the image quality of the object in the 3D component image within that frequency range is relatively high. For example, the image quality of the object in the 3D component image within that frequency range is higher than a preset quality parameter.

[0173] In another alternative example, when obtaining a target 3D component image that is adapted to a specified target object from multiple 3D component images, the target frequency range corresponding to the target object can be determined from a preset association table, and the target 3D component image whose frequency belongs to the target frequency range can be determined from multiple 3D component images.

[0174] The preset association table includes the frequency range corresponding to each object. Specifically, for each object, the image quality is higher in the three-dimensional component image corresponding to that frequency range, such as higher clarity and contrast.

[0175] In this way, users can input the identifier of the target object, and then determine the corresponding target frequency range from a preset association table based on this identifier. Then, the target three-dimensional component image whose corresponding frequency belongs to the target frequency range can be extracted from multiple decomposed three-dimensional component images.

[0176] In some optional examples, the frequency range adapted to each object can be determined in advance based on the performance of the object in the three-dimensional component images decomposed from different original three-dimensional images (in this example, the original three-dimensional image is referred to as the initial three-dimensional image).

[0177] In specific implementation, refer to Figure 6 As shown, a schematic diagram illustrates the process of determining each object and its corresponding frequency range, such as... Figure 6 As shown, the specific steps may include:

[0178] Step S601: For each object, acquire multiple initial 3D images including that object;

[0179] Step S602: Decompose the multiple initial three-dimensional images respectively to obtain multiple first three-dimensional component images corresponding to each initial three-dimensional image;

[0180] Step S603: Determine the sharpness of the object in each of the multiple first three-dimensional component images;

[0181] Step S604: Determine at least one second three-dimensional component image that meets preset conditions from multiple first three-dimensional component images;

[0182] Step S605: Determine the frequency range corresponding to the object based on the frequencies corresponding to multiple second three-dimensional component images.

[0183] In this optional example, for each object, multiple initial 3D images of that object can be acquired. These multiple initial 3D images can be acquired for different objects, but all different objects include that type of object. For example, if the object is a nodule, an initial 3D image of the lung (because there are lung nodules) can be acquired, or an initial 3D image of the breast (because there are breast nodules) can be acquired, or an initial 3D image of the thyroid gland (because there are thyroid nodules) can be acquired.

[0184] The preset conditions can be high image clarity, comprehensive image detail including the object, or both. The specific conditions can be determined by the user based on the actual situation.

[0185] Of course, to ensure accuracy in frequency determination, multiple initial 3D images of the object from multiple users can be acquired. For example, lung MRI images of users 1-10 can be acquired. The above is merely an illustrative example, and in practice, the acquisition methods are not limited to those described above.

[0186] Each initial 3D image can be decomposed to obtain multiple first 3D component images corresponding to each initial 3D image; specifically, the decomposition can be performed as described in the example above, and will not be repeated here.

[0187] In practice, it can be determined whether the object meets the above-mentioned preset conditions in multiple first three-dimensional component images. One specific implementation method is to send multiple first three-dimensional component images to the front end for display, instructing the user to score and evaluate the multiple first three-dimensional component images, and thus obtain the score value of each first three-dimensional component image based on the user's feedback and evaluation results. This score value can be understood as the quality of the component image, that is, whether the three-dimensional component image meets the preset conditions.

[0188] Next, from multiple first three-dimensional component images, the three-dimensional component image with a score value higher than a preset score value can be determined. This three-dimensional component image is the second three-dimensional component image that meets the preset conditions. Then, the frequency range between the lowest and highest frequencies in the second three-dimensional component image can be used as the frequency range, and this frequency range can be used as the suitable frequency range for this type of object.

[0189] For example, for lung nodules, lung MRI images of 10 users can be acquired. Each lung MRI image is then decomposed into eight first three-dimensional component images, resulting in a total of 80 first three-dimensional component images. These 80 first three-dimensional component images can then be distributed to the 10 users for evaluation. Each user can rate the detail and clarity of eight of the first three-dimensional component images. Next, based on the ratings given by the 10 users to the 80 first three-dimensional component images, images with ratings higher than a preset value can be selected. Assuming there are 6 such third-dimensional component images, the lowest and highest frequencies of these 6 second three-dimensional component images can be used to determine a frequency range. This frequency range represents the frequency of the component images containing clearer images or more detailed image information related to the lung nodule.

[0190] Using the above optional examples, multiple initial 3D images with target objects can be combined to more accurately determine the frequency of the 3D component images corresponding to the target object with higher clarity, so that the target 3D component images decomposed from the original 3D images can more effectively describe the target object.

[0191] The following uses lung MRI images from the medical field as an example to illustrate the overall process of this application:

[0192] S1: Provide Figure 5 The adversarial network shown.

[0193] S2: Prepare multiple three-dimensional lung MRI image samples and decompose each lung MRI image sample into multiple three-dimensional image samples of different frequencies. This can be done using three-dimensional empirical mode decomposition, assuming a volume of V(m, n, b), and mainly includes the following ten steps:

[0194] 1) Let i = 1, R i (m, n, b) = V(m, n, b); where i represents the i-th three-dimensional component image, and m, n, b represent the length, width, and height of the three-dimensional volume, respectively;

[0195] 2) Let j = 1, h i,j-1 (m, n, b) = R i (m, n, b); where j represents the j-th iteration;

[0196] 3) Browse h through a 3×3×3 adjacent window i,j-1 (m, n, b) yields a mapping between local maxima and minima, denoted as M. max (m, n, b) and M min (m, n, b);

[0197] 4) Determine the filter size (i.e., MAX / MIN filter and average filter) to construct the upper / lower extreme value packet and the smoothing average packet;

[0198] 5) Generate the upper envelope E of the maximum value through a MAX filter. max (m,n,b), the lower envelope E of the minimum value is generated by the MIN filter. min (m,n,b), then the average envelope M(m,n,b) is obtained:

[0199]

[0200] 6) Smooth the average envelope M(m,n,b) using an averaging filter. The smoothed envelope is denoted as M. s (m,n,b);

[0201] 7) Update the j-th intermediate value generated when extracting the i-th 3D component image by removing the smoothed envelope. h i，j (m, n, b ),Right now h i，j (m, n, b )= h i，j -1(m,n, b )-Ms(m,n, b );

[0202] 8) To determine whether the intermediate quantity hi,(m,n,b) is a three-dimensional component image, the standard deviation (SD) between two consecutive iterations can be used to verify whether to stop.

[0203]

[0204] Where (x, y, z) are the coordinates of the three-dimensional volume, and m, n, b are the length, width, and height of the three-dimensional volume. SDi j Indicating the identification of the first i The stopping criterion in the j-th iteration of a three-dimensional component image. If SDi j Smaller, then h i,(m,n, b ) is the first i Three-dimensional component images (Ci(m,n, b )= h i,(m,n, b Otherwise, let j = j + 1 and return to step 3);

[0205] 9) Update R i+1 (m, n, b) = R i (m, n, b)-C i(m, n, b);

[0206] 10) Check R i+1 If (m,n,b) contains less than three extreme values, then it is the final residual and the decomposition is completed; otherwise, let i = i + 1 and return to step 2).

[0207] Through the above steps, the original volume can be expressed as:

[0208]

[0209] R i+1 (m,n,b) is the final residue, C i (m,n,b) is the i-th three-dimensional component image.

[0210] S3: From the multiple three-dimensional component image samples of different frequencies decomposed from each lung MRI image sample, filter out the component image samples of lung nodules (i.e., the target three-dimensional component image samples);

[0211] S4: Use component image samples of lung nodules and lung MRI image samples as training samples to train the adversarial network; during training, the discriminator of the adversarial network adopts a convolutional neural network discriminator to distinguish between lung MRI image samples and generated predicted 3D images. The discrimination result is represented by the difference in quality parameters, which can be specifically represented by relative probability.

[0212] The loss function is designed as follows:

[0213]

[0214]

[0215] Where, x i and x f These represent component image samples of lung nodules and lung MRI image samples, respectively. C(x) represents the output of the discriminator for the transformation;

[0216] Among them, L D L represents the discriminator loss. G This represents the generator loss.

[0217] S5: The trained adversarial network is used as the image generation model for lung nodules, so as to generate high-quality lung MRI images.

[0218] S6: When it is necessary to enhance the quality of a patient's lung MRI images, the target three-dimensional component image of the lung nodule can be obtained from the lung MRI image;

[0219] S7: Input the target 3D component image into the image generation model to obtain a high-quality lung MRI image. In this high-quality lung MRI image, the image region of the lung nodule has clearer texture and stronger contrast.

[0220] Based on the same inventive concept, this application also provides an image processing apparatus, referring to... Figure 7 As shown, a schematic diagram of the structure of an image processing apparatus according to this application is illustrated. Figure 7 As shown, it can specifically include the following modules:

[0221] The image decomposition module 701 is used to decompose the original three-dimensional image into multiple three-dimensional component images, with different three-dimensional component images corresponding to different frequencies;

[0222] The image acquisition module 702 is used to acquire a target three-dimensional component image that is adapted to a specified target object from multiple three-dimensional component images;

[0223] Image generation module 703 is used to input the target three-dimensional component image into the image generation model corresponding to the target object, so as to generate a target three-dimensional image based on the target three-dimensional component image; wherein, the image quality of the target three-dimensional image is higher than the image quality of the original three-dimensional image, and the image quality includes at least one of the following: contrast and texture sharpness;

[0224] Image display module 704 is used to display the target three-dimensional image.

[0225] Optionally, the image acquisition module 702 includes:

[0226] A frequency range determination unit is used to determine the target frequency range corresponding to the target object from a preset association table; wherein the preset association table includes the frequency range corresponding to each type of object;

[0227] The image determination unit is used to determine, from multiple three-dimensional component images, a target three-dimensional component image whose frequency belongs to the target frequency range.

[0228] Optionally, determining the frequency range corresponding to each object includes:

[0229] For each object, acquire multiple initial 3D images including that object;

[0230] The initial three-dimensional images are decomposed to obtain multiple first three-dimensional component images corresponding to each initial three-dimensional image;

[0231] The sharpness of the object in multiple first three-dimensional component images is determined respectively;

[0232] From a plurality of first three-dimensional component images, at least one second three-dimensional component image that meets preset conditions is determined;

[0233] Based on the frequencies corresponding to multiple second- and third-dimensional component images, the frequency range corresponding to this type of object is determined.

[0234] Optionally, the image generation model includes: a convolution module, and a feature enhancement module and an upsampling module sequentially connected after the convolution module; the image generation module 703 is specifically used for:

[0235] The convolution module is used to extract three-dimensional features from the target three-dimensional component image to obtain a three-dimensional feature map;

[0236] The feature enhancement module iteratively performs multi-scale 3D convolution and fusion on the 3D feature map to obtain an enhanced 3D feature map.

[0237] The target 3D image is obtained by upsampling the enhanced 3D feature map using the upsampling module.

[0238] Optionally, the feature enhancement module includes multiple concatenated feature transfer layers and at least one convolutional module connected to each of the multiple feature transfer layers, wherein a three-dimensional convolutional unit is connected between each feature transfer layer and the convolutional module; the step of iteratively performing three-dimensional convolutions at multiple scales on the three-dimensional feature map through the feature enhancement module, and fusing the convolutional feature maps obtained from the three-dimensional convolutions at multiple scales to obtain an enhanced three-dimensional feature map includes:

[0239] The three-dimensional feature map is input into the feature transfer layer. After the feature transfer layer performs a three-dimensional convolution on the three-dimensional feature map input into the feature transfer layer, it is fused with all the three-dimensional feature maps before the feature transfer layer and then input into the next feature transfer layer and the three-dimensional convolution unit.

[0240] The input features are convolved at the corresponding scale by the three-dimensional convolution unit and then input into the corresponding convolution module. The convolution module then performs three-dimensional convolution on the features output by the three-dimensional convolution unit.

[0241] Optionally, the image generation module 703 is specifically used to input the target three-dimensional component image into the generator in the image generation model, so as to generate the target three-dimensional image through the generator.

[0242] Optionally, the apparatus further includes a training module for generating an image generation model, comprising:

[0243] The sample acquisition unit is used to acquire a training sample set, which includes multiple three-dimensional image samples and three-dimensional component image samples corresponding to the target object decomposed from each three-dimensional image sample.

[0244] An input unit is used to input multiple three-dimensional component image samples into a generator in an adversarial network, and generate a three-dimensional prediction image corresponding to the three-dimensional component image samples through the generator.

[0245] The difference determination unit is used to output the quality parameter difference between each of the three-dimensional predicted images and the corresponding three-dimensional image samples through the discriminator in the adversarial network;

[0246] The update unit is used to iteratively update the adversarial network based on the differences in the quality parameters to obtain the image generation model.

[0247] Optionally, the update unit includes:

[0248] The loss value determination subunit is used to determine a first loss value corresponding to the generator and a second loss value corresponding to the discriminator based on the difference in the quality parameters.

[0249] The parameter update subunit is used to update the parameters of the generator based on the first loss value, and to update the parameters of the discriminator based on the second loss value.

[0250] Optionally, the device further includes:

[0251] The overall loss determination subunit is used to determine the overall loss value based on the first loss value and the second loss value;

[0252] The parameter adjustment subunit is used to update the parameters of the discriminator and the generator after they have been updated, based on the overall loss value.

[0253] Optionally, the image decomposition module 701 is configured to perform the following steps:

[0254] Step 1: Based on adjacent windows of a preset size, determine the local maxima and local minima of the three-dimensional volume from the original three-dimensional image;

[0255] Step 2: Using the corresponding filter, construct the average envelope of the original 3D image based on the local maxima and local minima;

[0256] Step 3: Based on the average envelope, determine the corresponding three-dimensional component image from the original three-dimensional image, and repeat steps 1-3 above until the decomposed three-dimensional component image meets the first preset condition.

[0257] Obtain multiple 3D component images at the end of the iteration.

[0258] Optionally, the device further includes a verification module, which is used to perform the following steps:

[0259] The current three-dimensional component image is verified to determine whether the current three-dimensional component image meets the second preset condition;

[0260] If the condition is met, then based on the original three-dimensional image after removing the three-dimensional component image of the current time, repeat steps 1-3 above until the decomposed three-dimensional component image meets the first preset condition.

[0261] If the condition is not met, then based on the corresponding original 3D image, repeat steps 1 and 3 until the current 3D component image meets the second preset condition.

[0262] Optionally, the step of verifying the current three-dimensional component image to determine whether the current three-dimensional component image meets the second preset condition includes:

[0263] Obtain the previous three-dimensional component image that satisfies the second preset condition as determined in the previous iteration;

[0264] Based on the previous three-dimensional component image, the current three-dimensional component image, and their respective three-dimensional coordinates, the standard deviation between two consecutive iterations is determined.

[0265] Based on the standard deviation, determine whether the current three-dimensional component image meets the second preset condition.

[0266] It should be noted that the device embodiments are similar to the method embodiments, so the description is relatively simple. For relevant details, please refer to the method embodiments.

[0267] Reference Figure 8 As shown, a structural block diagram of an electronic device 800 according to an embodiment of this application is illustrated. Figure 8 As shown in the figure, an electronic device 800 provided by an embodiment of the present invention can be used to execute an image processing method. It may include a memory 801, a processor 802 and a computer program stored in the memory and executable on the processor. The processor 802 is configured to execute the image processing method.

[0268] like Figure 6As shown, in one embodiment, the electronic device 800 may completely include an input device 803, an output device 804, and an image acquisition device 805. When executing the image processing method of this application embodiment, the image acquisition device 805 can acquire an original three-dimensional image, and then the input device 803 can obtain the original three-dimensional image acquired by the image acquisition device 805. The original three-dimensional image can be processed by the processor 802 to perform image processing based on the original three-dimensional image. The output device 804 can output a target three-dimensional image obtained after quality enhancement of the original three-dimensional image.

[0269] Of course, in one embodiment, memory 801 may include volatile memory and non-volatile memory. Volatile memory can be understood as random access memory used to store and save data. Non-volatile memory refers to computer memory whose stored data will not disappear when the power is turned off. Of course, the computer program of the image processing method of this application may be stored in volatile memory and non-volatile memory, or in either one.

[0270] This application also provides a computer-readable storage medium storing a computer program that causes a processor to perform the image processing method as described in this application.

[0271] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0272] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0273] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0274] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0275] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0276] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0277] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0278] The above provides a detailed description of an image processing method, apparatus, device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image processing method, characterized in that, The method includes: Multiple three-dimensional component images are decomposed from the original three-dimensional image, and different three-dimensional component images correspond to different frequencies; From multiple 3D component images, obtain the target 3D component image that is adapted to the specified target object; The target 3D component image is input into the image generation model corresponding to the target object to generate a target 3D image based on the target 3D component image; wherein, the image quality of the target 3D image is higher than that of the original 3D image, and the image quality includes at least one of the following: contrast and texture sharpness; Display the three-dimensional image of the target; The step of decomposing multiple three-dimensional component images from the original three-dimensional image includes: decomposing multiple three-dimensional component images from the original three-dimensional image through three-dimensional empirical mode decomposition; Among these steps, obtaining a target 3D component image that matches the specified target object from multiple 3D component images includes: From a preset association table, the target frequency range corresponding to the target object is determined; wherein, the preset association table includes the frequency range corresponding to each type of object; From multiple three-dimensional component images, determine the target three-dimensional component image whose frequency belongs to the target frequency range; The determination of the frequency range corresponding to each type of object includes: For each object, acquire multiple initial 3D images including that object; The initial three-dimensional images are decomposed to obtain multiple first three-dimensional component images corresponding to each initial three-dimensional image; The sharpness of the object in multiple first three-dimensional component images is determined respectively; From a plurality of first three-dimensional component images, at least one second three-dimensional component image that meets preset conditions is determined; Based on the frequencies corresponding to multiple second- and third-dimensional component images, the frequency range corresponding to this type of object is determined.

2. The method according to claim 1, characterized in that, The image generation model includes: a convolution module, and a feature enhancement module and an upsampling module sequentially connected after the convolution module; inputting the target 3D component image into the image generation model corresponding to the target object to generate a target 3D image based on the target 3D component image includes: The convolution module is used to extract three-dimensional features from the target three-dimensional component image to obtain a three-dimensional feature map; The feature enhancement module iteratively performs multi-scale 3D convolution and fusion on the 3D feature map to obtain an enhanced 3D feature map. The target 3D image is obtained by upsampling the enhanced 3D feature map using the upsampling module.

3. The method according to claim 2, characterized in that, The feature enhancement module includes multiple concatenated feature transfer layers and at least one convolutional module connected to each of the multiple feature transfer layers, wherein a three-dimensional convolutional unit is connected between each feature transfer layer and the convolutional module; the step of iteratively performing three-dimensional convolutions at multiple scales on the three-dimensional feature map through the feature enhancement module, and fusing the convolutional feature maps obtained from the three-dimensional convolutions at multiple scales to obtain an enhanced three-dimensional feature map includes: The three-dimensional feature map is input into the feature transfer layer. After the feature transfer layer performs a three-dimensional convolution on the three-dimensional feature map input into the feature transfer layer, it is fused with all the three-dimensional feature maps before the feature transfer layer and then input into the next feature transfer layer and the three-dimensional convolution unit. The input features are convolved at the corresponding scale by the three-dimensional convolution unit and then input into the corresponding convolution module. The convolution module then performs three-dimensional convolution on the features output by the three-dimensional convolution unit.

4. The method according to any one of claims 1-3, characterized in that, The image generation model is an adversarial network model. It inputs the target 3D component image into the image generation model corresponding to the target object to generate a target 3D image based on the target 3D component image, including: The target 3D component image is input into the generator in the image generation model to generate the target 3D image.

5. The method according to claim 4, characterized in that, The image generation model is obtained through the following steps: Obtain a training sample set, which includes multiple three-dimensional image samples and three-dimensional component image samples corresponding to the target object decomposed from each three-dimensional image sample; Multiple three-dimensional component image samples are input into a generator in an adversarial network, and the generator generates a three-dimensional predicted image corresponding to the three-dimensional component image samples. The discriminator in the adversarial network outputs the quality parameter difference between each of the three-dimensional predicted images and the corresponding three-dimensional image samples. Based on the differences in the quality parameters, the adversarial network is iteratively updated to obtain the image generation model.

6. The method according to claim 5, characterized in that, Based on the differences in quality parameters, the adversarial network is iteratively updated to obtain the image generation model, including: Based on the difference in the quality parameters, a first loss value corresponding to the generator and a second loss value corresponding to the discriminator are determined. The parameters of the generator are updated based on the first loss value, and the parameters of the discriminator are updated based on the second loss value.

7. The method according to claim 6, characterized in that, The method further includes: Based on the first loss value and the second loss value, determine the overall loss value; Based on the overall loss value, the parameters of the discriminator and the generator are updated.

8. The method according to claim 1, characterized in that, The process of decomposing multiple three-dimensional component images from the original three-dimensional image includes: Step 1: Based on adjacent windows of a preset size, determine the local maxima and local minima of the three-dimensional volume from the original three-dimensional image; Step 2: Using the corresponding filter, construct the average envelope of the original 3D image based on the local maxima and local minima; Step 3: Based on the average envelope, determine the corresponding three-dimensional component image from the original three-dimensional image, and repeat steps 1-3 above until the decomposed three-dimensional component image meets the first preset condition. Obtain multiple 3D component images at the end of the iteration.

9. The method according to claim 8, characterized in that, After determining the corresponding 3D component image from the original 3D image based on the average envelope, the method further includes: The current three-dimensional component image is verified to determine whether the current three-dimensional component image meets the second preset condition; If the condition is met, then based on the original three-dimensional image after removing the three-dimensional component image of the current time, repeat steps 1-3 above until the decomposed three-dimensional component image meets the first preset condition. If the condition is not met, then based on the corresponding original 3D image, repeat steps 1 and 3 until the current 3D component image meets the second preset condition.

10. The method according to claim 9, characterized in that, Verify the current 3D component image to determine whether the current 3D component image meets the second preset condition, including: Obtain the previous three-dimensional component image that satisfies the second preset condition as determined in the previous iteration; Based on the previous three-dimensional component image, the current three-dimensional component image, and their respective three-dimensional coordinates, the standard deviation between two consecutive iterations is determined. Based on the standard deviation, determine whether the current three-dimensional component image meets the second preset condition.

11. An image processing apparatus, characterized in that, The device includes: The image decomposition module is used to decompose the original 3D image into multiple 3D component images, with different 3D component images corresponding to different frequencies; The image acquisition module is used to acquire a target three-dimensional component image that is adapted to the specified target object from multiple three-dimensional component images; An image generation module is used to input the target three-dimensional component image into the image generation model corresponding to the target object, so as to generate a target three-dimensional image based on the target three-dimensional component image; wherein, the image quality of the target three-dimensional image is higher than the image quality of the original three-dimensional image, and the image quality includes at least one of the following: contrast and texture sharpness; An image display module is used to display the three-dimensional image of the target; The step of decomposing multiple three-dimensional component images from the original three-dimensional image includes: decomposing multiple three-dimensional component images from the original three-dimensional image through three-dimensional empirical mode decomposition; The image acquisition module includes: A frequency range determination unit is used to determine the target frequency range corresponding to the target object from a preset association table; wherein the preset association table includes the frequency range corresponding to each type of object; An image determination unit is used to determine, from multiple three-dimensional component images, a target three-dimensional component image whose frequency belongs to the target frequency range; The determination of the frequency range corresponding to each type of object includes: For each object, acquire multiple initial 3D images including that object; The initial three-dimensional images are decomposed to obtain multiple first three-dimensional component images corresponding to each initial three-dimensional image; The sharpness of the object in multiple first three-dimensional component images is determined respectively; From a plurality of first three-dimensional component images, at least one second three-dimensional component image that meets preset conditions is determined; Based on the frequencies corresponding to multiple second- and third-dimensional component images, the frequency range corresponding to this type of object is determined.

12. An electronic device, characterized in that, include: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the image processing method as described in any one of claims 1-10.

13. A computer storage medium, characterized in that, The stored computer program causes the processor to perform the image processing method as described in any one of claims 1-10.

Images