Image enhancement method and device and storage medium

Abstract

The invention discloses an image enhancement method and device and a storage medium, and belongs to the technical field of artificial intelligence, and the method comprises the steps: obtaining an initial training set which comprises a plurality of images; pre-enhancement is carried out on each image to obtain a first enhancement training set, and the first enhancement training set comprises each image and a plurality of first enhancement images corresponding to the image; in the training scene, 2N times of random sampling is carried out on the first enhanced training set to obtain N sample pairs, each sample pair comprises two sample images, and N is a positive integer; performing multi-scale noise enhancement on each sample image of each sample pair to obtain a first sample image; performing normalization processing on the two first sample images of the sample pair to obtain two second sample images; and constructing a second enhanced training set based on each second sample image in each sample pair. According to the method, the performance of the sample set energy optimization model obtained after image enhancement is ensured from the overall perspective.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and particularly to an image enhancement method, apparatus, and storage medium. Background Art

[0002] In the deep learning method for medical image classification, the methods of data augmentation mainly include the following:

[0003] Basic geometric transformations: These methods mainly include:

[0004] Rotation: Rotate the original image by a certain angle, and then maintain the integrity of the image through interpolation methods such as the nearest neighbor method.

[0005] Scaling: Perform bilinear interpolation on the image to enlarge or reduce the rows and columns of the image, but it may cause the degradation of image details.

[0006] Mirroring: Perform horizontal or vertical mirroring on the image to exchange the pixel values on the left and right or up and down in the image.

[0007] Translation: Translate the image with equal step lengths to increase the generalization ability of the model.

[0008] Pixel-level enhancement:

[0009] These methods mainly involve the processing of image pixels, such as adjusting the intensity, color, etc. of the image.

[0010] Synthetic Minority Over-sampling Technique: This is a multi-sample data augmentation method that solves the problem of class imbalance by synthesizing new samples in the minority classes.

[0011] Generative Adversarial Network: Generate new and realistic medical image data based on the adversarial network.

[0012] However, the randomness and blindness of these image enhancement methods are relatively strong, it is difficult to guarantee the quality of the enhanced image, and it is difficult to ensure from an overall perspective that the sample set obtained after image enhancement can effectively optimize the performance of the image processing model trained based on this sample set. Summary of the Invention

[0013] Embodiments of this application provide an image enhancement method, apparatus, and storage medium, which can guarantee the quality of the enhanced image and ensure from an overall perspective that the sample set obtained after image enhancement can effectively optimize the performance of the image processing model trained based on this sample set.

[0014] According to one aspect of the embodiments of this application, an image enhancement method is provided, and the method includes:

[0015] An image enhancement method, characterized in that the method comprises:

[0016] Obtain an initial training set, where the initial training set includes a plurality of images;

[0017] Perform pre-enhancement on each of the images to obtain a first enhanced training set, where the first enhanced training set includes each of the images and its corresponding plurality of first enhanced images;

[0018] Under a training scenario, perform 2N random samplings on the first enhanced training set to obtain N sample pairs, where each sample pair includes two sample images, and N is a positive integer;

[0019] For each sample image of each sample pair, perform multi-scale noise enhancement to obtain a first sample image; perform normalization processing on the two first sample images of the sample pair to obtain two second sample images;

[0020] Construct a second enhanced training set based on each of the second sample images in each sample pair.

[0021] In an exemplary embodiment, the performing multi-scale noise enhancement on each sample image of each sample pair to obtain a first sample image includes:

[0022] After performing Mixup enhancement on the sample image, perform multi-scale noise enhancement on the sample image after Mixup enhancement to obtain a second sample image;

[0023] Perform Patch Swapping enhancement on the second sample image to obtain the first sample image.

[0024] In an exemplary embodiment, the multi-scale noise enhancement is performed by the following method:

[0025] Determine the number of layers of multi-scale noise;

[0026] Determine the first-level noise corresponding to each layer, where the power of the first-level noise is inversely correlated with the square of the sampling frequency of the corresponding layer;

[0027] Perform multi-linear interpolation on each of the first-level noises to obtain a second-level noise;

[0028] Based on the sum of each of the second-level noises, obtain multi-scale noise;

[0029] Add the multi-scale noise to the sample image.

[0030] In an exemplary embodiment, the performing multi-linear interpolation on each of the first-level noises to obtain a second-level noise includes:

[0031] For the first-level noise corresponding to each layer, determine the analytical formula for the variance change ratio after multi-linear interpolation sampling, which is used to compensate for the overall variance loss of the noise generated by multi-linear interpolation;

[0032] The second-level noise is positively correlated with the result obtained by performing multi-linear interpolation on the first-level noise and negatively correlated with the corresponding analytical formula.

[0033] In an exemplary embodiment, the analytical formula corresponding to the first-level noise of each layer satisfies the following formula:

[0034]

[0035] where n is the interpolation multiple of linear interpolation, and k corresponds to the number of noise layers.

[0036] In an exemplary embodiment, the second-level noise is also positively correlated with the random noise intensity, and the random noise intensity is an independent control parameter.

[0037] According to one aspect of the embodiments of the present application, there is provided an image enhancement device, the device includes:

[0038] An initial training set acquisition module, configured to acquire an initial training set, the initial training set including a plurality of images;

[0039] A pre-enhancement module, configured to perform pre-enhancement on each of the images to obtain a first enhanced training set, the first enhanced training set including each of the images and its corresponding plurality of first enhanced images;

[0040] A training-time enhancement module, configured to perform the following operations:

[0041] In a training scenario, perform 2N random samplings on the first enhanced training set to obtain N sample pairs, each sample pair including two sample images, and N is a positive integer;

[0042] For each sample image of each sample pair, perform multi-scale noise enhancement to obtain a first sample image; perform normalization processing on the two first sample images of the sample pair to obtain two second sample images;

[0043] Based on each of the second sample images in each sample pair, construct a second enhanced training set.

[0044] According to one aspect of the embodiments of the present application, a computer device is provided. The computer device includes a processor and a memory. At least one instruction, at least one program, a code set or an instruction set is stored in the memory. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the above-mentioned image enhancement method.

[0045] According to one aspect of the embodiments of the present application, a computer-readable storage medium is provided. At least one instruction, at least one program, a code set or an instruction set is stored in the storage medium. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the above-mentioned image enhancement method.

[0046] According to one aspect of the embodiments of the present application, a computer program product is provided. The computer program product includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes to implement the above-mentioned image enhancement method.

[0047] The technical solution provided by the embodiments of the present application can bring the following beneficial effects:

[0048] The embodiments of the present application propose an image enhancement method, device and storage medium. The core of the embodiments of the present application is to propose a two-stage data enhancement strategy: pre-enhancement and in-training enhancement. In the pre-enhancement stage, the quantity enhancement and preliminary quality enhancement of the images are carried out to obtain the first enhanced training set. In the training stage, by constructing sample pairs and performing multi-scale noise enhancement and normalization on the sample images of the sample pairs, the diversity and generalization of the sample images are greatly enhanced, and the quality of the second enhanced training set obtained after enhancement is controlled as a whole. Through the two-stage data enhancement strategy, the enhancement link is decoupled, which can ensure an increase in the number of images and a larger selection space in the pre-enhancement stage, and ensure the quality of the images after enhancement in the training stage. From an overall perspective, it is ensured that the sample set obtained after image enhancement can effectively optimize the performance of the image processing model trained based on the sample set. After a large number of experiments, it is confirmed that when the image is a nuclear magnetic resonance image, the second enhanced training set obtained by using the image enhancement method provided by the embodiments of the present application can effectively improve the performance of the deep learning classification network with magnetic resonance images as the input. Description of the Drawings

[0049] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0050] Figure 1 is a flowchart of an image enhancement method provided by an embodiment of the present application;

[0051] Figure 2 is a flowchart of a method for obtaining a first sample image provided by an embodiment of the present application;

[0052] Figure 3 is a schematic flowchart of a multi-scale noise enhancement method provided by an embodiment of the present application;

[0053] Figure 4 is a schematic diagram of multi-scale noise provided by an embodiment of the present application;

[0054] Figure 5 is a schematic diagram of multi-scale noise superposition and comparison provided by an embodiment of the present application;

[0055] Figure 6 is a block diagram of an image enhancement device provided by an embodiment of the present application;

[0056] Figure 7 is a block diagram of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0057] To make the objectives, technical solutions, and advantages of the present application clearer, the following briefly introduces the relevant background of the embodiments of the present application:

[0058] To make the objectives, technical solutions, and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings. It should be noted that all kinds of data used in the embodiments of the present application have been fully authorized by the relevant parties before use.

[0059] Please refer to Figure 1 , which shows a flowchart of an image enhancement method of an exemplary embodiment. This method can be applied to a computer device, and the above computer device refers to an electronic device with data calculation and processing capabilities. This method may include the following steps:

[0060] Step S101. Obtain an initial training set, where the initial training set includes multiple images.

[0061] The embodiments of the present application do not limit the number and source of image images. For example, they can come from a large number of nuclear magnetic resonance images or TOF-MRA images. TOF-MRA (Time-of-Flight Magnetic Resonance Angiography) is a non-invasive vascular imaging technique widely used in clinical diagnosis. It utilizes the time-of-flight effect of blood flow to generate three-dimensional images of blood vessels by measuring blood flow velocity and time. The main advantages of TOF-MRA include no need for injection of contrast agents, non-invasive, no radioactive damage, economical and fast, wide indications, etc.

[0062] Step S102. Perform pre-enhancement on each of the image images to obtain a first enhanced training set, where the first enhanced training set includes each of the image images and its corresponding multiple first enhanced images.

[0063] The embodiments of the present application initially expand the number of image images and improve the quality of image images through the pre-training stage, thereby laying a foundation for image enhancement in the training stage. The embodiments of the present application do not limit the specific implementation manner of pre-enhancement.

[0064] Exemplarily, if the resolution of the image image is 512*512*256, after obtaining the ROI (Region of Interest) mask, affine transformation enhancement, elastic deformation enhancement, and Gamma value adjustment enhancement can be performed on the image image. Finally, cropping is performed according to the centroid of the ROI mask after the enhancement transformation to increase the foreground area ratio of the ROI in the data, improve the network training efficiency, and obtain the corresponding first enhanced image. More first enhanced images can be obtained by changing the parameters of affine transformation enhancement, elastic deformation enhancement, and Gamma value adjustment enhancement, as well as the cropping parameters. Of course, after pre-enhancement, each image image corresponds to multiple first enhanced images. The embodiments of the present application do not limit the parameters of affine transformation enhancement, elastic deformation enhancement, and Gamma value adjustment enhancement, as well as the cropping parameters, and can be set according to the actual situation without constituting an implementation obstacle. In a specific implementation manner, after the foregoing operations, each image image will correspond to 7 first enhanced images, obtaining an 8-fold amplified first enhanced training set.

[0065] Step S103. Under the training scenario, perform 2N random samplings on the first enhanced training set to obtain N sample pairs, where each sample pair includes two sample images, and N is a positive integer.

[0066] The embodiments of the present application do not limit the size of N, which is subject to the actual training requirements. The sample image is the image in the first enhanced training set or the first enhanced image. For each sample image of the N sample pairs, enhancement processing can also be performed to further improve the enhancement effect. Exemplarily, enhancement processing such as random displacement cropping and horizontal flipping enhancement can be performed on each sample image respectively. Of course, the embodiments of the present application do not set the parameters used for random displacement cropping and horizontal flipping enhancement, which does not constitute an implementation obstacle and is subject to the actual requirements.

[0067] Step S104. For each sample image of each sample pair, perform multi-scale noise enhancement to obtain a first sample image; perform normalization processing on the two first sample images of the sample pair to obtain two second sample images; based on the second sample images in each sample pair, construct a second enhanced training set.

[0068] By performing multi-scale noise enhancement on each sample pair, the generalization effect is improved. The embodiments of the present application do not limit the normalization method. For example, Z-Score can be used. The second sample images in each sample pair can form a second enhanced training set for training. Each of the second sample images therein can also be referred to as a second enhanced image. The second enhanced images in the second enhanced training set are not used in the inference and testing environments, but only for the training stage to better train the deep learning model.

[0069] Please refer to Figure 2 , which shows the flowchart of the method for obtaining the first sample image in the embodiments of the present application. The step of performing multi-scale noise enhancement on each sample image of each sample pair to obtain a first sample image includes:

[0070] S201. After performing Mixup enhancement on the sample image, perform multi-scale noise enhancement on the sample image after Mixup enhancement to obtain a second sample image;

[0071] S202. Perform Patch Swapping enhancement on the second sample image to obtain the first sample image.

[0072] In an exemplary implementation manner, first performing Mixup enhancement can help the model soften the decision boundary and improve the model generalization. Then, multi-scale noise enhancement and Patch Swapping enhancement are respectively performed to obtain the corresponding first sample image. Finally, Z-Score normalization is performed on the two first sample images of the sample pair to ensure the stability of the data.

[0073] In the embodiments of the present application, when performing deformation enhancement during the multi-scale enhancement of an image, each voxel needs to be resampled through interpolation. Since the number of sampling points does not change during the process, some high-frequency information is lost each time a resampling is performed, resulting in a difference in the noise frequency distribution between the enhanced data and the original data. The embodiments of the present application propose that Gaussian noise enhancement is a reliable method to increase the robustness of the model, and it plays a role in bridging the difference when combined with deformation enhancement. However, due to the imaging principle of nuclear magnetic resonance, the noise in TOF-MRA images mostly presents a large block of speckle structure, and its noise signal frequency is less than the noise frequency generated by traditional Gaussian noise enhancement. To address the above problems, the embodiments of the present application propose an efficient multi-scale noise enhancement during training to better adapt to the impact of speckle noise. Please refer to Figure 3 , which shows a schematic flow diagram of the multi-scale noise enhancement method in the embodiments of the present application. The multi-scale noise enhancement is performed through the following method:

[0074] S301. Determine the number of layers of the multi-scale noise;

[0075] For example, the total number of layers of the multi-scale noise is T, and each layer is expressed by the variable t, that is, t ∈ {0,..., T - 1} represents the serial numbers of each layer. During deformation enhancement, the resampling frequencies of different layers can be determined.

[0076] S302. Determine the first-level noise corresponding to each layer, and the power of the first-level noise is inversely correlated with the square of the sampling frequency of the corresponding layer;

[0077] In the embodiments of the present application, noise classification is performed, and the power (i.e., variance) of each level of noise is set to be inversely correlated with the square of the sampling frequency. In an exemplary embodiment, the first-level noise can be set where D, H, and W respectively refer to the resolution parameters of the input image in the three dimensions of length, width, and depth, I refers to the identity matrix, and N(·, ·) is the Gaussian distribution.

[0078] S303. Perform multi-linear interpolation on each of the first-level noises to obtain the second-level noises; based on the sum of the second-level noises, obtain the multi-scale noise;

[0079] The embodiments of the present application do not limit the interpolation multiple of the multi-linear interpolation. For example, it can be trilinear interpolation, and the following will take trilinear interpolation as an example for illustration.

[0080] In an exemplary embodiment, performing multi-linear interpolation on each of the first-level noises to obtain second-level noises includes: for each corresponding first-level noise of each layer, determining an analytical formula for the variance change ratio after corresponding multi-linear interpolation sampling, and this analytical formula is used to make up for the overall variance loss of the noises generated by multi-linear interpolation; the second-level noises are positively correlated with the result obtained by performing multi-linear interpolation on the first-level noises and negatively correlated with the corresponding analytical formula.

[0081] Since linear interpolation will cause the overall variance of the noises to decrease, for this reason, the embodiments of the present application use an analytical formula for the variance change ratio to amplify the noises after trilinear interpolation by a certain proportion to make up for the lost variance. This analytical formula for the variance change ratio is directly proposed by the embodiments of the present application through research and is directly used to correct the second-level noises of each layer without actually calculating the standard deviation of the interpolation result, reducing the calculation cost. The analytical formula corresponding to each corresponding first-level noise of each layer satisfies the following formula:

[0082]

[0083] where n is the interpolation multiple of linear interpolation, and k corresponds to the number of noise layers. For each layer in the case of trilinear interpolation, the value of k is 2 t , and the value of n is 3.

[0084] In an exemplary embodiment, the formula for a multi-scale noise with a number of layers of T is as follows:

[0085] where TriLerp D×H×W (·) is a trilinear interpolation function for resampling the image to a size of D×H×W, U(·,·) is a continuous uniform distribution, and the second-level noises are also positively correlated with the random noise intensity s t , and the random noise intensity is an independently controlled parameter. In the embodiments of the present application, an independently controlled random intensity parameter s t ~U(0, s) is set to correct the second-level noises of each layer to increase the diversity of the generated noises, thereby enhancing the generalization ability of the model.

[0086] S304. Adding the multi-scale noises to the sample image.

[0087] Generally speaking, the method proposed by the embodiments of the present application adjusts the superimposed multi-layer noises of different sizes and frequencies through the total number of layers T and the noise intensity s to obtain a multi-scale noise with an overall mean of 0 and an expected standard deviation of , and finally additively attach it to the sample image during training, that is, x←x + z, where the vector form of x represents the sample image.

[0088] Please refer to Figure 4, which shows a schematic diagram of multi-scale noise in an embodiment of the present application. Figure 4 The left figure represents the second-level noise of each layer, and the superposition of the second-level noise of each layer can obtain Figure 4 the multi-scale noise effect in the right figure. Please refer to Figure 5 , which shows a schematic diagram of the comparison of multi-scale noise superposition in an embodiment of the present application. Figure 5 The left figure is a slice of the sample image before multi-scale noise is superimposed, Figure 5 and the right side is a slice of the sample image after multi-scale noise is superimposed. It can be seen that the embodiment of the present application can achieve an obvious image enhancement effect through multi-scale noise and trilinear interpolation.

[0089] The following is an embodiment of the device of the present application, which can be used to execute the method embodiment of the present application. For the details not disclosed in the device embodiment of the present application, please refer to the method embodiment of the present application.

[0090] Please refer to Figure 6 , which shows a block diagram of an image enhancement device according to an exemplary embodiment. The device has the function of implementing the above-mentioned image enhancement method, and the above function can be implemented by hardware or by hardware executing corresponding software. The device can be a computer device or can be set in a computer device. The device may include:

[0091] An initial training set acquisition module 601, configured to acquire an initial training set, where the initial training set includes a plurality of image images;

[0092] A pre-enhancement module 602, configured to perform pre-enhancement on each of the image images to obtain a first enhanced training set, where the first enhanced training set includes each of the image images and its corresponding plurality of first enhanced images;

[0093] A training-time enhancement module 603, configured to perform the following operations:

[0094] In a training scenario, perform 2N random samplings on the first enhanced training set to obtain N sample pairs, each sample pair including two sample images, and N is a positive integer;

[0095] For each sample image of each sample pair, perform multi-scale noise enhancement to obtain a first sample image; perform normalization processing on the two first sample images of the sample pair to obtain two second sample images;

[0096] Based on each of the second sample images in each sample pair, construct a second enhanced training set.

[0097] In an exemplary embodiment, the training-time enhancement module 603 is configured to perform the following operations:

[0098] After performing Mixup augmentation on the sample image, perform multi-scale noise augmentation on the sample image after Mixup augmentation to obtain a second sample image;

[0099] Perform Patch Swapping augmentation on the second sample image to obtain the first sample image.

[0100] In an exemplary embodiment, the training-time augmentation module 603 is used to perform the following operations:

[0101] Determine the number of layers of multi-scale noise;

[0102] Determine the first-level noise corresponding to each layer, and the power of the first-level noise is inversely correlated with the square of the sampling frequency of the corresponding layer;

[0103] Perform multi-linear interpolation on each of the first-level noises to obtain second-level noises;

[0104] Based on the sum of the second-level noises, obtain multi-scale noise;

[0105] Add the multi-scale noise to the sample image.

[0106] In an exemplary embodiment, the training-time augmentation module 603 is used to perform the following operations:

[0107] For the first-level noise corresponding to each layer, determine the corresponding variance change ratio analytical formula after multi-linear interpolation sampling, and this analytical formula is used to make up for the overall variance loss of the noise generated by multi-linear interpolation;

[0108] The second-level noise is positively correlated with the result of performing multi-linear interpolation on the first-level noise and inversely correlated with the corresponding analytical formula.

[0109] In an exemplary embodiment, the analytical formula corresponding to the first-level noise of each layer satisfies the following formula:

[0110]

[0111] Among them, n is the interpolation multiple of linear interpolation, and k corresponds to the number of noise layers.

[0112] In an exemplary embodiment, the second-level noise is also positively correlated with the random noise intensity, and the random noise intensity is an independent control parameter.

[0113] It should be noted that, when the device provided in the above embodiments realizes its functions, only the division of the above functional modules is used for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiments and the method embodiments belong to the same concept. For the specific implementation process, please refer to the method embodiments and will not be elaborated here.

[0114] Please refer to Figure 7 , which shows a block diagram of the structure of a computer device in an exemplary embodiment for performing the above image enhancement method. Specifically:

[0115] The computer device 700 includes a central processing unit (CPU) 701, a system memory 704 including a random access memory (RAM) 702 and a read only memory (ROM) 703, and a system bus 705 connecting the system memory 704 and the central processing unit 701. The computer device 700 also includes a basic input / output system (I / O system) 706 for facilitating information transfer between various components within the computer, and a mass storage device 707 for storing an operating system 713, application programs 714, and other program modules 715.

[0116] The basic input / output system 706 includes a display 708 for displaying information and input devices 709 such as a mouse and a keyboard for user input of information. Both the display 708 and the input devices 709 are connected to the central processing unit 701 through an input / output controller 710 connected to the system bus 705. The basic input / output system 706 may also include an input / output controller 710 for receiving and processing inputs from multiple other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 710 also provides output to a display screen, a printer, or other types of output devices.

[0117] The mass storage device 707 is connected to the central processing unit 701 through a mass storage controller (not shown) connected to the system bus 705. The mass storage device 707 and its associated computer-readable medium provide non-volatile storage for the computer device 700. That is to say, the mass storage device 707 may include computer-readable media (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0118] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cartridges, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that computer storage media is not limited to the above several types. The above system memory 704 and mass storage device 707 can be collectively referred to as memory.

[0119] According to various embodiments of the present application, the computer device 700 can also run on a remote computer on the network connected through a network such as the Internet. That is, the computer device 700 can be connected to the network 712 through the network interface unit 711 connected to the system bus 705, or in other words, the network interface unit 711 can also be used to connect to other types of networks or remote computer systems (not shown).

[0120] The above memory further includes a computer program, which is stored in the memory and is configured to be executed by one or more processors to implement the above image enhancement method.

[0121] In an exemplary embodiment, a computer-readable storage medium is also provided. At least one instruction, at least one program segment, a code set, or an instruction set is stored in the above storage medium. When the at least one instruction, the at least one program segment, the code set, or the instruction set is executed by a processor, the above image enhancement method is implemented.

[0122] Specifically, the image enhancement method includes:

[0123] Obtain an initial training set, where the initial training set includes a plurality of images;

[0124] Perform pre-enhancement on each of the images to obtain a first enhanced training set, where the first enhanced training set includes each of the images and its corresponding plurality of first enhanced images;

[0125] In the training scenario, perform 2N random samplings on the first enhanced training set to obtain N sample pairs, where each sample pair includes two sample images, and N is a positive integer;

[0126] For each sample image of each sample pair, perform multi-scale noise enhancement to obtain a first sample image; perform normalization processing on the two first sample images of the sample pair to obtain two second sample images;

[0127] Construct a second enhanced training set based on each second sample image in each sample pair.

[0128] In an exemplary embodiment, the performing multi-scale noise enhancement on each sample image of each sample pair to obtain a first sample image includes:

[0129] After performing Mixup enhancement on the sample image, perform multi-scale noise enhancement on the sample image after Mixup enhancement to obtain a second sample image;

[0130] Perform Patch Swapping enhancement on the second sample image to obtain the first sample image.

[0131] In an exemplary embodiment, the multi-scale noise enhancement is performed by the following method:

[0132] Determine the number of layers of multi-scale noise;

[0133] Determine the first-level noise corresponding to each layer, and the power of the first-level noise is inversely correlated with the square of the sampling frequency of the corresponding layer;

[0134] Perform multi-linear interpolation on each first-level noise to obtain a second-level noise;

[0135] Obtain multi-scale noise based on the sum of each second-level noise;

[0136] Add the multi-scale noise to the sample image.

[0137] In an exemplary embodiment, the performing multi-linear interpolation on each first-level noise to obtain a second-level noise includes:

[0138] For the first-level noise corresponding to each layer, determine the corresponding variance change ratio analytical formula after multi-linear interpolation sampling, and this analytical formula is used to compensate for the overall variance loss of the noise generated by multi-linear interpolation;

[0139] The second-level noise is positively correlated with the result obtained by performing multi-linear interpolation on the first-level noise and inversely correlated with the corresponding analytical formula.

[0140] In an exemplary embodiment, the analytical formula corresponding to the first-level noise of each layer satisfies the following formula:

[0141]

[0142] where n is the interpolation multiple of linear interpolation, and k corresponds to the number of noise layers.

[0143] In an exemplary embodiment, the second-level noise is also positively correlated with the random noise intensity, and the random noise intensity is an independent control parameter.

[0144] Optionally, the computer-readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or optical discs, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0145] In an exemplary embodiment, there is also provided a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above image enhancement method.

[0146] It should be understood that multiple as mentioned herein refers to two or more. And / or, describing the association relationship of associated objects means that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. In addition, the step numbers described in this article only exemplarily show a possible execution sequence between steps. In some other embodiments, the above steps may not be executed in the order of the numbers. For example, two steps with different numbers are executed simultaneously, or two steps with different numbers are executed in the reverse order of the illustration. The embodiments of the present application do not limit this.

[0147] In addition, in the specific implementation of the present application, when it comes to data related to user information, etc., when the above embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards of relevant countries and regions.

[0148] The above are only exemplary embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A method for image enhancement, characterized in that: The method comprises: Acquire an initial training set, wherein the initial training set includes a plurality of image images; Pre-enhance each of the image images to obtain a first enhanced training set, wherein the first enhanced training set includes each of the image images and a plurality of first enhanced images corresponding thereto; In a training scenario, the first enhanced training set is randomly sampled 2N times to obtain N sample pairs, each sample pair includes two sample images, and N is a positive integer; For each sample image of each sample pair, multi-scale noise enhancement is performed to obtain a first sample image; the two first sample images of the sample pair are normalized to obtain two second sample images; A second enhanced training set is constructed based on each second sample image in each sample pair.

2. The method according to claim 1, characterized in that: The step of performing multi-scale noise enhancement on each sample image of each sample pair to obtain a first sample image includes: After performing Mixup enhancement on the sample image, performing multi-scale noise enhancement on the sample image after Mixup enhancement to obtain a second sample image; Patch Swapping enhancement is performed on the second sample image to obtain the first sample image.

3. The method according to claim 1 or 2, characterized in that: The multi-scale noise enhancement is performed by the following method: Determine the number of layers of multi-scale noise; Determine a first-level noise corresponding to each layer, wherein the power of the first-level noise is inversely correlated with the square of the sampling frequency of the corresponding layer; Performing multilinear interpolation on each of the first-level noises to obtain second-level noises; Based on the sum of each second-level noise, multi-scale noise is obtained; The multi-scale noise is added to the sample image.

4. The method according to claim 3, characterized in that The performing multilinear interpolation on each of the first-level noises to obtain the second-level noises comprises: For the first-level noise corresponding to each layer, determine the variance change ratio analytical expression after the corresponding multilinear interpolation sampling, which is used to compensate for the overall variance loss of the noise generated by multilinear interpolation; The second-level noise is positively correlated with a result obtained by performing multilinear interpolation on the first-level noise, and is negatively correlated with the corresponding analytical expression.

5. The method according to claim 4, characterized in that The analytical expression corresponding to the first-level noise of each layer satisfies the following formula: Among them, n is the interpolation multiple of linear interpolation, and k corresponds to the number of noise layers.

6. The method according to claim 5, characterized in that The second level noise is also positively correlated with the random noise intensity, which is an independent control parameter.

7. An image enhancement device, characterized in that: The device comprises: An initial training set acquisition module, used to acquire an initial training set, wherein the initial training set includes a plurality of image images; A pre-enhancement module, used for pre-enhancing each of the image images to obtain a first enhanced training set, wherein the first enhanced training set includes each of the image images and a plurality of first enhanced images corresponding thereto; During training, the module is enhanced to perform the following operations: In a training scenario, the first enhanced training set is randomly sampled 2N times to obtain N sample pairs, each sample pair includes two sample images, and N is a positive integer; For each sample image of each sample pair, multi-scale noise enhancement is performed to obtain a first sample image; the two first sample images of the sample pair are normalized to obtain two second sample images; A second enhanced training set is constructed based on each second sample image in each sample pair.

8. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the image enhancement method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the image enhancement method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product includes computer instructions, a processor of a computer device reads the computer instructions, and the processor of the computer device executes the computer instructions to implement the image enhancement method as described in any one of claims 1 to 7.

Images