Model training method, image processing method and device

Abstract

The invention discloses a model training method and device, and an image processing method and device. Belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a first sample image and a second sample image, wherein the second sample image is an image obtained by performing image degradation processing on a target object in the first sample image; extracting a first image feature from the first sample image, and performing noise addition processing on the first image feature based on the first noise data to obtain a second image feature; extracting a third image feature from the second sample image, and inputting the third image feature and the second image feature into the image processing model to obtain second noise data output by the image processing model; and training an image processing model based on the first noise data and the second noise data.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and specifically to a model training method, an image processing method, and a device thereof. Background Art

[0002] With the development of terminal devices, more and more users use terminal devices to capture images. When users capture images under backlight conditions, problems such as unclear details, grayness, and color distortion of target objects such as people and animals in the image usually occur, which seriously reduces the quality and viewing experience of the image.

[0003] In the prior art, end-to-end image quality enhancement algorithms based on convolutional neural networks are usually used to process target objects in backlit images. However, such algorithms rely on a large number of training samples and have high requirements on the quality of training samples, and usually cannot achieve satisfactory restoration effects for backlit images. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a model training method, an image processing method and a device thereof, which can improve the restoration effect of images taken against the light.

[0005] In a first aspect, an embodiment of the present application provides a model training method, the method comprising: acquiring a first sample image and a second sample image, the second sample image being an image obtained after image degradation processing is performed on a target object in the first sample image; extracting a first image feature from the first sample image, and performing noise processing on the first image feature based on first noise data to obtain a second image feature; extracting a third image feature from the second sample image, and inputting the third image feature and the second image feature into an image processing model to obtain second noise data output by the image processing model; and training the image processing model based on the first noise data and the second noise data.

[0006] In a second aspect, an embodiment of the present application provides an image processing method, the method comprising: acquiring a first image taken against the light; extracting a first feature from the first image and sampling noise data; inputting the first feature and the noise data into an image processing model trained using the model training method in the first aspect to obtain a second feature output by the image processing model; decoding the second feature to obtain a second image after repairing the target object in the first image.

[0007] In a third aspect, an embodiment of the present application provides a model training device, which includes: an acquisition unit, used to acquire a first sample image and a second sample image, wherein the second sample image is an image obtained after image degradation processing is performed on a target object in the first sample image; a noise adding unit, used to extract a first image feature from the first sample image, and perform noise processing on the first image feature based on first noise data to obtain a second image feature; a processing unit, used to extract a third image feature from the second sample image, and input the third image feature and the second image feature into an image processing model to obtain second noise data output by the image processing model; and a training unit, used to train the image processing model based on the first noise data and the second noise data.

[0008] In a fourth aspect, an embodiment of the present application provides an image processing device, comprising: an acquisition unit, for acquiring a first image taken against the light; a first processing unit, for extracting a first feature from the first image and sampling noise data; a second processing unit, for inputting the first feature and the noise data into an image processing model trained by the model training method in the first aspect, to obtain a second feature output by the image processing model; and a generation unit, for decoding the second feature, to obtain a second image after repairing the target object in the first image.

[0009] In a fifth aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0010] In a sixth aspect, an embodiment of the present application provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0011] In a seventh aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0012] In an eighth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0013] In the embodiment of the present application, firstly, a first sample image and a second sample image obtained by performing image degradation processing on the target object in the first sample image are obtained; then, a first image feature is extracted from the first sample image, and the first image feature is subjected to noise processing based on the first noise data to obtain a second image feature; then, a third image feature is extracted from the second sample image, and the third image feature and the second image feature are input into the image processing model to obtain the second noise data output by the image processing model; finally, the image processing model is trained based on the first noise data and the second noise data. In the above process, the second sample image is obtained by performing image degradation processing on the target object in the first sample image, so that a sample pair of an ideal image taken under normal light and an image taken against light can be obtained conveniently and quickly. Compared with the method of obtaining a sample pair by improving the image quality of the image taken against light, the difficulty of obtaining the sample and the quality of the sample can be reduced, thereby improving the training effect of the image processing model, so that it can improve the restoration effect of the image taken against light. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flow chart of the model training method provided in an embodiment of the present application;

[0015] Figure 2 It is a structural schematic diagram of an image processing model in a model training method provided in an embodiment of the present application;

[0016] Figure 3 is a flowchart of an image processing method provided in an embodiment of the present application;

[0017] Figure 4 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;

[0018] Figure 5 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0019] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0020] Figure 7 It is a schematic diagram of the hardware structure of an electronic device suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0022] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0023] The model training method and device provided in the embodiments of the present application are described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0024] Please refer to Figure 1 , which shows a flow chart of the model training method provided in the embodiment of the present application. The model training method provided in the embodiment of the present application can be applied to electronic devices. In practice, the above electronic devices can be electronic devices such as servers.

[0025] The process of the model training method provided in the embodiment of the present application includes the following steps:

[0026] Step 101 : acquiring a first sample image and a second sample image, wherein the second sample image is an image obtained by performing image degradation processing on a target object in the first sample image.

[0027] In this embodiment, the first sample image may be an image taken under normal light. The target object may be the main photographed object in the first sample image. For example, it may be a person, an animal, etc. The second sample image is an image obtained by performing image degradation processing on the target object in the first sample image. The target object may be detected by an artificial intelligence-based target detection algorithm or an image segmentation algorithm, or may be manually selected by a technician, which is not limited here.

[0028] The image degradation processing may include but is not limited to at least one of the following: contrast degradation processing, saturation degradation processing, and brightness degradation processing. Specifically, the contrast degradation processing, saturation degradation processing, and brightness degradation processing may be performed by reducing the contrast, reducing the saturation, and reducing the brightness, respectively. Since the target object in the image shot in the backlit scene usually has low contrast, low saturation, and low brightness, the image with the backlit shooting effect can be obtained by performing image degradation processing such as low contrast, low saturation, and low brightness on the target object in the first sample image.

[0029] The following is an explanation of a specific application scenario. Taking the portrait image processing scenario as an example, in the scenario of repairing a portrait taken against the light, it is necessary to train an image processing model that can repair the human body area in the backlit image. At this time, the first sample image can be an image taken under normal light. The first sample image can be obtained directly from the Internet, or taken under normal light using an electronic device. The second sample image can be an image obtained after contrast degradation processing, saturation degradation processing, brightness degradation processing, etc. are performed on the human body area in the first sample image.

[0030] By performing image degradation processing on the target object in the first sample image to obtain the second sample image, a sample pair of an ideal image taken under normal light and an image taken against light can be obtained quickly and easily. Compared with the method of obtaining a sample pair by improving the image quality of the backlit image, the difficulty of obtaining samples and the quality of samples can be reduced, thereby improving the training effect of the image processing model in the subsequent steps, so that the restoration effect of the backlit image can be improved.

[0031] In some optional implementations, the first sample image and the second sample image may be obtained by the following steps:

[0032] The first step is to obtain a first sample image, which can be denoted as I0.

[0033] In the second step, the first sample image is segmented to obtain a first image corresponding to the target object in the first sample image and a second image corresponding to the background in the first sample image. Specifically, a conventional image segmentation tool can be used to perform image segmentation to obtain an image segmentation result M. The image segmentation result M can be used to mark the target object in the first sample image. The first image can be denoted as I p1 , I p1 =I0×M. The area outside the target object in the first sample image is the background. The second image is I0×(1-M).

[0034] The third step is to perform image degradation processing on the first image to obtain a third image. Specifically, the image degradation processing process can be recorded as f(), and the third image can be recorded as I p2 , then I p2 =f(I p1 ). As an example, f(I p1 )=D+I p1 ×fa, or, f(I p1 )=(I p1 / 255) ga ×255. Wherein, D is the dark area color level. fa and ga are preset parameters, and the specific values ​​can be preset according to needs. D≤255×(1-fa), 0≤ga≤1.

[0035] The fourth step is to fuse the third image with the second image to obtain a second sample image. Specifically, the image fusion can be performed by weighted summing the pixel values ​​of the pixels at the same position to obtain the second sample image. The second sample image can be denoted by I in , then I in =I0×(1-M)+I0×M.

[0036] By image segmentation, the photographing object region in the first image can be easily, quickly and accurately identified, and degradation processing of the photographing object region is performed on this basis, which can improve the generation quality and generation efficiency of the second sample image.

[0037] Step 102: extract a first image feature from the first sample image, and perform noise processing on the first image feature based on first noise data to obtain a second image feature.

[0038] In this embodiment, a feature extraction network may be used to extract features from the first sample image to obtain a first image feature. The feature extraction network may be a neural network with an image feature extraction function. The first image feature and the second image feature may be represented in the form of a feature map. The first noise data may be random Gaussian noise data, which may be represented in the form of a noise map. The feature map and the noise map may have the same size.

[0039] As an example, the first image feature can be recorded as x0, the first noise data can be recorded as ε, and the second image feature can be recorded as x t , then: in, Indicates the noise intensity of the first noise data.

[0040] Step 103: extract the third image feature from the second sample image, input the third image feature and the second image feature into the image processing model, and obtain the second noise data output by the image processing model.

[0041] In this embodiment, the feature extraction network can be used to extract features from the second sample image to obtain a third image feature. The third image feature can also be represented in the form of a feature map. After extracting the third image feature, the third image feature and the second image feature can be input into an image processing model to obtain second noise data output by the image processing model. The image processing model can be established based on a diffusion model, which can predict image noise.

[0042] In some optional implementations, see Figure 2, the image processing model may include a control network 201 and a diffusion network 202. The third image feature may be input into the control network to obtain a fourth image feature output by the control network; then, the fourth image feature and the second image feature may be input into the diffusion network to obtain second noise data output by the diffusion network.

[0043] Among them, the control network can be used to control and guide the diffusion network so that the diffusion network can perform image processing based on the fourth characteristic data output by the control network. The diffusion network is a generative model used in the field of deep learning and artificial intelligence, especially in image generation. The workflow of the diffusion network can be divided into a forward process and a reverse process. The forward process is also called the diffusion process, which is used to gradually add noise to the real data until it is completely converted into noise, so that the data gradually transitions from a structured state to an unstructured state. For example, noise is gradually added to an image until it is completely converted into a noise map that conforms to a Gaussian distribution. The reverse process is also called the denoising process, which is used to start from a pure noise state, gradually remove noise, and restore to structured data, such as an image. This process is achieved by training the diffusion network, which learns how to remove noise at each step and gradually restore the characteristics of the original data.

[0044] It should be noted that Figure 2 It corresponds to the inference process or testing process of the trained image processing model, rather than the training process.

[0045] In some optional implementations, the control network 201 may include an encoder and a decoder, which may be respectively referred to as a first encoder and a first decoder. The first encoder may include multiple convolutional layers, and the first decoder may include multiple first network layers, and the first network layer may include but is not limited to a deconvolutional layer.

[0046] As an example, the control network can adopt a lightweight U-Net structure. U-Net is a convolutional neural network (CNN) for image processing, which consists of an encoder and a decoder, and presents a U-shaped architecture, which enables the network to capture multi-scale feature information during encoding and decoding. The first encoder can be and the first decoder can be respectively, wherein the encoder can include multiple convolution blocks. Each convolution block includes a convolution layer (for example, a 3x3 convolution kernel can be used), a batch normalization (Batch Normalization, BN) layer and an activation function (for example, ReLU (Rectified Linear Unit, linear rectification function) can be used). The convolution block can be used to help the network learn low-level features of the image. The convolution block can be connected to the pooling layer, and the pooling layer can perform maximum pooling or average pooling to reduce the size of the feature map while retaining important information. The attention layer can be located between the convolution block and the pooling layer. The decoder can include multiple deconvolution blocks. Each deconvolution block may contain a deconvolution layer (also called transposed convolution), a batch normalization layer, and an activation function. These deconvolution blocks help learn high-level semantic information of the image. The deconvolution block may be connected to an upsampling layer to increase the size of the feature map and corresponds to a pooling layer in the encoder. Each layer in the decoder is connected to the corresponding layer in the encoder, thereby combining the features of the bottom layer and the high layer. In practice, the first decoder may adopt a Zero-Conv (zero padding convolution) structure, which is a convolution structure in a convolutional neural network. The core idea is to introduce zero padding during the convolution process to control the size of the output feature map. The use of the Zero-Conv section can avoid the destruction of prior knowledge in the diffusion network in the early stages of training.

[0047] On this basis, the fourth image feature may include sub-features output by each of the first network layers in the above-mentioned multiple first network layers. That is, the fourth image feature is a multi-scale image feature. Thus, the control network can extract multi-scale features of the second sample image, such as color features, texture features, shape features, edge features, spatial relationship features, semantic features, etc. By controlling and guiding the diffusion network in this way, the diffusion network can retain and accurately restore the details in the original image during the image processing process, thereby improving the training effect of the overall image processing model, thereby improving the restoration effect of the image taken against the light.

[0048] In some optional implementations, the diffusion network 202 may also include an encoder and a decoder, which may be respectively recorded as a second encoder and a second decoder. The diffusion network may also adopt a U-Net structure. The second decoder may include a plurality of second network layers corresponding to the plurality of first network layers.

[0049] On this basis, the second image feature can be input into the second encoder, and the sub-features output by each of the first network layers in the first network layers can be input into the corresponding second network layer in the second network layers. For each second network layer, the feature output by the second network layer can be added to the sub-features input to the second network layer as the final output feature of the second network layer, and then input into the next second network layer. The feature output by the last second network layer is the second noise data predicted by the diffusion network.

[0050] Therefore, during the training process, the diffusion network can be integrated with the multi-scale features of the second sample image extracted by the control network, so that the details in the original image can be retained and accurately restored during the image processing process, thereby improving the training effect of the overall image processing model and improving the restoration effect of the backlit image.

[0051] In some optional implementations, the diffusion network includes a stable diffusion model (Stable Diffusion Model, SDM). The stable diffusion model is a diffusion network based on a latent variable model, which is mainly used to generate detailed images. The stable diffusion model allows the image generation process to be guided by text or other information without retraining the model through the continuous application of denoising autoencoders. Since the stable diffusion model itself has richer prior knowledge of image reconstruction, by embedding the stable diffusion model in the image processing model, the trained image processing model can have higher quality image restoration capabilities, thereby further improving the restoration effect of images taken against the light.

[0052] In some optional implementations, see Figure 2 , the image processing model further includes a third encoder 203 and a third decoder 204, the third encoder may include an encoder in a vector quantized generative adversarial network (VQGAN), and the third decoder may be a decoder in the VQGAN. The first image feature and the second image feature are extracted by the third encoder. It should be noted that when training the image processing model, the parameters of the third encoder and the third decoder may be fixed to avoid overfitting.

[0053] Among them, the working principle of VQGAN is to map the input image or text into a discretized vector representation, which is then sent to the generative adversarial network for image generation. This model uses vector quantization technology to represent continuous data as discretized vectors, thereby achieving efficient image generation in the generative adversarial network. The encoder in VQGAN achieves efficient data compression by quantizing feature vectors and mapping them into discrete code books. This method not only reduces data storage space, but also speeds up model training and inference. In addition, the encoder in VQGAN extracts the key features of the image through the quantization process, and the decoder uses these features to generate high-quality images. This method can maintain high visual quality when generating images while reducing the demand for computing resources. Therefore, reusing the encoder in VQGAN for image feature extraction can improve the training speed and training effect of the overall image processing model.

[0054] Step 104: training an image processing model based on the first noise data and the second noise data.

[0055] In this embodiment, the parameters of the image processing model can be updated based on the difference between the first noise data and the second noise data to complete a training process. The above steps 101 to 105 can be iteratively performed multiple times until the image processing model training is completed.

[0056] In practice, whether training is completed can be determined in a variety of ways. For example, when the loss value of the image processing model converges, the training can be determined to be completed. For another example, if the number of training times of the image processing model is equal to a preset number of times, the training can be determined to be completed.

[0057] In some optional implementations, the image processing model can be trained by the following steps:

[0058] In the first step, the first noise data and the second noise data are input into the loss function to obtain the loss value of the above image processing model. Among them, the loss function is a non-negative real-valued function that can be used to characterize the difference between the detection result and the true result. The above loss value is the value of the loss function, and the loss function can be set according to actual needs. In general, the smaller the loss value, the better the robustness of the model. As an example, the loss function can be seen in the following formula 1:

[0059] L=E[‖ε-ε(x t ,t)‖ 2 ](1)

[0060] Among them, L is the loss function, E is the mathematical expectation, ε is the first noise data, ε(x t ,t) is the second noise data.

[0061] In the second step, the parameters of the diffusion network are fixed, and the parameters of the control network are updated based on the loss value. Thus, the training of the image processing model can be completed. Each time the steps 101 to 105 are performed, the parameters of the image processing model can be updated until the training is completed.

[0062] By fixing the parameters of the third encoder and the third decoder, overfitting can be avoided during the training of the image processing model, and the prior knowledge of the diffusion network on image reconstruction can be more fully utilized, so that the trained image processing model has high-quality image restoration capabilities, thereby improving the restoration effect of images taken against the light.

[0063] The method provided by the above embodiment of the present application first obtains a first sample image and a second sample image obtained by performing image degradation processing on the target object in the first sample image; then extracts the first image feature from the first sample image, performs noise processing on the first image feature based on the first noise data, and obtains the second image feature; then extracts the third image feature from the second sample image, inputs the third image feature and the second image feature into the image processing model, and obtains the second noise data output by the image processing model; finally, the image processing model is trained based on the first noise data and the second noise data. In the above process, the second sample image is obtained by performing image degradation processing on the target object in the first sample image, so that a sample pair of an ideal image taken under normal light and an image taken against light can be obtained conveniently and quickly. Compared with the method of obtaining a sample pair by improving the image quality of the image taken against light, the difficulty of obtaining the sample and the quality of the sample can be reduced, thereby improving the training effect of the image processing model, so that it can improve the restoration effect of the image taken against light.

[0064] Furthermore, since the diffusion network itself has prior knowledge of image reconstruction, by embedding the diffusion network in the image processing model, the trained image processing model can have high-quality image restoration capabilities, thereby improving the restoration effect of images taken against the light.

[0065] Please refer to Figure 3 , which shows one of the flow charts of the model training method provided in the embodiment of the present application. The model training method provided in the embodiment of the present application can be applied to electronic devices. In practice, the above-mentioned electronic devices can be electronic devices such as smart phones, tablet computers, laptops, and wearable devices.

[0066] The process of the model training method provided in the embodiment of the present application includes the following steps:

[0067] Step 301: Acquire a first image taken against the light.

[0068] Step 302: extract a first feature from a first image and sample noise data.

[0069] In this embodiment, the method of extracting the first feature from the first image, the structure of the image processing model, and the training method can be found in Figure 1 The corresponding embodiments will not be described in detail here.

[0070] In this embodiment, the noise data can be recorded as z t , the noise data z can be obtained by randomly sampling Gaussian noise t .

[0071] Step 303: input the first feature and the noise data into the image processing model to obtain the second feature output by the image processing model.

[0072] In some optional implementations of this embodiment, see Figure 2 , the image processing model may include a control network and a diffusion network. The first feature and the noise data may be first input into the control network to obtain the third feature. Then, the noise data and the third feature may be input into the diffusion network in the image processing model, and the noise data may be denoised based on the third feature by the diffusion network to obtain the second feature.

[0073] See also Figure 2 , the diffusion network performs the reverse process, i.e., the denoising process, which can start from the pure noise state, gradually remove the noise, and restore the structured data. This process can be performed through T time steps, and the noise data z is gradually denoised through T time steps. t Denoising is performed, and the denoised images z are obtained in turn. t-1 、z t-2 、z t-3 ,…,z0. Wherein, T is a positive integer. The above z0 is the second feature.

[0074] The following takes the denoising process corresponding to the first time step as an example. t and the third feature x t After inputting into the diffusion network, the predicted noise ε can be obtained. The denoising result can be calculated, that is, the output image x0 predicted in the current step. The calculation process of x0 can be seen in the following formula 2:

[0075]

[0076] in, Indicates a1 to a t The result of the continuous multiplication. a1 to a t It is a parameter related to noise intensity. Different time steps may correspond to different values, and its specific value can be preset as needed.

[0077] Combined with Bayes' theorem, we can get the following formula 3:

[0078]

[0079] Among them, p(z t-1 |z t ,x0) means that when z is known t ,x0,z t-1 The distribution of .

[0080] N means it follows Gaussian distribution. I is 1.

[0081] For the subsequent time steps, refer to the above process to get z t-2 、z t-3 Wait until the set sampling step number T is reached and the second feature z0 is obtained.

[0082] Step 304: decode the second feature to obtain a second image after repairing the target object in the first image.

[0083] In this embodiment, the second feature can be decoded by a commonly used decoding network. For example, see Figure 2 , decoded by a third decoder, to obtain a second image after the target object in the first image is restored. The third decoder can be found in Figure 1 The contents in the corresponding embodiments will not be repeated here.

[0084] Due to the adoption Figure 1 The image processing model trained by the corresponding embodiment has better image processing effect, can repair the target object in the image taken against the light, and obtain high-quality restoration effect. Therefore, the target object in the second image can be clearer and more realistic than the target object in the first image, thereby achieving improved image quality.

[0085] It should be noted that the image processing method of this embodiment can be used to test the image processing model trained in the above embodiment, and then the image processing model can be continuously optimized according to the test results. The image processing method of this embodiment can also be a practical application method of the image processing model trained in the above embodiment. Using the image processing model trained in the above embodiment, the target object in the image taken against the light can be repaired to generate a clear and high-quality image. Since the above image processing model is embedded with a diffusion network, the prior knowledge of the diffusion network can be embedded in the image processing process, thereby improving the quality of the second image generated by the image processing model.

[0086] It should be noted that the model training method provided in the embodiment of the present application can be executed by a model training device. In the embodiment of the present application, the model training device executing the model training method is taken as an example to illustrate the model training device provided in the embodiment of the present application.

[0087] like Figure 4 As shown, the model training device 400 of this embodiment includes: an acquisition unit 401, used to acquire a first sample image and a second sample image, where the second sample image is an image obtained after image degradation processing is performed on a target object in the first sample image; a noise adding unit 402, used to extract a first image feature from the first sample image, and perform noise processing on the first image feature based on first noise data to obtain a second image feature; a processing unit 403, used to extract a third image feature from the second sample image, and input the third image feature and the second image feature into an image processing model to obtain second noise data output by the image processing model; and a training unit 404, used to train the image processing model based on the first noise data and the second noise data.

[0088] In some optional implementations of this embodiment, the acquisition unit 401 is further used to acquire a first sample image; perform image segmentation on the first sample image to obtain a first image corresponding to the target object in the first sample image and a second image corresponding to the background area in the first sample image; perform image degradation processing on the first image to obtain a third image; and perform image fusion on the third image and the second image to obtain a second sample image. Through image segmentation, the shooting object area in the first image can be easily, quickly and accurately identified, and degradation processing of the shooting object area is performed on this basis, which can improve the generation quality and generation efficiency of the second sample image.

[0089] In some optional implementations of this embodiment, the image processing model includes a control network and a diffusion network; the control network includes a first encoder and a first decoder, the diffusion network includes a second encoder and a second decoder, the first decoder includes a plurality of first network layers; the second decoder includes a plurality of second network layers corresponding to the plurality of first network layers; the processing unit 403 further includes: inputting the third image feature into the control network to obtain a fourth image feature output by the control network, the fourth image feature including a sub-feature output by each first network layer in the plurality of first network layers; inputting the second image feature into the second encoder, inputting the sub-feature output by each first network layer in the plurality of first network layers into a corresponding second network layer in the plurality of second network layers, and obtaining the second noise data output by the diffusion network. Thus, during the training process, the diffusion network can be integrated with the multi-scale features of the second sample image extracted by the control network, so that the details in the original image can be retained and accurately restored during the image processing process, thereby improving the training effect of the overall image processing model, thereby improving the restoration effect of the image taken against the light.

[0090] In some optional implementations of this embodiment, the training unit 404 is further used to: input the first noise data and the second noise data into a loss function to obtain a loss value of the image processing model; fix the parameters of the diffusion network, and update the parameters of the control network based on the loss value. By fixing the parameters of the third encoder and the third decoder, overfitting can be avoided during the training of the image processing model, so that the prior knowledge of the diffusion network for image reconstruction can be more fully utilized, so that the trained image processing model has high-quality image restoration capabilities, thereby improving the restoration effect of images taken against the light.

[0091] In some optional implementations of this embodiment, the diffusion network includes a steady-state diffusion model; the image processing model also includes a third encoder and a third decoder, the third encoder includes an encoder in a vector quantization generative adversarial network, the third decoder includes a decoder in the vector quantization generative adversarial network, and the first image feature and the second image feature are extracted by the third encoder. Since the steady-state diffusion model itself has richer prior knowledge of image reconstruction, by embedding the steady-state diffusion model in the image processing model, the trained image processing model can have a higher quality image restoration capability, thereby further improving the restoration effect of images taken against the light. Reusing the encoder in the vector quantization generative adversarial network for image feature extraction can improve the training speed and training effect of the overall image processing model.

[0092] The device provided by the above embodiment of the present application first obtains a first sample image and a second sample image obtained by performing image degradation processing on the target object in the first sample image; then extracts the first image feature from the first sample image, performs noise processing on the first image feature based on the first noise data, and obtains the second image feature; then extracts the third image feature from the second sample image, inputs the third image feature and the second image feature into the image processing model, and obtains the second noise data output by the image processing model; finally, the image processing model is trained based on the first noise data and the second noise data. In the above process, the second sample image is obtained by performing image degradation processing on the target object in the first sample image, so that a sample pair of an ideal image taken under normal light and an image taken against light can be obtained conveniently and quickly. Compared with the method of obtaining a sample pair by improving the image quality of the image taken against light, the difficulty of obtaining the sample and the quality of the sample can be reduced, thereby improving the training effect of the image processing model, so that it can improve the restoration effect of the image taken against light.

[0093] It should be noted that the image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0094] like Figure 5 As shown, the image processing device 500 of this embodiment includes: an acquisition unit 501, used to acquire a first image taken against the light; a first processing unit 502, used to extract a first feature from the first image and sample noise data; a second processing unit 503, used to input the first feature and the noise data into an image processing model to obtain a second feature output by the image processing model; a generation unit 504, used to decode the second feature to obtain a second image after repairing the target object in the first image.

[0095] It is understandable that the units recorded in the device 500 correspond to the steps in the above-mentioned image processing method. Therefore, the operations, features and beneficial effects described above for the image processing method are also applicable to the device 500 and the units contained therein, and will not be described in detail here.

[0096] The model training device in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplary, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (Augmented Reality, AR) / virtual reality (Virtual Reality, VR) device, a robot, a wearable device, a super mobile personal computer (Ultra-Mobile Personal Computer, UMPC), a netbook or a personal digital assistant (Personal Digital Assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (Personal Computer, PC), a television (Television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0097] The model training device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0098] The model training device provided in the embodiment of the present application can achieve Figure 1 or Figure 3 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0099] Alternatively, if Figure 6 As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601 and a memory 602, wherein the memory 602 stores programs or instructions that can be executed on the processor 601, and when the program or instructions are executed by the processor 601, the various steps of the above-mentioned model training method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0100] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0101] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0102] The electronic device 700 includes but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710.

[0103] Those skilled in the art will appreciate that the electronic device 700 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 710 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0104] Among them, the processor 710 is used to obtain a first sample image and a second sample image, where the second sample image is an image obtained after image degradation processing is performed on the target object in the first sample image; extract a first image feature from the first sample image, and perform noise processing on the first image feature based on first noise data to obtain a second image feature; extract a third image feature from the second sample image, input the third image feature and the second image feature into an image processing model to obtain second noise data output by the image processing model; and train the image processing model based on the first noise data and the second noise data.

[0105] In the above process, the second sample image is obtained by performing image degradation processing on the target object in the first sample image, so that a sample pair of an ideal image taken under normal light and an image taken against light can be easily and quickly constructed. Compared with the method of obtaining a sample pair by improving the image quality of the backlit image, the difficulty of obtaining samples and the quality of samples can be reduced, thereby improving the training effect of the image processing model, so that it can improve the restoration effect of the backlit image.

[0106] In some optional implementations of this embodiment, the processor 710 is further used to obtain a first sample image; perform image segmentation on the first sample image to obtain a first image corresponding to the target object in the first sample image and a second image corresponding to the background area in the first sample image; perform image degradation processing on the first image to obtain a third image; and perform image fusion on the third image and the second image to obtain a second sample image. Through image segmentation, the shooting object area in the first image can be easily, quickly and accurately identified, and degradation processing of the shooting object area is performed on this basis, which can improve the generation quality and generation efficiency of the second sample image.

[0107] In some optional implementations of the present embodiment, the image processing model includes a control network and a diffusion network; the control network includes a first encoder and a first decoder, the diffusion network includes a second encoder and a second decoder, the first decoder includes multiple first network layers; the second decoder includes multiple second network layers corresponding one to one to the multiple first network layers; the processor 710 is also used to input the third image feature into the control network to obtain a fourth image feature output by the control network, the fourth image feature includes a sub-feature output by each first network layer in the multiple first network layers; the second image feature is input into the second encoder, and the sub-feature output by each first network layer in the multiple first network layers is input into the corresponding second network layer in the multiple second network layers to obtain second noise data output by the diffusion network.

[0108] In some optional implementations of this embodiment, the processor 710 is further used to: input the first noise data and the second noise data into a loss function to obtain a loss value of the image processing model; fix the parameters of the diffusion network, and update the parameters of the control network based on the loss value. By fixing the parameters of the third encoder and the third decoder, overfitting can be avoided during the training of the image processing model, so that the prior knowledge of the diffusion network for image reconstruction can be more fully utilized, so that the trained image processing model has high-quality image restoration capabilities, thereby improving the restoration effect of images taken against the light.

[0109] In some optional implementations of this embodiment, the diffusion network includes a steady-state diffusion model; the image processing model also includes a third encoder and a third decoder, the third encoder includes an encoder in a vector quantization generative adversarial network, the third decoder includes a decoder in the vector quantization generative adversarial network, and the first image feature and the second image feature are extracted by the third encoder. Since the steady-state diffusion model itself has richer prior knowledge of image reconstruction, by embedding the steady-state diffusion model in the image processing model, the trained image processing model can have a higher quality image restoration capability, thereby further improving the restoration effect of images taken against the light. Reusing the encoder in the vector quantization generative adversarial network for image feature extraction can improve the training speed and training effect of the overall image processing model.

[0110] In addition, the processor 710 is also used to obtain a first image taken against the light; extract a first feature from the first image and sample noise data; input the first feature and the noise data into an image processing model to obtain a second feature output by the image processing model; and decode the second feature to obtain a second image after repairing the target object in the first image.

[0111] It should be understood that in the embodiment of the present application, the input unit 704 may include a graphics processor (Graphics Processing Unit, GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and at least one of other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0112] The memory 709 can be used to store software programs and various data. The memory 709 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 709 may include a volatile memory or a non-volatile memory, or the memory 709 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 709 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0113] The processor 710 may include one or more processing units; optionally, the processor 710 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 710.

[0114] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned model training method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0115] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0116] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned model training method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0117] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0118] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned model training method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0119] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0121] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a first sample image and a second sample image, where the second sample image is an image obtained after image degradation processing is performed on a target object in the first sample image; Extracting a first image feature from the first sample image, and performing noise processing on the first image feature based on first noise data to obtain a second image feature; Extracting a third image feature from the second sample image, inputting the third image feature and the second image feature into an image processing model, and obtaining second noise data output by the image processing model; The image processing model is trained based on the first noise data and the second noise data.

2. The method according to claim 1, characterized in that The obtaining of the first sample image and the second sample image comprises: Acquire a first sample image; Performing image segmentation on the first sample image to obtain a first image corresponding to the target object in the first sample image and a second image corresponding to the background in the first sample image; performing image degradation processing on the first image to obtain a third image; The third image is fused with the second image to obtain a second sample image.

3. The method according to claim 1, characterized in that The image processing model includes a control network and a diffusion network; the control network includes a first encoder and a first decoder, the diffusion network includes a second encoder and a second decoder, the first decoder includes a plurality of first network layers; the second decoder includes a plurality of second network layers corresponding to the plurality of first network layers one by one; The step of inputting the third image feature and the second image feature into an image processing model to obtain second noise data output by the image processing model includes: Inputting the third image feature into the control network to obtain a fourth image feature output by the control network, wherein the fourth image feature includes sub-features output by each of the first network layers in the plurality of first network layers; The second image feature is input into the second encoder, and the sub-feature output by each first network layer in the plurality of first network layers is input into the corresponding second network layer in the plurality of second network layers to obtain the second noise data output by the diffusion network.

4. The method according to claim 3, characterized in that The step of training the image processing model based on the first noise data and the second noise data comprises: Inputting the first noise data and the second noise data into a loss function to obtain a loss value of the image processing model; The parameters of the diffusion network are fixed, and the parameters of the control network are updated based on the loss value.

5. An image processing method, characterized in that: The method comprises: Acquire a first image shot against the light; Extracting a first feature from the first image and sampling noise data; Inputting the first feature and the noise data into an image processing model trained by the model training method according to any one of claims 1 to 4 to obtain a second feature output by the image processing model; The second feature is decoded to obtain a second image after the target object in the first image is restored.

6. A model training device, characterized in that: The device comprises: an acquisition unit, configured to acquire a first sample image and a second sample image, wherein the second sample image is an image obtained by performing image degradation processing on a target object in the first sample image; a noise adding unit, configured to extract a first image feature from the first sample image, and perform noise adding processing on the first image feature based on first noise data to obtain a second image feature; a processing unit, configured to extract a third image feature from the second sample image, input the third image feature and the second image feature into an image processing model, and obtain second noise data output by the image processing model; A training unit is used to train the image processing model based on the first noise data and the second noise data.

7. The device according to claim 6, characterized in that The acquisition unit is also used to Acquire a first sample image; Performing image segmentation on the first sample image to obtain a first image corresponding to the target object in the first sample image and a second image corresponding to the background area in the first sample image; performing image degradation processing on the first image to obtain a third image; The third image is fused with the second image to obtain a second sample image.

8. The device according to claim 6, characterized in that The image processing model includes a control network and a diffusion network; the control network includes a first encoder and a first decoder, the diffusion network includes a second encoder and a second decoder, the first decoder includes a plurality of first network layers; the second decoder includes a plurality of second network layers corresponding to the plurality of first network layers one by one; the processing unit is further used for: Inputting the third image feature into the control network to obtain a fourth image feature output by the control network, wherein the fourth image feature includes sub-features output by each of the first network layers in the plurality of first network layers; The second image feature is input into the second encoder, and the sub-feature output by each first network layer in the plurality of first network layers is input into the corresponding second network layer in the plurality of second network layers to obtain the second noise data output by the diffusion network.

9. The device according to claim 8, characterized in that The training unit is further used for: Inputting the first noise data and the second noise data into a loss function to obtain a loss value of the image processing model; The parameters of the diffusion network are fixed, and the parameters of the control network are updated based on the loss value.

10. An image processing device, characterized in that: The device comprises: An acquisition unit, used for acquiring a first image taken against the light; A first processing unit, configured to extract a first feature from the first image and sample noise data; A second processing unit, configured to input the first feature and the noise data into an image processing model trained by the model training method according to any one of claims 1 to 4, to obtain a second feature output by the image processing model; A generating unit is used to decode the second feature to obtain a second image after repairing the target object in the first image.

Images