Image processing method, image processing device and electronic equipment
By separately inputting the image to be processed and processing conditions into the encoder and decoder, and introducing a conditional classification loss function, the difficulty in adjusting various image processing effects in the prior art is solved, and a better multi-conditional image processing effect is achieved.
Patent Information
- Application Number
- CN202310658201.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-05
- Publication Date
- 2024-12-10
AI Technical Summary
Existing image processing models are difficult to support multiple processing effects at the same time, and the effect is not ideal when adjusting the conditional values of each effect.
By inputting the to-processed images into the encoder unit to generate the encoded image vector, directly input the processing conditions into the decoder unit to generate the condition vector, and generate the processed image based on the image vector and the condition vector, a condition classification loss function is introduced to train the image processing device, and to realize Integration of multiple conditional processing.
It realizes better multi-condition image processing effect, solves the problem that condition information is difficult to learn after encoding, and improves the flexibility and effect of image processing.
Smart Images
Figure CN119107221A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of image processing, and in particular to an image processing method, an image processing device, an electronic device, and a computer-readable storage medium. Background Art
[0002] With the development of artificial intelligence technology, image processing based on artificial intelligence technology provides users with a wealth of personalized processing methods such as filter effects, sticker effects, deformation effects, stylization effects, etc., thereby achieving a better user experience.
[0003] In the image processing process based on artificial intelligence technology, it is hoped that the image processing model can simultaneously support multiple processing effects such as those mentioned above, and can adjust the degree of each effect according to the corresponding conditional value. Existing image processing models are difficult to achieve ideal effects for the conditional processing of such multiple effects. Summary of the invention
[0004] The present disclosure is proposed in view of the above problems. The present disclosure provides an image processing method, an image processing device, an electronic device and a computer-readable storage medium.
[0005] According to one aspect of the present disclosure, there is provided an image processing method for an image processing model including an encoder unit and a decoder unit, the method comprising: inputting an image to be processed into the encoder unit to generate an encoded image vector to be processed; inputting one or more processing conditions into the decoder unit to generate a conditional vector corresponding to the one or more processing conditions; and generating a generated image corresponding to the one or more processing conditions based on the image vector to be processed and the conditional vector.
[0006] According to another aspect of the present disclosure, there is provided an image processing device, comprising: an encoder unit, configured to generate an encoded image vector to be processed based on the image to be processed; and a decoder unit, configured to generate a corresponding conditional vector based on one or more processing conditions, and to generate a generated image corresponding to one or more processing conditions based on the image vector to be processed and the conditional vector.
[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory for storing instructions executable by the at least one processor; wherein the at least one processor is used to execute the instructions to implement the image processing method as described above.
[0008] According to another aspect of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image processing method as described above.
[0009] As will be described in detail below, according to the image processing method, image processing device, electronic device and computer-readable storage medium of the embodiments of the present disclosure, conditional information is directly input into the decoder unit, and the encoder unit mainly encodes the picture information of the image and extracts the original image features, so that the decoder can better learn the conditional information, thereby solving the defect that the conditional information is difficult to be learned by the decoder after being encoded when both the image information and the conditional information are input into the encoder for processing, thereby achieving better multi-conditional image processing effects; in addition, by introducing a conditional classification loss function for the image processing conditions, it is achieved that the trained image processing device can better perform image processing that integrates multiple conditional processing.
[0010] It is to be understood that both the foregoing general description and the following detailed description are exemplary, and are intended to provide further explanation of the technology as claimed. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same reference numerals generally represent the same components or steps.
[0012] Figure 1 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure.
[0013] Figure 2 is a schematic diagram illustrating an image processing apparatus according to an exemplary embodiment of the present disclosure.
[0014] Figure 3 is a schematic diagram illustrating an image processing method according to an exemplary embodiment of the present disclosure.
[0015] Figure 4 is a flowchart illustrating a training method according to an exemplary embodiment of the present disclosure.
[0016] Figure 5 is a flow chart further illustrating a training method according to an exemplary embodiment of the present disclosure.
[0017] Figure 6 is a flow chart further illustrating a training method according to an exemplary embodiment of the present disclosure.
[0018] Figure 7 is a schematic diagram illustrating a training method according to an exemplary embodiment of the present disclosure.
[0019] Figure 8is a hardware block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure.
[0020] Fig. 9 is a schematic diagram illustrating a computer-readable storage medium according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0021] In order to make the purpose, technical solution and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.
[0022] First, refer to Figures 1 to 3 An image processing method according to an exemplary embodiment of the present disclosure is described. As described in detail below, Figure 1 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure. Figure 2 is a schematic diagram illustrating an image processing apparatus according to an exemplary embodiment of the present disclosure. Figure 3 is a schematic diagram illustrating an image processing method according to an exemplary embodiment of the present disclosure.
[0023] like Figure 1 As shown, the image processing method according to an exemplary embodiment of the present disclosure includes the following steps.
[0024] In step S101 , the image to be processed is input into an encoder unit to generate an encoded image vector to be processed.
[0025] Reference Figure 2 , an image processing device 200 according to an exemplary embodiment of the present disclosure is illustrated. The image processing device 200 includes an encoder unit 201 and a decoder unit 202. Those skilled in the art will readily appreciate that these unit modules may be implemented in various ways by hardware alone, software alone, or a combination thereof, and the present disclosure is not limited to any of them.
[0026] Image to be processed I I Input encoder unit Input encoder unit 201 has generated an encoded image vector to be processed. For example, the image vector to be processed can be represented as [b, c, w, h], where b represents the number of images input to encoder unit 201. If 4 images are input at a time, then b = 4; c is the number of channels of the image. Color images are all three-channel RGB, c = 3; w and h are the width and height of the image, respectively.
[0027] In step S102, one or more processing conditions are input into a decoder unit to generate a condition vector corresponding to the one or more processing conditions. More specifically, the decoder unit performs a convolution operation on the one or more processing conditions to obtain a convolution condition, and normalizes the convolution condition to generate a condition vector.
[0028] Reference Figure 2 , one or more treatment conditions C1-C n Input decoder unit 202. In the embodiment of the present disclosure, a processing condition C represents a processing effect on the image to be processed. For example, the condition vector can be expressed as [b,cond_size,1,1], where cond_size represents the number of processing conditions. In the example of performing Van Gogh style transformation, Disney style transformation, and ink style transformation on the image, the number of processing conditions represented by cond_size is 3. Add a convolution layer (conv layer) and a normalization processing layer (sigmoid layer, used to normalize the value to the range of [0-1] for network convergence) in the decoder unit to obtain the condition vector ai_cond after network learning.
[0029] In step S103, a generated image corresponding to one or more processing conditions is generated based on the image vector to be processed and the condition vector. In the decoder unit, the features decoded by the decoder for the image vector to be processed are combined with the condition vector ai_cond learned by the decoder. Different feature dimensions may control different characteristics, some dimensions mainly affect the first effect, and some dimensions mainly affect the second effect. Through the combination of features and condition vectors, the decoder unit is better guided to combine the output of multiple conditions.
[0030] Reference Figure 2 , based on the image vector to be processed and the condition vector, a generated image I corresponding to one or more processing conditions is generated F .
[0031] Further, refer to Figure 3 As shown, for example, using the image processing method and image processing device according to the exemplary embodiment of the present disclosure, for the image to be processed I F Execute processing, Figure 3 3 processing conditions C1-C3 are shown, each of which corresponds to a processing effect and a numerical value of the degree of the effect. In actual image processing, the 3 processing conditions C1-C3 will correspond to processed images I1-I3 respectively. By using the image processing method and image processing device according to the exemplary embodiment of the present disclosure, the processing conditions C1-C3 are input to the decoder unit, so that the image processing device composed of the encoder and the decoder directly outputs the generated image I1 of the comprehensive processing conditions C1-C3. F .
[0032] pass Figures 1 to 3 The image processing method and image processing device described in accordance with the embodiments of the present disclosure directly input conditional information into the decoder unit, while allowing the encoder unit to mainly encode the picture information of the image and extract the original image features, so that the decoder can better learn the conditional information, thereby solving the defect that the conditional information is difficult to be learned by the decoder after being encoded when both the image information and the conditional information are input into the encoder for processing, thereby achieving better multi-conditional image processing effects.
[0033] Below, we will further refer to Figures 4 to 7 The training of the image processing apparatus according to the embodiment of the present disclosure is described. Figure 4 is a flowchart illustrating a training method according to an exemplary embodiment of the present disclosure. Figure 5 is a flow chart further illustrating a training method according to an exemplary embodiment of the present disclosure.
[0034] Figure 6 is a flow chart further illustrating a training method according to an exemplary embodiment of the present disclosure. Figure 7 is a schematic diagram illustrating a training method according to an exemplary embodiment of the present disclosure.
[0035] like Figure 7 As shown, in order to train the image processing apparatus 200 according to the embodiment of the present disclosure composed of the encoder unit 201 and the decoder unit 202, the discriminator unit 300 is further configured. That is, a generative adversarial network (GAN) model consisting of a generator of the encoder unit 201 and the decoder unit 202 and the discriminator unit 300 is formed for training.
[0036] like Figure 4 As shown, the training method according to an exemplary embodiment of the present disclosure includes the following steps.
[0037] In step S401, the original image in the training image pair is input into the encoder unit to generate an encoded original image vector to be processed.
[0038] In the training process using the generative adversarial network model, the image pair used for training includes the original image and the target image corresponding to a conditional processing. For example, for the original image A, n target images {B_0_n-1} can be generated. When two conditions (condition 1 and condition 2) can be processed, for condition 1=0, condition 2=50, B_0 is obtained; for condition 1=30, condition 2=100, B_1 is obtained; and so on, n target images with different conditions are obtained, and n can be 5, for example. In this way, the original image A and the target image {B_0_n-1} form an image pair for training.
[0039] In step S402, one or more processing conditions are input into a decoder unit to generate a condition vector corresponding to the one or more processing conditions.
[0040] In step S403, a generated image corresponding to one or more processing conditions is generated based on the original image vector to be processed and the condition vector.
[0041] It can be understood that in the process of steps S401 to S403, the processing performed by the encoder unit and the decoder unit is the same as that performed by the reference Figure 1 The described steps S101 to S103 are substantially the same.
[0042] In step S404, the target image in the generated image and training image pair is input into the discriminator unit to generate a discrimination result.
[0043] like Figure 7 As shown, the discrimination result generated by the discriminator unit 300 includes two parts, one of which is the discrimination result D, which indicates the generated image I output by the decoder unit. F . Relative to the target image I in the training image pair T , is true or false, if true, then output 1, if false, then output 0. Another part of the discrimination result is the conditional classification result C, which is used to indicate the condition corresponding to the target image obtained by regression. For example, as mentioned above, the two conditions of the target image B_0 are condition 1 = 50 and condition 2 = 100, then the output of the conditional classification result C (cond_pred) is an array = (condition 1_pred, condition 2_pred), and the conditional gt array = (50, 100).
[0044] In step S405, based at least on the discrimination result, the parameters of the encoder unit, the decoder unit and the discriminator unit are adjusted. In an embodiment of the present disclosure, based on the discrimination result, a loss function is constructed for the generator composed of the encoder unit and the decoder unit, and the encoder unit and the decoder unit to perform training.
[0045] Figure 5 The training function construction and training process for the discriminator unit are shown.
[0046] In step S501, a first adversarial loss function is generated based on the true and false discrimination result.
[0047] For example, Figure 7 As shown, the first adversarial loss function L is generated based on the discrimination result D D , used to train the discriminator unit 300, and to optimize the discriminator unit so that it can better distinguish between true and false images.
[0048] In step S502, a first conditional classification loss function is generated based on the conditional classification result, one or more conditions and the classification weights of the discriminator unit.
[0049] For example, Figure 7 As shown, based on the condition classification result C, one or more conditions C1-C n , and the classification weights of the discriminator unit, generating the first conditional classification loss function L C1 .
[0050] The first conditional classification loss function L C1 It is a function of the cross entropy loss between the conditional prediction classification result of the discriminator unit and the training condition and the classification weight of the discriminator unit. The first conditional classification loss function L C1 It is used to optimize the discriminator unit so that the discriminator unit can predict the change conditions of the target image relative to the original image as accurately as possible.
[0051] In step S503, the parameters of the discriminator unit are adjusted based on the first adversarial loss function and the first conditional classification loss function.
[0052] Figure 6 The training function construction and training process for a generator consisting of an encoder unit and a decoder unit are shown.
[0053] In step S601, a second adversarial loss function is generated based on the true and false discrimination result.
[0054] For example, Figure 7 As shown, the second adversarial loss function L is generated based on the discrimination result D G , used to train the encoder unit and the decoder unit, and to optimize the encoder unit and the decoder unit so that the pictures they generate are more and more realistic.
[0055] In step S602, a reconstruction loss function is generated based on the generated image and the target image.
[0056] For example, Figure 7 As shown, based on the generated image I F With the target image I T Generate reconstruction loss function L R It is used to approximate the generated image and the target image at the pixel level or feature level, so as to optimize the encoder unit and the decoder unit so that the generated image is closer and closer to the target image.
[0057] In step S603, a second conditional classification loss function is generated based on the conditional classification result, one or more processing conditions, and the encoder unit and decoder unit classification weights.
[0058] For example, Figure 7As shown, based on the condition classification result C, one or more conditions C1-C n , and the classification weights of the encoder unit and the decoder unit, generating the second conditional classification loss function L C2 .
[0059] The second conditional classification loss function L C2 It is a function of the cross entropy loss between the conditional prediction classification results of the encoder unit and the decoder unit and the training conditions, and the classification loss weight of the encoder unit and the decoder unit. C2 It is used to optimize the encoder unit and the decoder unit so that the conditions of the generated images of the encoder unit and the decoder unit regress to the conditions corresponding to the target image, making the condition-based image processing more accurate.
[0060] In an embodiment of the present disclosure, the discriminator unit classification weight is greater than the encoder unit and decoder unit classification weights.
[0061] In step S604, parameters of the encoder unit and the decoder unit are adjusted based on the second adversarial loss function, the reconstruction loss function and the second conditional classification loss function.
[0062] By reference Figures 4 to 6 The described adversarial generative training, by introducing a conditional classification loss function for image processing conditions, enables the trained image processing device to better perform image processing that integrates multiple conditional processing.
[0063] Figure 7 is a hardware block diagram of an electronic device 700 according to an embodiment of the present disclosure. The electronic device according to an embodiment of the present disclosure includes at least a processor; and a memory for storing computer-readable instructions. When the computer-readable instructions are loaded and executed by the processor, the processor executes the image processing method and the training method as described above.
[0064] Figure 7 The electronic device 700 shown specifically includes: a central processing unit (CPU) 701, a graphics processing unit (GPU) 702 and a main memory 703. These units are connected to each other via a bus 704. The central processing unit (CPU) 701 and / or the graphics processing unit (GPU) 702 can be used as the above-mentioned processor, and the main memory 703 can be used as the above-mentioned memory for storing computer-readable instructions. In addition, the electronic device 700 may also include a communication unit 705, a storage unit 706, an output unit 707, an input unit 708 and an external device 709, which are also connected to the bus 704.
[0065] Figure 8 is a schematic diagram illustrating a computer-readable storage medium according to an embodiment of the present disclosure. Figure 8 As shown, a computer-readable storage medium 800 according to an embodiment of the present disclosure has computer-readable instructions 801 stored thereon. When the computer-readable instructions 801 are executed by a processor, the image processing method and training method according to the embodiment of the present disclosure described with reference to the above figures are executed. The computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, an optical disk, a magnetic disk, etc.
[0066] The image processing method, the training method of the image processing model, the image processing device, the electronic device and the computer-readable storage medium according to the embodiments of the present disclosure are described above with reference to the accompanying drawings. The present disclosure directly inputs the conditional information into the decoder unit, and allows the encoder unit to mainly encode the picture information of the image and extract the original image features, so that the decoder can better learn the conditional information, thereby solving the defect that the conditional information is difficult to be learned by the decoder after being encoded when both the image information and the conditional information are input into the encoder for processing, thereby achieving a better multi-condition image processing effect; in addition, by introducing a conditional classification loss function for the image processing condition, it is achieved that the trained image processing device can better perform image processing that integrates multiple conditional processing.
[0067] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.
[0068] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.
[0069] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.
[0070] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.
[0071] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.
[0072] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.
[0073] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
[0074] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.
Claims
1. An image processing method, characterized in that: The method is used for an image processing model including an encoder unit and a decoder unit, the method comprising: Inputting the image to be processed into the encoder unit to generate an encoded image vector to be processed; inputting one or more processing conditions into the decoder unit to generate a condition vector corresponding to the one or more processing conditions; and Based on the to-be-processed image vector and the condition vector, a generated image corresponding to the one or more processing conditions is generated.
2. The image processing method according to claim 1, characterized in that: The inputting one or more processing conditions into the decoder unit to generate a condition vector corresponding to the one or more processing conditions comprises: performing a convolution operation on the one or more processing conditions to obtain a convolution condition; and The convolution condition is normalized to generate the condition vector.
3. The image processing method according to claim 1, characterized in that: The encoder unit and the decoder unit are obtained based on the following steps: Inputting the original image in the training image pair into the encoder unit to generate an encoded original image vector to be processed; inputting one or more processing conditions into the decoder unit to generate a condition vector corresponding to the one or more processing conditions; generating a generated image corresponding to the one or more processing conditions based on the original image vector to be processed and the condition vector; Inputting the generated image and the target image in the training image pair into a discriminator unit to generate a discrimination result; as well as Based at least on the discrimination result, parameters of the encoder unit, the decoder unit and the discriminator unit are adjusted.
4. The image processing method according to claim 3, characterized in that: The discrimination result includes a true or false discrimination result of the generated image relative to the target image, and a condition classification result corresponding to the one or more processing conditions, The adjusting the parameters of the encoder unit, the decoder unit and the discriminator unit at least based on the discrimination result comprises: Generate a first adversarial loss function based on the true and false discrimination result; generating a first conditional classification loss function based on the conditional classification result, the one or more processing conditions, and the discriminator unit classification weights; and Based on the first adversarial loss function and the first conditional classification loss function, adjusting parameters of the discriminator unit.
5. The image processing method according to claim 4, characterized in that: The adjusting the parameters of the encoder unit, the decoder unit and the discriminator unit at least based on the discrimination result comprises: Generate a second adversarial loss function based on the true and false discrimination result; Generate a reconstruction loss function based on the generated image and the target image; generating a second conditional classification loss function based on the conditional classification result, the one or more processing conditions, and the encoder unit and decoder unit classification weights; and Based on the second adversarial loss function, the reconstruction loss function and the second conditional classification loss function, parameters of the encoder unit and the decoder unit are adjusted.
6. The image processing method according to claim 5, characterized in that: The discriminator unit classification weight is greater than the encoder unit and decoder unit classification weights.
7. An image processing device, characterized in that: The device comprises: An encoder unit configured to generate an encoded image vector to be processed based on the image to be processed; and The decoder unit is configured to generate a corresponding condition vector based on one or more processing conditions, and to generate a generated image corresponding to the one or more processing conditions based on the image vector to be processed and the condition vector.
8. The image processing device according to claim 7, wherein: The decoder unit performs a convolution operation on the one or more processing conditions to obtain a convolution condition, and normalizes the convolution condition to generate the condition vector.
9. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is used to execute the instructions to implement the image processing method according to any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image processing method according to any one of claims 1 to 6.