Image processing method and device, electronic equipment and computer readable storage medium
By identifying and processing target objects in the image and expanding the image data set, the poor processing effect caused by a single training data type in image processing is solved, and the better processing effect and deformation processing of different types of target objects is achieved. .
Patent Information
- Application Number
- CN202310636677.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-06
AI Technical Summary
In the image processing process based on artificial intelligence technology, the training data set has a single type of training data, resulting in poor results in some types of image processing, especially the small proportion of stylized target objects processing is not effective.
By acquiring multiple sets of original images and corresponding style images, identifying the target object, and processing the image based on the recognition results, generating multiple sets of original processing images and style processing images of image types, expanding the training data set, including different target objects The size and proportion categories are trained in image stylized network models.
It achieves better processing effects for different types of stylized target objects, especially small-scale target objects, and when large deformation is introduced in image stylized processing, the processing effect is not affected.
Smart Images

Figure CN119091240A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and computer-readable storage medium. Background Art
[0002] With the development of artificial intelligence technology, image processing based on artificial intelligence technology provides users with a wealth of personalized processing methods such as filter effects, sticker effects, deformation effects, stylization effects, etc., thereby achieving a better user experience.
[0003] However, in the process of image processing based on artificial intelligence technology, it is usually necessary to pre-train the network model used for image processing, and the training data set will determine the efficiency of the entire training and the image processing effect of the trained network model. In the training process of network models such as those that perform image stylization processing, since the training data type in the training data set is single, when the model is applied, due to the rich types of images to be processed, the processing results of certain types of images will be poor. Summary of the invention
[0004] The present disclosure is proposed in view of the above problems. The present disclosure provides an image processing method, an apparatus, an electronic device and a computer-readable storage medium.
[0005] According to one aspect of the present disclosure, there is provided an image processing method, comprising: acquiring multiple groups of original images and corresponding style images; identifying target objects in the original images and style images; and processing the original images and style images based on the target object recognition results to obtain multiple groups of original processed images of image types and corresponding style processed images.
[0006] According to another aspect of the present disclosure, there is provided an image processing device, comprising: an image acquisition module, configured to acquire multiple groups of original images and corresponding style images; a target object recognition module, configured to recognize target objects in the original images and style images; and an image processing module, configured to process the original images and style images based on the target object recognition results to obtain multiple groups of original processed images of image types and corresponding style processed images.
[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory for storing at least one processor executable instruction; wherein the at least one processor is used to execute instructions to implement the above image processing method.
[0008] According to another aspect of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can perform the above image processing method.
[0009] As will be described in detail below, according to the image processing method, device, electronic device and computer-readable storage medium of the embodiments of the present disclosure, the types of training data in the image dataset are expanded, such as the size and proportion categories of various target object images, without the need to additionally acquire image data pairs, so that the image stylization network model trained using the expanded image dataset can achieve better processing effects for different types of stylized target objects, especially stylized target objects with a small proportion, and when the image stylization processing introduces larger deformations, the processing effect of the image stylization network model will not be affected.
[0010] It is to be understood that both the foregoing general description and the following detailed description are exemplary, and are intended to provide further explanation of the technology as claimed. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same reference numerals generally represent the same components or steps.
[0012] Figure 1 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure;
[0013] Figure 2 is a flowchart further illustrating an image processing method according to an exemplary embodiment of the present disclosure;
[0014] Figure 3 is a flowchart further illustrating an image processing method according to an exemplary embodiment of the present disclosure;
[0015] Figure 4 is a schematic diagram illustrating an image processing process according to an exemplary embodiment of the present disclosure;
[0016] Figure 5 is a schematic diagram illustrating another image processing process according to an exemplary embodiment of the present disclosure;
[0017] Figure 6 is a flowchart further illustrating an image processing method according to an exemplary embodiment of the present disclosure;
[0018] Figure 7 is a schematic diagram illustrating an image transformation processing network model according to an exemplary embodiment of the present disclosure;
[0019] Figure 8 is a functional module block diagram illustrating an image processing apparatus according to an exemplary embodiment of the present disclosure;
[0020] Fig. 9 is a hardware block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure; and
[0021] Fig.10 is a schematic diagram illustrating a computer-readable storage medium according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0022] In order to make the purpose, technical solution and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.
[0023] First, refer to Figures 1 to 5 A flowchart describing an image processing method according to an exemplary embodiment of the present disclosure is provided. As will be described in detail below, Figures 1 to 3 An image processing method according to an exemplary embodiment of the present disclosure is described, which performs corresponding processing on a training dataset to address the problem of lack of stylized target objects of different proportions in a training dataset, especially the problem of lack of stylized target objects of small proportions. Figure 4 and Figure 5 The image processing process corresponding to the image processing method according to the exemplary embodiment of the present disclosure is specifically illustrated.
[0024] like Figure 1 As shown, the image processing method according to an exemplary embodiment of the present disclosure includes the following steps.
[0025] In step S101, multiple groups of original images and style images corresponding thereto are obtained. Specifically, the image data set includes multiple image data pairs, and the image data pairs are composed of original images and style images corresponding to the original images.
[0026] In step S102, target objects in the original image and the style image are identified.
[0027] Reference Figure 4 and Figure 5 , Figure 4 and Figure 5 It is shown that target objects 501A and 501B in the original image 500A and the style image 500B are identified in step S102 .
[0028] In step S103, based on the target object recognition result, the original image and the style image are processed to obtain multiple groups of original processed images of image types and corresponding style processed images. In the embodiment of the present disclosure, the image types include: no target object type, cropped target object area type, and downsampled target object area type.
[0029] Further, refer to Figure 2 Specific description Figure 1 The process of generating multiple groups of original processed images and the corresponding style processed images in step S103 is shown.
[0030] In step S201 , in response to detecting a target object, a region rectangular frame and key points of the target object are acquired.
[0031] Reference Figure 4 and Figure 5 , showing that in response to detecting the target objects 501A and 501B, the area rectangular frame of the target object is obtained ( Figure 4 and Figure 5 The dashed box in the figure) and key points ( Figure 4 and Figure 5 It is not shown in the figure, but it is easy to understand that the key points are the positioning key points such as the eyes, nose, mouth, etc. of the face).
[0032] In step S202, the original image and the style image are cropped based on the regional rectangular frame to obtain a first original processed image and a first style processed image.
[0033] Reference Figure 4 , the original image 500A and the style image 500B are cropped based on the regional rectangular frame to obtain a first original processed image 502A and a first style processed image 502B. It is easy to understand that the proportion of the processing target object in the image data pair composed of the first original processed image 502A and the first style processed image 502B relative to the entire image is different from the corresponding proportion in the image data pair composed of the original image 500A and the style image 500B. Here, the first original processed image and the first style processed image belong to the cropped target object region type.
[0034] In step S203, the target objects of the original image and the style image are cropped based on the key points to obtain the original target object image, the style target object image, the original non-target object image and the style non-target object image.
[0035] In step S204, a second original processed image and a second style processed image are obtained based on the original target object image, the style target object image, the original non-target object image and the style non-target object image.
[0036] Reference Figure 3 Specific description Figure 2 In step S204 shown, a second original processed image and a second style processed image are obtained based on the original target object image, the style target object image, the original non-target object image and the style non-target object image.
[0037] In step S301, the original target object image and the style target object image are down-sampled to obtain an original target object down-sampled image and a style target object down-sampled image.
[0038] In step S302, the original target object downsampled image and the style target object downsampled image are respectively attached to the original non-target object image and the style non-target object image to obtain a second original processed image and a second style processed image.
[0039] Reference Figure 5 , based on the key points, the target objects 501A and 501B of the original image 500A and the style image 500B are cropped to obtain the original target object image 501A, the style target object image 501B, the original non-target object image 503A, and the style non-target object image 503B. The original target object image 501A and the style target object image 501B are downsampled to obtain the original target object downsampled image 504A and the style target object downsampled image 504B. Further, the original target object downsampled image 504A and the style target object downsampled image 504B are respectively attached to the original non-target image 503A and the style non-target object image 503B to obtain the second original processed image 505A and the second style processed image 505B. The proportion of the processed target object in the image data pair composed of the second original processed image 505A and the second style processed image 505B relative to the whole image is less than the corresponding proportion in the image data pair composed of the original image 500A and the style image 500B. Here, the original non-target object image and the style non-target object image belong to the type of not containing the target object, and the second original processed image and the second style processed image belong to the type of down-sampled target object area.
[0040] It should be understood that although downsampling processing is used as an example here, the scope of the present disclosure is not limited to this, but may include performing any image processing that changes the size of the target object image, such as resampling processing, upsampling processing, interpolation processing, etc.
[0041] In one embodiment of the present disclosure, multiple groups of first original processed images and first style processed images can be obtained based on a preset first ratio; multiple groups of second original processed images and second style processed images can be obtained based on a preset second ratio; and an image stylization model can be trained based on multiple groups of first original processed images and first style processed images, multiple groups of second original processed images and second style processed images. The image data set thus obtained, which includes multiple groups of first original processed images and first style processed images and multiple groups of second original processed images and second style processed images, expands the proportion category of target object images relative to the original image data set. Furthermore, for example, the first ratio can be configured to be smaller than the second ratio, so that the generated image data set has more training data pairs with a small proportion of target object images, thereby resulting in a better processing effect of the trained image stylization network model for stylized target objects with a small proportion.
[0042] Below, we will refer to Figure 6 and Figure 7 Further description using reference to the above Figures 1 to 5 The image data set obtained by the image processing method is used to perform the training process of the image transformation processing network model. The image transformation processing can be, for example, image stylization processing. Figure 6 is a flowchart further illustrating an image processing method according to an exemplary embodiment of the present disclosure. Figure 7 is a schematic diagram illustrating an image transformation processing network model according to an exemplary embodiment of the present disclosure, such as Figure 7 As shown, the image transformation processing network model 700 includes an encoder unit 701, a decoder unit 702 and a discriminator unit 703. The image transformation processing network model 700 may adopt a network architecture of a generative adversarial network, for example.
[0043] like Figure 6 As shown, in step S601, the original image is input into the encoder unit to generate an encoded original content image and transformation modulation parameters. In an exemplary embodiment of the present disclosure, the encoder unit 701 outputs the encoded original content image as a constant matrix Const, for example, whose matrix dimension is 4×4×512. The encoder unit 701 also outputs a high-dimensional latent vector z as a transformation modulation parameter, which corresponds to different image features that undergo image transformation.
[0044] In step S602, the encoded original content image and the transform modulation parameters are input to the decoder unit, and the encoded original content image is modulated by the transform modulation parameters to generate a transformed image of the original image. Specifically, the decoder unit 702 has a plurality of stacked synthetic network layer structures, and in each synthetic network layer structure, the corresponding image feature transformation is controlled by the corresponding transform modulation parameters.
[0045] In step S603, the transformed image and the transformed target image are input to the discriminator unit to generate a discrimination result. In the discriminator unit 703, it generates a discrimination result based on the transformed target image corresponding to the original image input to the encoder unit 701 and the transformed image generated by the decoder unit 702. That is, the discriminator unit 703 discriminates whether the transformed image is sufficiently similar to the transformed target image.
[0046] In step S604, the parameters of the encoder unit, the decoder unit, and the discriminator unit are adjusted based on at least the discrimination result. In an exemplary embodiment of the present disclosure, the parameters of the encoder unit, the decoder unit, and the discriminator unit are adjusted so that the discriminator unit 703 can distinguish between the real sample (transformation target image) and the generated sample (transformation image) as accurately as possible, and at the same time, the transformed image generated by the encoder unit 701 and the decoder unit 702 is as close to the transformed target image as possible, even if the discriminator unit 703 makes a wrong discrimination. That is, the encoder unit 701 and the decoder unit 702 and the discriminator unit 703 respectively improve their respective image transformation generation capabilities and discrimination capabilities in multiple training iterations, and finally obtain the ideal result that the discriminator unit 703 cannot distinguish whether the transformed image generated by the encoder unit 701 and the decoder unit 702 performing the image transformation is a generated sample or a real sample.
[0047] More specifically, the adversarial loss function L2_Loss is determined based on the probability of incorrect discrimination by the discriminator unit 703, and the generated image loss function L1_Loss (e.g., LPIPS Loss) is determined based on the difference between the generated sample (transformed image) and the real sample (transformed target image). The generated image loss function L1_Loss and the adversarial loss function L2_Loss are used to construct the overall training loss function Loss=W1×L1_Loss+W2×L2_Loss, where W1 and W2 are the corresponding weights of the generated image loss function and the adversarial loss function, respectively. During the training process, the parameters of the encoder unit 701, the decoder unit 702, and the discriminator unit 703 are adjusted according to the loss function Loss, so that the loss function Loss finally converges, thereby completing the training.
[0048] By adopting Figure 6 and Figure 7 The image transformation processing network model and its training method shown in FIG. Figures 1 to 5 The image data set is obtained by the image processing method, and the image transformation processing network model finally obtained can achieve ideal image transformation processing effects for transformation processing target objects of different proportions, such as transformation target objects of small proportions, and when the image transformation processing introduces a large deformation, the processing effect of the image transformation processing network model will not be affected.
[0049] Figure 8 800 according to an exemplary embodiment of the present disclosure includes an image acquisition module 801, a target object recognition module 802, and an image processing module 803. It is easy for those skilled in the art to understand that these unit modules can be implemented in various ways by hardware alone, software alone, or a combination thereof, and the present disclosure is not limited to any of them.
[0050] Specifically, the image acquisition module 801 is configured to acquire multiple groups of original images and style images corresponding thereto.
[0051] The target object recognition module 802 is configured to recognize the target object in the original image and the style image.
[0052] The image processing module 803 is configured to process the original image and the style image based on the target object recognition result to obtain multiple groups of original processed images of image types and corresponding style processed images.
[0053] For example, in response to detecting the target object, the image processing module 803 obtains the area rectangular frame and key points of the target object; based on the area rectangular frame, the original image and the style image are cropped to obtain the first original processed image and the first style processed image; based on the key points, the target objects of the original image and the style image are cropped to obtain the original target object image, the style target object image, the original non-target object image, and the style non-target object image; based on the original target object image, the style target object image, the original non-target object image, and the style non-target object image, the second original processed image and the second style processed image are obtained. More specifically, the image processing module 803 downsamples the original target object image and the style target object image to obtain the original target object downsampled image and the style target object downsampled image; the original target object downsampled image and the style target object downsampled image are pasted to the original non-target object image and the style non-target object image, respectively, to obtain the second original processed image and the second style processed image.
[0054] According to the exemplary embodiment of the present disclosure, the image processing device 800 expands the size and proportion categories of the target object image in the image data set without additionally acquiring image data pairs, so that the image stylization network model trained using the expanded image data set can achieve better processing effects for stylized target objects of different proportions, especially stylized target objects of small proportions, and when the image stylization processing introduces larger deformations, the processing effect of the image stylization network model will not be affected.
[0055] Fig. 91 is a hardware block diagram of an electronic device 900 according to an embodiment of the present disclosure. The electronic device according to an embodiment of the present disclosure includes at least a processor; and a memory for storing computer-readable instructions. When the computer-readable instructions are loaded and executed by the processor, the processor performs the image processing method as described above.
[0056] Fig. 9 The electronic device 900 shown specifically includes: a central processing unit (CPU) 901, a graphics processing unit (GPU) 902 and a main memory 903. These units are connected to each other via a bus 904. The central processing unit (CPU) 901 and / or the graphics processing unit (GPU) 902 can be used as the above-mentioned processor, and the main memory 903 can be used as the above-mentioned memory for storing computer-readable instructions. In addition, the electronic device 900 may also include a communication unit 905, a storage unit 906, an output unit 907, an input unit 908 and an external device 909, which are also connected to the bus 904.
[0057] Fig.10 is a schematic diagram illustrating a computer-readable storage medium according to an embodiment of the present disclosure. Fig.10 As shown, a computer-readable storage medium 1000 according to an embodiment of the present disclosure has computer-readable instructions 1001 stored thereon. When the computer-readable instructions 1001 are executed by a processor, the image processing method according to the embodiment of the present disclosure described with reference to the above figures is executed. The computer-readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, an optical disk, a magnetic disk, etc.
[0058] In the above, the image processing method, device, electronic device and computer-readable storage medium according to the embodiments of the present disclosure are described with reference to the accompanying drawings. The present disclosure expands the types of training data in the image dataset without the need to additionally acquire image data pairs, such as expanding the size and proportion categories of the target object images, so that the image stylization network model trained using the expanded image dataset can achieve better processing effects for stylized target objects of different proportions, especially stylized target objects of small proportions, and when the image stylization processing introduces larger deformations, the processing effect of the image stylization network model will not be affected.
[0059] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.
[0060] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limiting. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.
[0061] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.
[0062] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.
[0063] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.
[0064] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.
[0065] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
[0066] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.
Claims
1. An image processing method, characterized in that: The method comprises: Obtain multiple groups of original images and their corresponding style images; Identify the target object in the original image and the style image; Based on the target object recognition result, the original image and the style image are processed to obtain multiple groups of original processed images of image types and corresponding style processed images.
2. The image processing method according to claim 1, characterized in that: The image types include: a type not including the target object, a type cropping the target object region, and a type downsampling the target object region.
3. The image processing method according to claim 1 or 2, characterized in that: The processing of the original image and the style image based on the target object recognition result to obtain multiple groups of original processed images and corresponding style processed images includes: In response to detecting the target object, obtaining a region rectangular frame and key points of the target object; Cropping the original image and the style image based on the regional rectangular frame to obtain a first original processed image and a first style processed image; Cropping the target object of the original image and the style image based on the key points to obtain an original target object image, a style target object image, an original non-target object image, and a style non-target object image; Based on the original target object image, the style target object image, the original non-target object image and the style non-target object image, a second original processed image and a second style processed image are obtained.
4. The image processing method according to claim 3, characterized in that: The obtaining of a second original processed image and a second style processed image based on the original target object image, the style target object image, the original non-target object image and the style non-target object image further comprises: Downsampling the original target object image and the style target object image to obtain an original target object downsampled image and a style target object downsampled image; The original target object downsampled image and the style target object downsampled image are respectively attached to the original non-target object image and the style non-target object image to obtain the second original processed image and the second style processed image.
5. The image processing method according to claim 4, characterized in that: The original non-target object image and the style non-target object image belong to a type that does not contain the target object, the first original processed image and the first style processed image belong to a type that crops the target object area, and the second original processed image and the second style processed image belong to a type that downsamples the target object area.
6. The image processing method according to claim 3, characterized in that: The method further comprises: Based on a preset first ratio, acquiring multiple groups of the first original processed images and the first style processed images; Based on a preset second ratio, acquiring multiple groups of the second original processed images and the second style processed images; An image stylization model is trained based on multiple groups of the first original processed images and the first style processed images and multiple groups of the second original processed images and the second style processed images.
7. The image processing method according to claim 6, characterized in that: The first ratio is smaller than the second ratio.
8. An image processing device, characterized in that: The device comprises: An image acquisition module is configured to acquire multiple groups of original images and style images corresponding thereto; A target object recognition module, configured to recognize target objects in the original image and the style image; The image processing module is configured to process the original image and the style image based on the target object recognition result to obtain multiple groups of original processed images of image types and corresponding style processed images.
9. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as claimed in any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 6.
