Image data processing method and device
By obtaining the key point information and feature data of the objects to be processed in the image, and using generator technology, the problems of texture details loss and unnatural skin parts in image processing in the prior art are solved, and high-quality clothing category conversion and skin generation are achieved.
Patent Information
- Application Number
- CN202010218117.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-03-25
- Publication Date
- 2025-03-07
- Estimated Expiration
- 2040-03-25
AI Technical Summary
The prior art cannot effectively process the modified objects in the image, especially when converting different categories of clothing while maintaining the original clothing texture, the texture details of the original clothing will be lost, and the generated skin part will be unnatural.
By obtaining the key point information of the object to be processed in the original image, the image feature data is extracted, and the feature data of the original image is provided to the generator, and the processed image is generated, so that the object to be processed is transformed into the target object type.
The conversion of different categories of clothing while maintaining the original clothing texture is achieved, avoiding the loss of texture details in the image, and the generated skin part is more natural.
Smart Images
Figure CN113449748B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to an image data processing method, device, electronic device and computer storage medium. Background Art
[0002] With the development of image processing technology and Internet technology, image processing is becoming more and more important. At the same time, image processing technology is developing towards more optimized processing algorithms, faster processing speeds, and higher definition of processed images. The realization of intelligent generation, processing, and recognition of images is the future development trend of image processing technology.
[0003] Taking clothing image processing as an application example of image processing technology, the existing clothing image processing methods mainly use the generative adversarial network that has become increasingly popular in recent years to process clothing images. These clothing image processing methods mainly involve the migration of clothing styles. For example, the CycleGan network model can be used to migrate the texture on a certain clothing to the texture of another similar clothing, thereby achieving the effect of changing the texture of other clothing. That is, by using the above-mentioned CycleGan network model to process clothing images, the migration of the texture level style of clothing of the same category can be completed. However, this image processing method does not involve the conversion of different categories of clothing while maintaining the original clothing texture. Although, with the rapid development of image technology, there are also some clothing image processing methods that can realize the conversion of different categories of clothing while maintaining the original clothing texture. For example, the IntaGan network model can be used to convert clothing categories. However, in the generated clothing image, the texture details of the original clothing will be lost. At the same time, in the process of clothing category conversion, if the acquired clothing image needs to have a new skin part of the model generated, the image of the relevant skin part generated in the image is also very unnatural. As a result, the existing image processing technology cannot effectively process the modified object in the image. Summary of the invention
[0004] The present application provides an image data processing method to solve the problem that the image processing technology in the prior art cannot effectively process the modified object in the image. The present application also provides an image data processing device, an electronic device and a computer storage medium.
[0005] The present application provides an image data processing method, comprising:
[0006] Acquire an original image and a target object type; the original image contains at least one object to be processed;
[0007] Acquire key point information related to the object to be processed in the original image according to the original image;
[0008] The image feature data of the object to be processed is obtained according to the key point information, and the image feature data of the object to be processed and the feature data of the original image are provided to a generator to generate a processed image; in the processed image, the object to be processed is transformed into a processed object of the target object type.
[0009] Optionally, the acquiring key point information related to the object to be processed in the original image includes:
[0010] The original image is used as input data of a network model for obtaining key point information, and the output result of the network model for obtaining key point information is used as the key point information related to the object to be processed in the original image; wherein the network model for obtaining key point information is a network model that can obtain the key point information related to the object to be processed in the original image based on the original image.
[0011] Optionally, acquiring image feature data of the object to be processed according to the key point information includes:
[0012] According to the key point information, obtaining key point position information of the key point in the original image;
[0013] Image feature data at key point positions in the original image are extracted, and the extracted image feature data at key point positions in the original image are used as image feature data of the object to be processed.
[0014] Optionally, acquiring image feature data of the object to be processed according to the key point information includes:
[0015] According to the key point information, obtaining the position information of the object to be processed in the original image;
[0016] According to the position information of the object to be processed in the original image, the mask information of the object to be processed is acquired, and the mask information of the object to be processed is used as the image feature data of the object to be processed.
[0017] Optionally, before providing the image feature data of the object to be processed and the feature data of the original image to the generator to generate the processed image, it also includes: merging and encoding the mask information of the object to be processed and the feature data of the original image to obtain the encoding result.
[0018] Optionally, providing the image feature data of the object to be processed and the feature data of the original image to a generator to generate a processed image includes:
[0019] The encoding result, the image feature data of the object to be processed and the feature data of the original image are provided to a generator to generate a processed image.
[0020] Optionally, before providing the image feature data of the object to be processed and the feature data of the original image to the generator, the method further includes acquiring the feature data of the original image based on the original image.
[0021] Optionally, acquiring feature data of the original image according to the original image includes:
[0022] The original image is input into a first feature extractor, first feature data extracted by the first feature extractor for the original image is obtained, and the first feature data is used as feature data of the original image.
[0023] Optionally, the target object type includes at least one of the following types:
[0024] A type of category transformation for the object to be processed;
[0025] The image feature transformation type for the object to be processed.
[0026] Optionally, the method further includes: performing adversarial training on the generator and the discriminator based on the original image samples and the target image samples.
[0027] Optionally, the adversarial training of the generator and the discriminator based on the original image samples and the target image samples includes:
[0028] Acquire an original image sample and a target image sample; the target image sample refers to an image sample after the object to be processed of the original image sample is transformed according to the target object type;
[0029] Providing the image feature data of the original image sample and the image feature data of the object to be processed in the original image sample to an initial generator to generate a first processed image;
[0030] An initial discriminator is used to determine whether the difference between the first processed image and the target image sample makes the loss value of the loss function meet the preset target training condition; wherein the loss function is a function constructed based on the Cycle loss function and the L2 loss function of the CycleGAN network model;
[0031] If the loss value does not meet the preset target training condition, adjusting the model parameters of the initial generator and the initial discriminator;
[0032] Providing the image feature data of the original image sample, the image feature data of the object to be processed in the original image sample, and the image feature data of the first processed image to the generator after adjusting the model parameters, to generate a second processed image;
[0033] The discriminator after adjusting the model parameters determines whether the difference between the second processed image and the target image sample makes the loss value of the loss function meet the preset target loss condition; if the loss value meets the preset target training condition, stop adjusting the model parameters of the generator and the discriminator, obtain the trained generator, and use the trained generator as the generator;
[0034] Otherwise, continue to adjust the model parameters of the generator and the discriminator, and generate a third processed image, and continue to determine whether the difference between the third processed image and the target image sample makes the loss value of the loss function meet the preset target training conditions; and so on, until a processed image is obtained in which the difference makes the loss value of the loss function meet the preset target training conditions.
[0035] Optionally, the determining whether the difference between the first processed image and the target image sample makes the loss value of the loss function meet a preset target training condition includes:
[0036] Acquire first key point information related to the object to be processed in the first processed image according to the first processed image; acquire image feature data of the object to be processed in the first processed image according to the first key point information;
[0037] Acquire second key point information related to the object to be processed in the target image sample according to the target image sample; acquire image feature data of the object to be processed in the target image sample according to the second key point information;
[0038] Based on the image feature data of the object to be processed in the first processed image and the image feature data of the object to be processed in the target image sample, it is determined whether the difference between the first processed image and the target image sample makes the loss value of the loss function meet the preset target loss condition.
[0039] Optionally, the method further includes: encoding first key point information related to the object to be processed in the first processed image by matrix multiplication to obtain first key point information encoding data;
[0040] The step of acquiring image feature data of the object to be processed in the first processed image according to the first key point information includes:
[0041] Image feature data of the object to be processed in the first processed image is obtained according to the first key point information encoding data and the feature data of the first processed image.
[0042] Optionally, before using the initial discriminator to determine whether the difference between the first processed image and the target image sample makes the loss value of the loss function meet a preset target loss condition, the method further includes:
[0043] The first processed image is input into a second feature extractor, second feature data extracted by the second feature extractor for the first processed image is obtained, and the second feature data is used as feature data of the first processed image.
[0044] Correspondingly, the present application provides an image data processing device, comprising:
[0045] A first acquisition unit is used to acquire an original image and a target object type; the original image contains at least one object to be processed;
[0046] A key point information acquisition unit, used to acquire key point information related to the object to be processed in the original image according to the original image;
[0047] An image feature data acquisition unit, used to acquire image feature data of the object to be processed according to the key point information;
[0048] The image generating unit is used to provide the image feature data of the object to be processed and the feature data of the original image to a generator to generate a processed image; in the processed image, the object to be processed is transformed into a processed object of the target object type.
[0049] Correspondingly, the present application also provides an electronic device, including:
[0050] processor;
[0051] The memory is used to store a computer program, the computer program is executed by the processor to perform an image data processing method, the method comprising the following steps:
[0052] Acquire an original image and a target object type; the original image contains at least one object to be processed;
[0053] Acquire key point information related to the object to be processed in the original image according to the original image;
[0054] The image feature data of the object to be processed is obtained according to the key point information, and the image feature data of the object to be processed and the feature data of the original image are provided to a generator to generate a processed image; in the processed image, the object to be processed is transformed into a processed object of the target object type.
[0055] Correspondingly, the present application further provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is executed by a processor to perform an image data processing method, wherein the method comprises the following steps:
[0056] Acquire an original image and a target object type; the original image contains at least one object to be processed;
[0057] Acquire key point information related to the object to be processed in the original image according to the original image;
[0058] The image feature data of the object to be processed is obtained according to the key point information, and the image feature data of the object to be processed and the feature data of the original image are provided to a generator to generate a processed image; in the processed image, the object to be processed is transformed into a processed object of the target object type.
[0059] Compared with the prior art, this application has the following advantages:
[0060] The present application provides an image data processing method, including: obtaining an original image and a target object type; the original image contains at least one object to be processed; obtaining key point information related to the object to be processed in the original image according to the original image; obtaining image feature data of the object to be processed according to the key point information, providing the image feature data of the object to be processed and the feature data of the original image to a generator to generate a processed image; in the processed image, the object to be processed is transformed into a processed object of the target object type. The image data processing method of the present application, based on the key point information related to the object to be processed in the original image, obtains the image feature data of the object to be processed in the original image, so that the image feature data of the object to be processed, the feature data of the original image and the processed image generated by the generator can meet the processing requirements of the target object type, thereby achieving the effect of effectively processing the object to be processed in the image. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0062] Figure 1-A Schematic diagram of application scenario of the image data processing method of this application.
[0063] Figure 1-B Schematic diagram of application scenario of the image data processing method of this application.
[0064] Figure 1 This is a flowchart of the image data processing method according to the first embodiment of the present application.
[0065] Figure 2 FIG. 1 is a schematic diagram of an image data processing device according to a second embodiment of the present application.
[0066] Figure 3 This is a schematic diagram of an image data processing electronic device according to the third embodiment of the present application. DETAILED DESCRIPTION
[0067] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.
[0068] In order to present the present application more clearly, the application scenarios of the image data processing method provided by the present application are first introduced.
[0069] Some embodiments provided in this application can be applied to scenarios where a client interacts with a server, such as Figure 1-A As shown, it is a schematic diagram of an application scenario embodiment provided by the present application. The client first establishes a connection with the server. After the connection is established, the client sends the original image and the preset target object type to the server. After the server receives the original image and the preset target object type, the server inputs the original image into the network model for obtaining key point information to obtain key point information related to the object to be processed in the original image, wherein the network model for obtaining key point information can obtain key point information related to the object to be processed in the original image based on the original image.
[0070] Afterwards, the image feature data of the object to be processed is obtained according to the key point information related to the object to be processed in the original image. Specifically, first, the mask information of the object to be processed can be obtained by using the key point information related to the object to be processed in the original image. Then, the mask information of the object to be processed is input into the first feature extractor together with the original image, so as to obtain the image feature data of the object to be processed. Since the original image is input into the first feature extractor in the above process, the feature data of the original image can also be obtained.
[0071] After obtaining the feature data of the original image and the image feature data of the object to be processed, the image feature data of the object to be processed and the feature data of the original image are provided to the generator to generate a processed image. In the processed image, the object to be processed is transformed into a processed object of the target object type. Finally, the server sends the processed image to the client, and the client receives the processed image.
[0072] In the above-mentioned image data processing method, the original image refers to the initial image to be processed, and the processed image can be obtained after the processing. The processed image refers to the image in which the object to be processed in the original image is processed. The object to be processed refers to a certain object to be processed in the original image, and can be any object in the original image. For example, the object to be processed can refer to a certain image area in the original image that needs to be processed. Specifically, processing the object to be processed refers to transforming the object to be processed into a processed object that conforms to the target object type. For example, assuming that the object to be processed in the original image is A, and the target object type is to transform A in the original image into B, then the processed object is B. The processed object is the object after the object to be processed in the original image is transformed.
[0073] The target object type is a preset object transformation condition type or a target object. In other words, it is a preset condition for transforming an object in an original image; the target object type can also be a pointer to an image feature transformation type of an object to be processed, such as a texture transformation condition for an object to be processed input by a user.
[0074] The key point information related to the object to be processed in the original image refers to the information of the key points of the object to be processed itself in the original image. It can be used for the image feature data of the object to be processed. The image feature data of the object to be processed refers to the image-related feature information of the object to be processed in the original image, for example, it can be the image texture information of the object to be processed in the original image, the position information of the object to be processed in the original image. The feature data of the original image can refer to the image texture information of the original image or other feature information related to the image. The generator refers to a network model that uses the above-mentioned image feature data of the object to be processed and the feature data of the original image as input data to obtain the processed image. The generator can be a generator that is being trained or a generator that has been trained.
[0075] Before using the generator to generate the processed image, the generator and the discriminator need to be trained adversarially in advance. The specific adversarial training process is as follows: First, obtain the original image sample for training and the training sample of the target image sample. After that, obtain the key point information related to the object to be processed of the original image sample, and obtain the image feature data of the object to be processed of the original image sample through the key point information related to the object to be processed of the original image sample. After that, the image feature data of the object to be processed of the original image sample and the image feature data of the original image sample are provided to the initial generator to obtain the first processed image, wherein the first processed image refers to the processed image obtained by the initial discriminator, and of course the first processed image here is the processed image for the original image sample. After obtaining the first processed image, the initial discriminator determines whether the difference between the first processed image and the target image sample makes the loss value of the loss function meet the preset target training condition; wherein the loss function is a function constructed based on the Cycle loss function and the L2 loss function of the CycleGAN network model. If the above judgment result is no, the model parameters of the initial generator and the initial discriminator are adjusted. The generator after adjusting the parameters generates a second processed image of the original image sample, and the discriminator after adjusting the parameters is used to determine whether the difference between the second processed image and the target image sample makes the loss value of the loss function meet the preset target training condition. If the above judgment result is yes, stop adjusting the model parameters of the generator and the discriminator, and use the generator after the last adjustment of the model parameters as the trained generator. If the above judgment result is no, continue to obtain the third processed image, and so on, until the above judgment result is yes.
[0076] Reference Figure 1-B , which is a schematic diagram of another application scenario embodiment provided by the present application. The schematic diagram of the scenario embodiment shows the process of training the generator and the discriminator before using the method of the present application, and can use the trained generator to transform an original image of a model wearing jeans into a processed image of the model wearing a denim skirt, and the image texture of the jeans in the original image is consistent with the image texture of the denim skirt in the processed image. At the same time, the newly generated leg image texture of the model in the processed image is consistent with the model skin image texture in the original image.
[0077] The main process of training the generator and discriminator is as follows: First, the original image of the model wearing jeans is used as the original image sample. At the same time, the target image of the model wearing a denim skirt obtained by other methods can be used as the target image sample. During the training process, the model parameters of the generator and the discriminator are continuously adjusted so that the processed image obtained by taking the original image sample as the input of the generator gradually approaches the target image of the model wearing a denim skirt obtained by other methods, until the discriminator finally determines the difference between the processed image generated by the generator and the target image of the model wearing a denim skirt obtained by other methods so that the loss value of the loss function meets the preset target training conditions.
[0078] Since this scenario embodiment is to generate images of other categories by cross-category of clothing images, and it is necessary to transfer the image texture of the object to be processed to the image texture of the processed object. Therefore, when training the generator and trainer, the image texture of the object to be processed is actually constrained. In this scenario embodiment, the key point information of human skin and clothing is used in the training process to train the generator and trainer to enhance the generator and trainer's ability to generate and discriminate the features of the object to be processed in the image.
[0079] exist Figure 1-B In , the training process is described as follows: First, the original image (the original image of the model wearing jeans) is used as the input data of the network model for obtaining key point information, and the key point information of the original image and the mask information of the jeans are obtained. After that, the key point information related to the jeans filtered out from the above key point information, and the jeans mask information and the original image are used to obtain the mask features and image feature data of the jeans through the first feature extractor. The image feature data may include the image texture of each part of the jeans. After that, the mask features and image feature data of the jeans are subjected to the concat function operation, and the processed image and the mask information of the denim skirt are obtained according to the mask generator and the image generator in the generator. The image processed by the first process mentioned above is marked as the first processed image.
[0080] After obtaining the first processed image and the denim skirt mask information, the first processed image can be used to continuously optimize the network model for obtaining key point information, so that the key point information obtained by the network model for obtaining key point information is more useful for obtaining jeans mask information. At the same time, the first processed image and the denim skirt mask information are input into the first feature extractor, and after passing through the image discriminator and the mask discriminator respectively, the output result after the image discriminator is subjected to matrix multiplication operation. Afterwards, the matrix multiplication result and the output result of the mask discriminator are subjected to concat function operation, and then input into the initial discriminator. The initial discriminator determines whether the difference between the image feature data of the object to be processed in the first processed image and the target image of the model wearing a denim skirt makes the loss value of the loss function meet the preset target training conditions. If not, the model parameters of the initial generator and the initial discriminator are adjusted. Iterate according to the above process until the processed image obtained meets the above conditions, and then stop adjusting the model parameters of the generator and the discriminator. Thus, a trained generator is obtained.
[0081] The image of the model wearing a denim skirt generated by the trained generator realizes the change of clothing category in the clothing image, for example, the jeans in the image change from the clothing category of jeans to the clothing category of a denim skirt, while maintaining the image texture of the original clothing category clothing, and can also ensure that the image texture of the newly generated skin part is relatively natural (that is: the image of the new skin part generated due to the clothing category change from jeans to denim skirt in the image). It should be noted that the application scenario is only an embodiment of the application scenario, and the purpose of providing this application scenario embodiment is to facilitate the understanding of the image data processing method of the present application, and is not used to limit the image data processing method of the present application.
[0082] The present application provides a method and device for processing image data, an electronic device, and a computer storage medium.
[0083] The following are specific embodiments.
[0084] like Figure 1 As shown, it is a flow chart of the image data processing method of the first embodiment of the present application. The method comprises the following steps.
[0085] Step S101: obtaining an original image and a target object type; the original image contains at least one object to be processed.
[0086] In the image processing method of this embodiment, the original image is first acquired, and the original image is used as a prerequisite for generating the processed image.
[0087] The original image refers to the initial image to be processed, and the processed image can be obtained after processing. The processed image refers to the image after the object to be processed in the original image is processed. Specifically, processing the object to be processed means transforming the object to be processed into a processed object that conforms to the target object type. For example, assuming that the object to be processed in the original image is A, and the target object type is to transform A in the original image into B, then the processed object is B. The processed object is the object after the object to be processed in the original image is transformed.
[0088] The target object type is a preset transformation condition type of an object in the image or can also be a target object. In other words, it is a preset condition for transforming an object in the original image; the target object type can also be a pointer to the image feature transformation type of the object to be processed, such as a texture transformation condition for the object to be processed input by the user through custom input. The object to be processed refers to an object to be processed in the original image, and can be any object in the original image. For example, the object to be processed can refer to an image area in the original image that needs to be processed.
[0089] Step S102: acquiring key point information related to the object to be processed in the original image according to the original image.
[0090] After the original image is acquired in step S101, key point information related to the object to be processed in the original image is acquired according to the original image. As one of the ways to acquire key point information related to the object to be processed in the original image: the original image is used as input data of a network model for acquiring key point information, and the output result of the network model for acquiring key point information is used as key point information related to the object to be processed in the original image; wherein the network model for acquiring key point information is a network model that can acquire key point information related to the object to be processed in the original image according to the original image.
[0091] In the above scenario of obtaining a target image of a model wearing a denim skirt, the network model for obtaining key point information is used to obtain the clothing key points and human key points of the input image. The clothing key points are pre-trained on the Deepfashion2 dataset, and the human key points are pre-trained on the COCO dataset. The clothing key points and human key points obtained by the network model for obtaining key point information can be superimposed on the feature extractor to obtain image features, enhance important image features, and weaken non-important image features, so that the discriminant ability of the discriminator is effectively improved; on the other hand, the obtained clothing key points and human key points can obtain the position information of the object to be processed in the original image.
[0092] Step S103: Obtain image feature data of the object to be processed according to the key point information, provide the image feature data of the object to be processed and the feature data of the original image to the generator, and generate a processed image; in the processed image, the object to be processed is transformed into a processed object of the target object type.
[0093] After obtaining the key point information, image feature data of the object to be processed is obtained according to the key point information.
[0094] As an implementation method of obtaining image feature data of an object to be processed based on key point information, first, key point position information of the key point in the original image is obtained based on the key point information. Then, image feature data at the key point position in the original image is extracted, and the extracted image feature data at the key point position in the original image is used as the image feature data of the object to be processed.
[0095] As another implementation method of obtaining the image feature data of the object to be processed according to the key point information, first, the position information of the object to be processed in the original image is obtained according to the key point information. Then, the mask information of the object to be processed is obtained according to the position information of the object to be processed in the original image, and the mask information of the object to be processed is used as the image feature data of the object to be processed.
[0096] While obtaining the image feature data of the object to be processed, and before providing the image feature data of the object to be processed and the feature data of the original image to the generator, the feature data of the original image also needs to be obtained. Here, the generator can be a generator being trained or a trained generator.
[0097] There are many ways to obtain the feature data of the original image, one of which can be to obtain the feature data of the original image based on the original image. Specifically, the feature data of the original image can be obtained by: inputting the original image into the first feature extractor, obtaining the first feature data extracted by the first feature extractor for the original image, and using the first feature data as the feature data of the original image. In this embodiment, the first feature extractor is used as a feature extractor together with the generator in the generator part to obtain the feature data of the image before the generator.
[0098] After obtaining the image feature data of the object to be processed and the feature data of the original image, the image feature data of the object to be processed and the feature data of the original image are provided to the generator. Before generating the processed image, the mask information of the object to be processed is merged and encoded with the feature data of the original image to obtain the encoding result.
[0099] After obtaining the encoding result, the encoding result, the image feature data of the object to be processed and the feature data of the original image are provided to the generator, and the generator generates a processed image based on the encoding result, the image feature data of the object to be processed and the feature data of the original image.
[0100] In this embodiment, adversarial training is performed on the generator and the discriminator based on the original image samples and the target image samples.
[0101] Specifically, the process of adversarial training of the generator and the discriminator based on the original image samples and the target image samples is described as follows.
[0102] First, an original image sample and a target image sample are obtained; the target image sample refers to an image sample obtained by transforming the object to be processed of the original image sample according to the target object type.
[0103] Afterwards, the image feature data of the original image sample and the image feature data of the object to be processed in the original image sample are provided to the initial generator to generate a first processed image.
[0104] After that, an initial discriminator is used to determine whether the difference between the first processed image and the target image sample makes the loss value of the loss function meet the preset target training conditions; wherein the loss function is a function constructed by the Cycle loss function and the L2 loss function based on the CycleGAN network model. In this embodiment, the generator and the discriminator are trained based on the CycleGAN network model, so the loss function is the Cycle loss function based on the CycleGAN network model. At the same time, in this embodiment, it is necessary to ensure that the image texture of the object to be processed is migrated to the processed object. Therefore, it is necessary to have an L2 loss function in the loss function. In the above scenario embodiment, it is mainly reflected in the image texture loss of denim fabric and the image texture loss of human skin. Specifically, the image texture loss of denim fabric and the image texture loss of human skin are both calculated by mean square error, which is mainly achieved by the L2 loss function.
[0105] The process of adjusting the parameters of the generator and discriminator models using the loss function is as follows.
[0106] If the above loss value does not meet the preset target training conditions, the model parameters of the initial generator and the initial discriminator are adjusted.
[0107] Afterwards, the image feature data of the original image sample, the image feature data of the object to be processed in the original image sample and the image feature data of the first processed image are provided to the generator after adjusting the model parameters to generate a second processed image.
[0108] Similarly, the discriminator after adjusting the model parameters is used to determine whether the difference between the second processed image and the target image sample makes the loss value of the loss function meet the preset target training conditions; if the loss value meets the preset target training conditions, stop adjusting the model parameters of the generator and the discriminator, obtain the trained generator, and use the trained generator as the generator.
[0109] Otherwise, continue to adjust the model parameters of the generator and the discriminator, generate a third processed image, and continue to determine whether the difference between the third processed image and the target image sample makes the loss value of the loss function meet the preset target training condition. And so on, until the difference is obtained so that the loss value of the loss function meets the preset target training condition.
[0110] Specifically, it is determined whether the difference between the first processed image and the target image sample makes the loss value of the loss function meet the preset target training condition in the following manner.
[0111] First, first key point information related to the object to be processed in the first processed image is obtained according to the first processed image; image feature data of the object to be processed in the first processed image is obtained according to the first key point information. At the same time, second key point information related to the object to be processed in the target image sample is obtained according to the target image sample; image feature data of the object to be processed in the target image sample is obtained according to the second key point information. Afterwards, according to the image feature data of the object to be processed in the first processed image and the image feature data of the object to be processed in the target image sample, it is determined whether the difference between the first processed image and the target image sample makes the loss value of the loss function meet the preset target training condition.
[0112] Before obtaining the image feature data of the object to be processed in the first processed image according to the first key point information, the method further includes: encoding the first key point information related to the object to be processed in the first processed image by matrix multiplication to obtain the first key point information encoding data. Afterwards, obtaining the image feature data of the object to be processed in the first processed image according to the first key point information encoding data and the feature data of the first processed image.
[0113] Since the feature data of the first processed image is needed when the initial discriminator is used to determine whether the difference between the first processed image and the target image sample makes the loss value of the loss function meet the preset target training condition, the feature data of the first processed image needs to be obtained in advance.
[0114] Specifically, obtaining the feature data of the first processed image may be inputting the first processed image into the second feature extractor, obtaining the second feature data extracted by the second feature extractor for the first processed image, and using the second feature data as the feature data of the first processed image. In this embodiment, the second feature extractor is used as a feature extractor together with the discriminator in the discriminator part to obtain the feature data of the image before the discriminator.
[0115] The image data processing method of this embodiment is used to obtain key point information related to the object to be processed in the original image, and then obtain image feature data of the object to be processed in the original image, so that the image feature data of the object to be processed, the feature data of the original image and the processed image generated by the generator can meet the processing requirements of the target object type, thereby achieving the effect of effectively processing the object to be processed in the image.
[0116] In the first embodiment described above, a method for processing image data is provided. Correspondingly, the present application provides an image data processing device. Figure 2 As shown, it is a schematic diagram of an image data processing device according to a second embodiment of the present application. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.
[0117] A second embodiment of the present application provides an image data processing device, including:
[0118] The first acquisition unit 201 is used to acquire an original image and a target object type; the original image contains at least one object to be processed;
[0119] A key point information acquisition unit 202 is used to acquire key point information related to the object to be processed in the original image according to the original image;
[0120] An image feature data acquisition unit 203 is used to acquire image feature data of the object to be processed according to the key point information;
[0121] The image generating unit 204 is used to provide the image feature data of the object to be processed and the feature data of the original image to a generator to generate a processed image; in the processed image, the object to be processed is transformed into a processed object of the target object type.
[0122] Optionally, the key point information acquisition unit is specifically used to:
[0123] The original image is used as input data of a network model for obtaining key point information, and the output result of the network model for obtaining key point information is used as the key point information related to the object to be processed in the original image; wherein the network model for obtaining key point information is a network model that can obtain the key point information related to the object to be processed in the original image based on the original image.
[0124] Optionally, the image feature data acquisition unit is specifically used to:
[0125] According to the key point information, obtaining key point position information of the key point in the original image;
[0126] Image feature data at key point positions in the original image are extracted, and the extracted image feature data at key point positions in the original image are used as image feature data of the object to be processed.
[0127] Optionally, the image feature data acquisition unit is specifically used to:
[0128] According to the key point information, obtaining the position information of the object to be processed in the original image;
[0129] According to the position information of the object to be processed in the original image, the mask information of the object to be processed is acquired, and the mask information of the object to be processed is used as the image feature data of the object to be processed.
[0130] Optionally, it also includes a coding unit;
[0131] The encoding unit is used to combine and encode the mask information of the object to be processed and the feature data of the original image before providing the image feature data of the object to be processed and the feature data of the original image to the generator to generate a processed image, so as to obtain an encoding result.
[0132] Optionally, the image generating unit is specifically used to:
[0133] The encoding result, the image feature data of the object to be processed and the feature data of the original image are provided to a generator to generate a processed image.
[0134] Optionally, it also includes a feature data acquisition unit for the original image;
[0135] The original image feature data acquisition unit is used to acquire the feature data of the original image based on the original image before providing the image feature data of the object to be processed and the feature data of the original image to the generator.
[0136] Optionally, the original image feature data acquisition unit is specifically used to:
[0137] The original image is input into a first feature extractor, first feature data extracted by the first feature extractor for the original image is obtained, and the first feature data is used as feature data of the original image.
[0138] Optionally, the target object type includes at least one of the following types:
[0139] A type of category transformation for the object to be processed;
[0140] The image feature transformation type for the object to be processed.
[0141] Optionally, a training unit is also included;
[0142] The training unit is used to perform adversarial training on the generator and the discriminator based on the original image samples and the target image samples.
[0143] Optionally, the training unit is specifically used to:
[0144] Acquire an original image sample and a target image sample; the target image sample refers to an image sample after the object to be processed of the original image sample is transformed according to the target object type;
[0145] Providing the image feature data of the original image sample and the image feature data of the object to be processed in the original image sample to an initial generator to generate a first processed image;
[0146] An initial discriminator is used to determine whether the difference between the first processed image and the target image sample makes the loss value of the loss function meet the preset target training condition; wherein the loss function is a function constructed based on the Cycle loss function and the L2 loss function of the CycleGAN network model;
[0147] If the loss value does not meet the preset target training condition, adjusting the model parameters of the initial generator and the initial discriminator;
[0148] Providing the image feature data of the original image sample, the image feature data of the object to be processed in the original image sample, and the image feature data of the first processed image to the generator after adjusting the model parameters, to generate a second processed image;
[0149] The discriminator after adjusting the model parameters determines whether the difference between the second processed image and the target image sample makes the loss value of the loss function meet the preset target training condition; if the loss value meets the preset target training condition, stop adjusting the model parameters of the generator and the discriminator, obtain the trained generator, and use the trained generator as the generator;
[0150] Otherwise, continue to adjust the model parameters of the generator and the discriminator, and generate a third processed image, and continue to determine whether the difference between the third processed image and the target image sample makes the loss value of the loss function meet the preset target training conditions; and so on, until a processed image is obtained in which the difference makes the loss value of the loss function meet the preset target training conditions.
[0151] Optionally, the training unit is specifically used to:
[0152] Acquire first key point information related to the object to be processed in the first processed image according to the first processed image; acquire image feature data of the object to be processed in the first processed image according to the first key point information;
[0153] Acquire second key point information related to the object to be processed in the target image sample according to the target image sample; acquire image feature data of the object to be processed in the target image sample according to the second key point information;
[0154] Based on the image feature data of the object to be processed in the first processed image and the image feature data of the object to be processed in the target image sample, it is determined whether the difference between the first processed image and the target image sample makes the loss value of the loss function meet the preset target loss condition.
[0155] Optionally, it also includes a key point information encoding data acquisition unit;
[0156] The key point information encoding data acquisition unit is used to encode the first key point information related to the object to be processed in the first processed image by matrix multiplication to obtain the first key point information encoding data;
[0157] The training unit is specifically used to obtain image feature data of the object to be processed in the first processed image according to the first key point information encoding data and the feature data of the first processed image.
[0158] Optionally, it also includes a feature data acquisition unit for processing the image;
[0159] The feature data acquisition unit for the processed image is specifically used to input the first processed image into the second feature extractor before using the initial discriminator to determine whether the difference between the first processed image and the target image sample makes the loss value of the loss function meet the preset target loss condition, obtain the second feature data extracted by the second feature extractor for the first processed image, and use the second feature data as the feature data of the first processed image.
[0160] The first embodiment of the present application provides an image data processing method, and the third embodiment of the present application provides an electronic device corresponding to the method of the first embodiment.
[0161] like Figure 3 As shown, it shows a schematic diagram of the electronic device of this embodiment.
[0162] This embodiment provides an electronic device, including:
[0163] Processor 301;
[0164] The memory 302 is used to store a computer program. The computer program is executed by the processor to perform an image data processing method. The method includes the following steps:
[0165] Acquire an original image and a target object type; the original image contains at least one object to be processed;
[0166] Acquire key point information related to the object to be processed in the original image according to the original image;
[0167] The image feature data of the object to be processed is obtained according to the key point information, and the image feature data of the object to be processed and the feature data of the original image are provided to a generator to generate a processed image; in the processed image, the object to be processed is transformed into a processed object of the target object type.
[0168] The first embodiment of the present application provides an image data processing method, and the fourth embodiment of the present application provides a computer storage medium corresponding to the method of the first embodiment.
[0169] This embodiment provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is executed by a processor to perform an image data processing method, wherein the method comprises the following steps:
[0170] Acquire an original image and a target object type; the original image contains at least one object to be processed;
[0171] Acquire key point information related to the object to be processed in the original image according to the original image;
[0172] The image feature data of the object to be processed is obtained according to the key point information, and the image feature data of the object to be processed and the feature data of the original image are provided to a generator to generate a processed image; in the processed image, the object to be processed is transformed into a processed object of the target object type.
[0173] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.
[0174] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
[0175] 1. Computer readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include non-transitory computer readable storage media (non-transitorycomputer readable storage media), such as modulated data signals and carrier waves.
[0176] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
Claims
1. A method for processing image data, characterized in that: include: Acquire an original image and a target object type; the original image contains at least one object to be processed; Acquire key point information related to the object to be processed in the original image according to the original image; Acquire image feature data of the object to be processed according to the key point information, provide the image feature data of the object to be processed and the feature data of the original image to a generator, and generate a processed image; in the processed image, the object to be processed is transformed into a processed object of the target object type; The generator is obtained by performing adversarial training on the generator and the discriminator based on the original image sample and the target image sample, and the training method includes: Acquire an original image sample and a target image sample; the target image sample refers to an image sample after the object to be processed of the original image sample is transformed according to the target object type; Providing the image feature data of the original image sample and the image feature data of the object to be processed in the original image sample to an initial generator to generate a first processed image; Using an initial discriminator to determine whether the difference between the first processed image and the target image sample makes the loss value of the loss function meet a preset target training condition; If the loss value does not meet the preset target training condition, adjusting the model parameters of the initial generator and the initial discriminator; Providing the image feature data of the original image sample, the image feature data of the object to be processed in the original image sample, and the image feature data of the first processed image to the generator after adjusting the model parameters, to generate a second processed image; The discriminator after adjusting the model parameters determines whether the difference between the second processed image and the target image sample makes the loss value of the loss function meet the preset target training condition; if the loss value meets the preset target training condition, stop adjusting the model parameters of the generator and the discriminator, obtain the trained generator, and use the trained generator as the generator; Otherwise, continue to adjust the model parameters of the generator and the discriminator, and generate a third processed image, and continue to determine whether the difference between the third processed image and the target image sample makes the loss value of the loss function meet the preset target training conditions; and so on, until a processed image is obtained in which the difference makes the loss value of the loss function meet the preset target training conditions.
2. The method according to claim 1, characterized in that The step of obtaining key point information related to the object to be processed in the original image includes: The original image is used as input data of a network model for obtaining key point information, and the output result of the network model for obtaining key point information is used as key point information related to the object to be processed in the original image; wherein the network model for obtaining key point information is a network model that can obtain key point information related to the object to be processed in the original image based on the original image.
3. The method according to claim 1, characterized in that: The step of acquiring the image feature data of the object to be processed according to the key point information includes: According to the key point information, obtaining key point position information of the key point in the original image; Image feature data at key point positions in the original image are extracted, and the extracted image feature data at key point positions in the original image are used as image feature data of the object to be processed.
4. The method according to claim 1, characterized in that The step of acquiring the image feature data of the object to be processed according to the key point information includes: According to the key point information, obtaining the position information of the object to be processed in the original image; According to the position information of the object to be processed in the original image, the mask information of the object to be processed is acquired, and the mask information of the object to be processed is used as the image feature data of the object to be processed.
5. The method according to claim 4, characterized in that Before providing the image feature data of the object to be processed and the feature data of the original image to the generator to generate the processed image, the method further includes: merging and encoding the mask information of the object to be processed and the feature data of the original image to obtain a coding result.
6. The method according to claim 5, characterized in that The step of providing the image feature data of the object to be processed and the feature data of the original image to a generator to generate a processed image comprises: The encoding result, the image feature data of the object to be processed and the feature data of the original image are provided to a generator to generate a processed image.
7. The method according to claim 1, characterized in that Before providing the image feature data of the object to be processed and the feature data of the original image to the generator, the method further includes acquiring the feature data of the original image based on the original image.
8. The method according to claim 7, characterized in that The step of acquiring feature data of the original image according to the original image includes: The original image is input into a first feature extractor, first feature data extracted by the first feature extractor for the original image is obtained, and the first feature data is used as feature data of the original image.
9. The method according to claim 1, characterized in that: The target object type includes at least one of the following types: A type of category transformation for the object to be processed; The image feature transformation type for the object to be processed.
10. The method according to claim 1, characterized in that in, The loss function is a function constructed based on the Cycle loss function and the L2 loss function of the CycleGAN network model.
11. The method according to claim 1, characterized in that The determining whether the difference between the first processed image and the target image sample makes the loss value of the loss function meet the preset target training condition includes: Acquire first key point information related to the object to be processed in the first processed image according to the first processed image; acquire image feature data of the object to be processed in the first processed image according to the first key point information; Acquire second key point information related to the object to be processed in the target image sample according to the target image sample; acquire image feature data of the object to be processed in the target image sample according to the second key point information; Based on the image feature data of the object to be processed in the first processed image and the image feature data of the object to be processed in the target image sample, it is determined whether the difference between the first processed image and the target image sample makes the loss value of the loss function meet the preset target loss condition.
12. The method according to claim 11, characterized in that Also includes: Encoding first key point information related to the object to be processed in the first processed image by using matrix multiplication to obtain first key point information encoding data; The step of acquiring image feature data of the object to be processed in the first processed image according to the first key point information includes: Image feature data of the object to be processed in the first processed image is obtained according to the first key point information encoding data and the feature data of the first processed image.
13. The method according to claim 1, characterized in that Before using the initial discriminator to determine whether the difference between the first processed image and the target image sample makes the loss value of the loss function meet the preset target loss condition, the method further includes: The first processed image is input into a second feature extractor, second feature data extracted by the second feature extractor for the first processed image is obtained, and the second feature data is used as feature data of the first processed image.
14. An image data processing device, characterized in that: include: A first acquisition unit is used to acquire an original image and a target object type; the original image contains at least one object to be processed; A key point information acquisition unit, used for acquiring key point information related to the object to be processed in the original image according to the original image; An image feature data acquisition unit, used to acquire image feature data of the object to be processed according to the key point information; An image generating unit, configured to provide the image feature data of the object to be processed and the feature data of the original image to a generator, so as to generate a processed image; in the processed image, the object to be processed is transformed into a processed object of the target object type; The generator is obtained by performing adversarial training on the generator and the discriminator based on the original image sample and the target image sample, and the training method includes: Acquire an original image sample and a target image sample; the target image sample refers to an image sample after the object to be processed of the original image sample is transformed according to the target object type; Providing the image feature data of the original image sample and the image feature data of the object to be processed in the original image sample to an initial generator to generate a first processed image; Using an initial discriminator to determine whether the difference between the first processed image and the target image sample makes the loss value of the loss function meet a preset target training condition; If the loss value does not meet the preset target training condition, adjusting the model parameters of the initial generator and the initial discriminator; Providing the image feature data of the original image sample, the image feature data of the object to be processed in the original image sample, and the image feature data of the first processed image to the generator after adjusting the model parameters, to generate a second processed image; The discriminator after adjusting the model parameters determines whether the difference between the second processed image and the target image sample makes the loss value of the loss function meet the preset target training condition; if the loss value meets the preset target training condition, stop adjusting the model parameters of the generator and the discriminator, obtain the trained generator, and use the trained generator as the generator; Otherwise, continue to adjust the model parameters of the generator and the discriminator, and generate a third processed image, and continue to determine whether the difference between the third processed image and the target image sample makes the loss value of the loss function meet the preset target training conditions; and so on, until a processed image is obtained in which the difference makes the loss value of the loss function meet the preset target training conditions.
15. An electronic device, characterized in that: include: processor; The memory is used to store a computer program, the computer program is executed by the processor to perform an image data processing method, the method comprising the following steps: Acquire an original image and a target object type; the original image contains at least one object to be processed; Acquire key point information related to the object to be processed in the original image according to the original image; Acquire image feature data of the object to be processed according to the key point information, provide the image feature data of the object to be processed and the feature data of the original image to a generator, and generate a processed image; in the processed image, the object to be processed is transformed into a processed object of the target object type; The generator is obtained by performing adversarial training on the generator and the discriminator based on the original image sample and the target image sample, and the training method includes: Acquire an original image sample and a target image sample; the target image sample refers to an image sample after the object to be processed of the original image sample is transformed according to the target object type; Providing the image feature data of the original image sample and the image feature data of the object to be processed in the original image sample to an initial generator to generate a first processed image; Using an initial discriminator to determine whether the difference between the first processed image and the target image sample makes the loss value of the loss function meet a preset target training condition; If the loss value does not meet the preset target training condition, adjusting the model parameters of the initial generator and the initial discriminator; Providing the image feature data of the original image sample, the image feature data of the object to be processed in the original image sample, and the image feature data of the first processed image to the generator after adjusting the model parameters, to generate a second processed image; The discriminator after adjusting the model parameters determines whether the difference between the second processed image and the target image sample makes the loss value of the loss function meet the preset target training condition; if the loss value meets the preset target training condition, stop adjusting the model parameters of the generator and the discriminator, obtain the trained generator, and use the trained generator as the generator; Otherwise, continue to adjust the model parameters of the generator and the discriminator, and generate a third processed image, and continue to determine whether the difference between the third processed image and the target image sample makes the loss value of the loss function meet the preset target training conditions; and so on, until a processed image is obtained in which the difference makes the loss value of the loss function meet the preset target training conditions.
16. A computer storage medium, characterized in that: The computer storage medium stores a computer program, which is executed by the processor to perform an image data processing method, the method comprising the following steps: Acquire an original image and a target object type; the original image contains at least one object to be processed; Acquire key point information related to the object to be processed in the original image according to the original image; Acquire image feature data of the object to be processed according to the key point information, provide the image feature data of the object to be processed and the feature data of the original image to a generator, and generate a processed image; in the processed image, the object to be processed is transformed into a processed object of the target object type; The generator is obtained by performing adversarial training on the generator and the discriminator based on the original image sample and the target image sample, and the training method includes: Acquire an original image sample and a target image sample; the target image sample refers to an image sample after the object to be processed of the original image sample is transformed according to the target object type; Providing the image feature data of the original image sample and the image feature data of the object to be processed in the original image sample to an initial generator to generate a first processed image; Using an initial discriminator to determine whether the difference between the first processed image and the target image sample makes the loss value of the loss function meet a preset target training condition; If the loss value does not meet the preset target training condition, adjusting the model parameters of the initial generator and the initial discriminator; Providing the image feature data of the original image sample, the image feature data of the object to be processed in the original image sample, and the image feature data of the first processed image to the generator after adjusting the model parameters, to generate a second processed image; The discriminator after adjusting the model parameters determines whether the difference between the second processed image and the target image sample makes the loss value of the loss function meet the preset target training condition; if the loss value meets the preset target training condition, stop adjusting the model parameters of the generator and the discriminator, obtain the trained generator, and use the trained generator as the generator; Otherwise, continue to adjust the model parameters of the generator and the discriminator, and generate a third processed image, and continue to determine whether the difference between the third processed image and the target image sample makes the loss value of the loss function meet the preset target training conditions; and so on, until a processed image is obtained in which the difference makes the loss value of the loss function meet the preset target training conditions.