Image processing method and device, electronic equipment, medium and computer program product

Abstract

The invention discloses an image processing method and device, electronic equipment, a medium and a computer program product, and belongs to the technical field of artificial intelligence, the method comprises the steps that a first image and a first text are acquired, the first image comprises an image main body, and the first text is used for describing an expected image main body posture; and adjusting an image main body posture in the first image according to the first text to obtain a second image.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an image processing method, device, electronic device, medium and computer program product. Background Art

[0002] With the popularity of social media and the prevalence of electronic device photography, people have an increasing demand for taking and sharing photos. In this context, portrait photography has become a popular way of expression, and users hope to show the best of themselves through the photos they take. It is understandable that the posture of the person in the photo is crucial to the beauty of the image. A graceful posture can make the photo better, while a poor posture may reduce the attractiveness of the photo and even affect the user's satisfaction with the photo and the degree of sharing. At present, for images that have been taken, users can correct the portrait posture in the image through professional image processing applications.

[0003] However, the above methods require multiple editing of the image, which is cumbersome and time-consuming, and the processing effect usually does not meet expectations. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device, medium and computer program product, which can simplify the operation process of image processing and improve the effect of image processing.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising: acquiring a first image and a first text, the first image including an image subject, the first text being used to describe a desired image subject posture; according to the first text, adjusting the image subject posture in the first image to obtain a second image.

[0006] In a second aspect, an embodiment of the present application provides an image processing device, which includes: an acquisition module and a processing module; the acquisition module is used to acquire a first image and a first text, the first image includes an image subject, and the first text is used to describe the desired image subject posture; the processing module is used to adjust the image subject posture in the first image according to the first text acquired by the acquisition module to obtain a second image.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0011] In the embodiment of the present application, the electronic device may first obtain a first image and a first text, wherein the first image includes an image subject, and the first text is used to describe the desired image subject posture; then, the image subject posture in the first image is adjusted according to the first text to obtain a second image. Through this solution, the posture of the image subject in the image can be directly adjusted according to the description of the desired image subject posture in the text, without the user having to manually process the image multiple times, and an image containing the image subject posture that meets the user's needs can be obtained, thereby simplifying the image processing operation process and improving the image processing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is one of the flow charts of an image processing method provided in an embodiment of the present application;

[0013] Figure 2 This is the second flowchart of an image processing method provided in an embodiment of the present application;

[0014] Figure 3 This is the third flowchart of an image processing method provided in an embodiment of the present application;

[0015] Figure 4 is a schematic diagram of a processing process of a posture estimation module provided in an embodiment of the present application;

[0016] Figure 5 This is a fourth flowchart of an image processing method provided in an embodiment of the present application;

[0017] Figure 6 is a schematic diagram of an example of generating a second image through a first model provided by an embodiment of the present application;

[0018] Figure 7 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0019] Figure 8 This is one of the hardware structure diagrams of an electronic device provided in an embodiment of the present application;

[0020] Fig. 9 This is the second schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0022] The terms "first", "second", etc. in the specification of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0023] The terms "at least one (item)", "at least one of" and the like in the specification of the present application refer to any one, any two or a combination of more than two of the objects contained therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)". The logo in the present application is text, symbol, image, etc. used to indicate information, and logos or other containers can be used as carriers for displaying information, including but not limited to text logos, image logos, symbol logos, etc.

[0024] The image processing method, device, electronic device, medium and computer program product provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0025] The image processing method, device, electronic device, medium and computer program product provided in the embodiments of the present application can be applicable to scenarios where images are processed, especially scenarios where the posture of the image subject is adjusted.

[0026] The image processing method provided in the embodiment of the present application can directly adjust the posture of the image subject in the image according to the description of the desired image subject posture in the text, without the user having to manually process the image multiple times, so as to obtain an image containing the image subject posture that meets the user's needs, thereby simplifying the image processing operation process and improving the image processing effect.

[0027] The execution subject of the image processing method provided in the embodiment of the present application may be an image processing device. Exemplarily, the image processing device may be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The image processing method provided in the embodiment of the present application will be exemplarily described below using an electronic device as an example.

[0028] The present application provides an image processing method. Figure 1 FIG. 1 is a flowchart of an image processing method provided by an embodiment of the present application, which can be applied to electronic devices. Figure 1 As shown, the image processing method provided in the embodiment of the present application may include the following steps 101 and 102.

[0029] Step 101: An electronic device acquires a first image and a first text.

[0030] The first image may include an image subject, and the first text may be used to describe a desired posture of the image subject.

[0031] In some embodiments of the present application, the above-mentioned image subject may refer to an important component in the image.

[0032] Exemplarily, the above-mentioned image subject may be a photographed object in the image.

[0033] For example, the image subject may be a person or an animal in the image.

[0034] In some embodiments of the present application, the first image may be an image stored in the electronic device, an image captured by the electronic device, or an image acquired by the electronic device from other devices. This embodiment of the present application does not specifically limit this.

[0035] In some embodiments of the present application, the first text may be a description text of the desired image subject posture input by the user, or may be a description text of the image subject posture acquired by the electronic device from other devices.

[0036] For example, Figure 2 As shown, the second text may include descriptive texts such as two hands on hips, one hand on hips, upright posture, running, sexy, lively and cute, making a "V" sign, bending one leg, etc.

[0037] It can be understood that the posture of the image subject described in the first text is different from the posture of the image subject in the first image.

[0038] Exemplarily, assuming that the subject of the first image is a person, and the person's posture is standing with hands on hips, the first text may be: standing with hands raised above the head.

[0039] Step 102: The electronic device adjusts the posture of the image subject in the first image according to the first text to obtain a second image.

[0040] In some embodiments of the present application, the posture of the image subject in the second image is consistent with the description of the image subject posture in the first text.

[0041] For example, assuming that the subject of the image in the first image is a person, and the person's posture is standing with hands on hips, and the first text is: standing with hands raised above the head. Then, after the electronic device adjusts the posture of the subject of the image in the first image according to the first text, in the obtained second image, the posture of the person in the second image is standing with hands raised above the head.

[0042] The image processing method provided in the embodiment of the present application can directly adjust the posture of the image subject in the image according to the description of the desired image subject posture in the text, without the user having to manually process the image multiple times, so as to obtain an image containing the image subject posture that meets the user's needs, thereby simplifying the image processing operation process and improving the image processing effect.

[0043] In some embodiments of the present application, Figure 1 ,like Figure 2 As shown, the above step 102 can be specifically implemented through the following steps 102a and 102b.

[0044] Step 102a: The electronic device estimates the posture of the image subject based on the first text and the first image to obtain a first posture skeleton image.

[0045] In some embodiments of the present application, the first posture skeleton image may be: a posture skeleton image of an image subject of the first image that includes posture features described in the first text.

[0046] It should be noted that the posture skeleton image may be an image similar to a skeleton structure obtained by the electronic device by locating key points of the image subject in the image, and the posture skeleton image may describe the posture and shape of the image subject.

[0047] In some embodiments of the present application, the electronic device may estimate the posture of the image subject in the expected first image based on the first text and the first image to obtain a first posture skeleton image.

[0048] For example, taking the first image as an example, the subject of the image is a person. Assuming that the posture of the person in the first image is standing with hands on hips, and the first text is standing with hands raised above the head, the electronic device can estimate the posture of the person based on the posture of the person and the first text, and obtain a first posture skeleton image containing the posture of the person standing with hands raised above the head.

[0049] For example, taking the first image as an example, the dog is in a squatting posture, and the first text is squatting with the left front foot raised. Then, the electronic device can perform posture estimation based on the posture of the dog in the first image and the first text, and obtain a first posture skeleton image including the posture of the dog squatting with the left front foot raised.

[0050] In some embodiments of the present application, Figure 2 ,like Figure 3 As shown, the above step 102a can be specifically implemented through the following steps 102a1 to 102a3.

[0051] Step 102a1: The electronic device obtains first posture feature information of the first text and second posture feature information of the first image.

[0052] In some embodiments of the present application, the electronic device may convert the first text into a text semantic vector, and then extract first posture feature information of the posture of the image subject described by the first text.

[0053] For example, the electronic device may first input the first text (Text) into a Chinese language-vision pair pre-training (Contrastive Language-Image Pre-Training, CLIP) text encoder to obtain a text semantic vector TextEmbedding output by the Chinese CLIP text encoder, and the shape of the text semantic vector TextEmbedding may be 1×128. Thus, the electronic device may extract the first posture feature information of the first text from the text semantic vector TextEmbedding.

[0054] In some embodiments of the present application, the second posture feature information may include, but is not limited to: size feature information and length feature information of each part of the image subject in the first image.

[0055] In some embodiments of the present application, the electronic device may further perform block processing on the first image, and map the obtained blocks into posture feature information of multiple dimensions.

[0056] Exemplarily, the electronic device may perform block processing on the first image ReferenceImageNormed to obtain 1024 16×16×3 blocks, and use a linear layer to map the 1024 16×16×3 blocks into 196 posture feature vectors ReferenceImageToken with a dimension of 128. Thus, the electronic device may extract the second posture feature information of the posture of the image subject in the first image from the posture feature vector ReferenceImageToken.

[0057] Step 102a2: The electronic device determines N key points based on the first posture feature information and the second posture feature information.

[0058] Among them, the above N key points can be used to determine the posture of the image subject after posture adjustment, and N is a positive integer.

[0059] In some embodiments of the present application, the key points are important parts that can represent the posture of the image subject of the first image.

[0060] Exemplarily, the above-mentioned key points may be key points of the human skeleton, such as important parts of the human body such as the shoulder, elbow, wrist, hip, knee, ankle, etc.

[0061] In some embodiments of the present application, the electronic device may include a posture estimation module, so that the electronic device can perform posture estimation based on the first posture feature information and the second posture feature information through the posture estimation module to determine N key points, thereby obtaining a first posture skeleton image.

[0062] In some embodiments of the present application, the above-mentioned posture estimation module may include a variational autoencoder (VAE) model.

[0063] It can be understood that the VAE model is a generative model that can be composed of 3 cross-attention layers, combining the encoder and decoder together to learn efficient representation of data. The VAE model can assume that the hidden variables follow a certain prior distribution (such as a standard normal distribution), and map the input data to the posterior distribution of the hidden variables through the encoder, and then restore the hidden variables to the generated samples through the decoder.

[0064] For example, Figure 4As shown, the electronic device can input the first text into the Chinese CLIP text encoder to obtain the text semantic vector TextEmbedding, and after dividing the first image into blocks to obtain the posture feature vector ReferenceImageToken, it can input the text semantic vector TextEmbedding and the posture feature vector ReferenceImageToken simultaneously into the posture estimation module to obtain N key points output by the posture estimation module.

[0065] For example, after simultaneously inputting the text semantic vector TextEmbedding and the posture feature vector ReferenceImageToken into the posture estimation module, the electronic device can use the text semantic vector TextEmbedding as the K and V of the cross-attention layer, and the posture feature vector ReferenceImageToken as the Q of the cross-attention layer, thereby obtaining the first posture feature information and the second posture feature information through the interaction of QKV in the cross-attention layer.

[0066] Assuming that the characteristic shape of the first posture characteristic information is 1024×128 and the characteristic shape of the second posture characteristic information is also 1024×128, the electronic device can obtain the third posture characteristic information by obtaining the average of the first posture characteristic information and the second posture characteristic information.

[0067] It can be understood that the third posture feature information may be comprehensive posture feature information of the first posture feature information and the second posture feature information.

[0068] In some embodiments of the present application, after obtaining the third posture feature information, the electronic device may map the third posture feature information into N key point coordinates describing the posture of the image subject through a linear mapping layer.

[0069] Exemplarily, the electronic device may map the third posture feature information into 133 key point coordinates describing the posture of the image subject through a linear mapping layer.

[0070] In some embodiments of the present application, before acquiring the first posture characteristic information and the second posture characteristic information, the electronic device may also preprocess the first image to better acquire the second posture characteristic information of the posture of the image subject in the first image.

[0071] Exemplarily, the electronic device may resize the first image to 512×512×3 to adapt to the input size of the posture estimation module.

[0072] Exemplarily, the electronic device may also normalize the pixel values ​​of the first image to a range of -1 to 1 to adapt to the input distribution of the posture estimation module.

[0073] In some embodiments of the present application, the electronic device may perform normalization processing on the first image using the following formula (1).

[0074]

[0075] Wherein, Image may be the original first image; ImageNormed may be the first image after normalization processing. It is understandable that, for each pixel in the first image, the electronic device may calculate the corresponding pixel value of each pixel after normalization processing by the above formula (1).

[0076] Step 102a3: The electronic device connects N key points to obtain a first posture skeleton image.

[0077] In some embodiments of the present application, the electronic device may connect the above-mentioned N key points based on the key point connection rules of the species to which the image subject belongs to obtain a first posture skeleton image.

[0078] Exemplarily, assuming that the image subject of the first image is a person, the electronic device can connect the above-mentioned N key points based on the key point connection rule of the human body to obtain the first posture skeleton image.

[0079] It is understood that the human body key point connection rules generally follow the mutual relationship and connection order between the key points defined in the human body's anatomical structure. For example, the neck key point can be connected to the right shoulder key point or the left shoulder key point; the right shoulder key point can be connected to the right elbow key point; the left shoulder key point can be connected to the left elbow key point; the right elbow key point can be connected to the right wrist key point; and the left elbow key point can be connected to the left wrist key point.

[0080] In this way, since the electronic device can first determine N key points based on the first posture feature information of the image subject posture described in the first text and the second posture feature information of the image subject posture in the first image, the first posture skeleton image can be gradually obtained by predicting each key point, thereby improving the accuracy of adjusting the image posture based on the text.

[0081] Step 102b: The electronic device adjusts the posture of the image subject in the first image based on the first posture skeleton image to obtain a second image.

[0082] In some embodiments of the present application, the electronic device may adjust the posture of the image subject in the first image based on the distribution of key points in the first posture skeleton image to obtain a second image.

[0083] In this way, the posture of the image subject in the image can be directly adjusted according to the description of the desired image subject posture in the text, without the user having to manually process the image multiple times. An image containing the image subject posture that meets the user's needs can be obtained, thereby simplifying the image processing operation process and improving the image processing effect.

[0084] In some embodiments of the present application, Figure 2 ,like Figure 5 As shown, the above step 102b can be specifically implemented through the following steps 102b1 to 102b3.

[0085] Step 102b1, the electronic device receives a first input of a first posture skeleton image by a user.

[0086] Step 102b2: The electronic device adjusts the first posture skeleton image in response to the first input based on the input parameters of the first input to obtain a second posture skeleton image.

[0087] In some embodiments of the present application, the first input may be used to adjust the first posture skeleton image. In other words, the first input may be an adjustment input of the first posture skeleton image.

[0088] It can be understood that the above adjustment input of the first posture skeleton image can be an adjustment input of the position of the key points in the first posture skeleton image, so that the first posture skeleton image is changed by changing the position of the key points to obtain the second posture skeleton image.

[0089] In some embodiments of the present application, the above-mentioned first input includes but is not limited to: the user's touch input of the first posture skeleton image through a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible inputs, or a combination of the above inputs, which can be determined according to actual use requirements and are not limited in the embodiments of the present application. Among them, the specific gesture in the embodiments of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiments of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input.

[0090] In some embodiments of the present application, when the first input is a touch input, the input parameters of the first input may include but are not limited to: the starting position of the first input, the ending position of the first input, the pressing strength of the first input, and the pressing duration of the first input.

[0091] In some embodiments of the present application, when the first input is a voice input, the input parameter of the first input may also include the voice content input by the user.

[0092] Exemplarily, the first input may be a user's dragging input of a key point in the first posture skeleton image, so that the electronic device may move the key point from the starting position of the dragging input to the ending position of the dragging input to obtain a second posture skeleton image.

[0093] Exemplarily, the first input may also be a user pressing input on a key point in the first posture skeleton image, so that the electronic device may increase the distance moved by the key point as the user presses for an increasing period of time to obtain a second posture skeleton image.

[0094] Exemplarily, the first input may also be a voice input, and the user may adjust the left and right corresponding key points in the first posture skeleton image by voice "raise your left hand 45 degrees upward".

[0095] In some embodiments of the present application, after obtaining the first posture skeleton image, the electronic device may display a first interface for adjusting the first posture skeleton image.

[0096] In some embodiments of the present application, the electronic device may display a first control on the screen after obtaining the first posture skeleton image, and the first control may be used to display a first interface so that the user can adjust the first posture skeleton image.

[0097] In some embodiments of the present application, after obtaining the second posture skeleton image, the electronic device may receive a user input to save the posture, and save the posture corresponding to the second posture skeleton image.

[0098] Step 102b3: The electronic device adjusts the posture of the image subject in the first image based on the second posture skeleton image to obtain a second image.

[0099] It should be noted that, for the description of the electronic device adjusting the posture of the image subject in the first image based on the second posture skeleton image to obtain the second image, please refer to the detailed description of step 102b5 below. To avoid repetition, it is not described here.

[0100] In this way, the user can make further adjustments to the first posture skeleton image estimated by the electronic device based on the first text and the first image through the first input, thereby improving the accuracy of the posture adjustment.

[0101] In some embodiments of the present application, the above step 102b may also be specifically implemented through the following step 102b4.

[0102] Step 102b4: The electronic device obtains a second image based on the reference posture image, the first posture skeleton image and the first image.

[0103] In some embodiments of the present application, the reference posture image may be a preset posture skeleton image.

[0104] In some embodiments of the present application, the reference posture image may be a default posture skeleton image stored in the electronic device, a user-defined posture skeleton image, or a posture skeleton image obtained by the electronic device from other devices. This embodiment of the present application is not specifically limited.

[0105] In some embodiments of the present application, a user can select a desired reference posture image from a plurality of reference posture images, so that the electronic device can further adjust the first posture skeleton image according to the reference posture image to obtain a third posture skeleton image. Furthermore, the electronic device can adjust the first image according to the third posture skeleton image to obtain a second image.

[0106] It should be noted that, for the description of the electronic device adjusting the first image according to the third posture skeleton image to obtain the second image, please refer to the detailed description of step 102b5 below. To avoid repetition, it will not be described here.

[0107] In this way, the posture of the image subject in the first image can be adjusted by referring to the posture image and the first posture skeleton image to obtain the second image, thereby improving the adjustment efficiency of the image subject posture.

[0108] In some embodiments of the present application, the above step 102b may also be implemented through the following steps 102b5 to 102b7.

[0109] Step 102b5: The electronic device inputs the first posture skeleton image and the first image into the first model.

[0110] Among them, the above-mentioned first model may include a feature acquisition module and a posture adjustment module.

[0111] In some embodiments of the present application, the first model may be a diffusion model.

[0112] It should be noted that the diffusion model is a type of generative model, and its working principle can be summarized into two stages: forward diffusion process and reverse diffusion process. In the forward diffusion process, the diffusion model can gradually convert data into noise. This process can be regarded as a Markov chain, which destroys the training data by continuously adding Gaussian noise. The reverse diffusion process is to learn how to recover the original data from the noise. This process is achieved through a neural network, which is trained to predict the best estimate of the noise removal at each step.

[0113] In some embodiments of the present application, the above-mentioned feature acquisition module can be used to acquire features required for posture adjustment.

[0114] In some embodiments of the present application, the posture adjustment module may generate an image containing the image subject posture desired by the user based on the features acquired by the feature acquisition module.

[0115] Step 102b6: The electronic device obtains the third posture feature information of the first posture skeleton image, the second posture feature information of the first image, and the background feature information of the first image based on the first posture skeleton image, the first image, and the feature acquisition module.

[0116] In some embodiments of the present application, the feature acquisition module may include an encoder, which may be used for feature extraction.

[0117] It can be understood that the encoder can extract the third posture feature information of the first posture skeleton image, the second posture feature information of the first image, and the background feature information of the first image by compressing the input first posture skeleton image and the first image into a low-dimensional representation.

[0118] In some embodiments of the present application, the third posture feature information may include posture features of bones in the first posture bone image.

[0119] In some embodiments of the present application, the above-mentioned background feature information may include, but is not limited to: color features of the first image, texture features of the first image, shape features of the first image, and spatial relationship features of the first image.

[0120] In some embodiments of the present application, the color feature of the first image may be a global feature, which may describe the distribution and characteristics of the colors in the first image.

[0121] In some embodiments of the present application, the texture features of the first image may be used to describe the texture and texture structure of the image content of the first image.

[0122] In some embodiments of the present application, the shape features of the first image may be used to describe contour features and regional distribution features of the image content in the first image.

[0123] In some embodiments of the present application, the spatial relationship features of the first image may be used to describe the spatial positions and layout relationships between objects in the first image.

[0124] It should be noted that for a detailed description of the second posture characteristic information, please refer to the relevant description in the above step 102a1. To avoid repetition, it will not be described here.

[0125] Step 102b7: The electronic device denoises the random noise image based on the third posture feature information, the second posture feature information, the background feature information and the posture adjustment module to adjust the posture of the image subject in the first image to obtain a second image.

[0126] In some embodiments of the present application, the above-mentioned posture adjustment module may include a cross-attention module, which may include at least one cross-attention layer, so that the electronic device can denoise the random noise image through the cross-attention layer based on the third posture feature information, the second posture feature information and the background feature information to adjust the image subject posture in the first image to obtain the second image.

[0127] For example, Figure 6 As shown, the feature acquisition module may include a posture encoder 61 and a VAE encoder 62, and the posture adjustment module may include a cross-attention module 63. The electronic device may extract the third posture feature information of the first posture skeleton image through the posture encoder 61, and extract the second posture feature information of the first image and the background feature information of the first image through the VAE encoder 62. After splicing the feature information output by the VAE encoder 62 with the noise image, the electronic device may input the feature information together with the third posture feature information into the cross-attention module 63, and obtain the output second image after multiple iterations of the denoising process.

[0128] It can be understood that the output of the first model can be a latent space representation of the second image, and the electronic device can decode it into the image space through the VAE decoder to obtain the final second image.

[0129] It should be noted that, when the electronic device further adjusts the first posture skeleton image, the electronic device can process the adjusted posture skeleton image and the first image according to the above step 102b5 to obtain a second image. For example, the electronic device can process the above second posture skeleton image and the first image according to the above step 102b5 to obtain the second image; or the electronic device can also process the above third posture skeleton image and the first image according to the above step 102b5 to obtain the second image.

[0130] Exemplarily, the first image is an image of a person taken by an electronic device. Assuming that the person's posture in the image of the person has a hunchback phenomenon, the user can select the image of the person and enter the text description of the desired person's posture "upright posture", and then the electronic device can preprocess the image of the person to make its size and numerical distribution suitable for the VAE model; the electronic device can input the preprocessed person image and the first text into the VAE model to estimate the person posture skeleton image corresponding to the "upright posture". It can be understood that after obtaining the estimated person posture skeleton image corresponding to the "upright posture", the user can also further fine-tune the estimated person posture skeleton image corresponding to the "upright posture" through the first input. Finally, the estimated person posture skeleton image corresponding to the "upright posture" and the person image are input into the first model by the electronic device, and the first model can extract background and person feature information from the preprocessed person image, and extract person posture features from the person posture skeleton image corresponding to the estimated "upright posture", and finally generate a person image after the posture is adjusted.

[0131] In this way, the posture of the image subject in the image can be directly adjusted through the first model to obtain an image containing the image subject posture that meets the user's needs, thereby simplifying the image processing operation process and improving the image processing effect.

[0132] In some embodiments of the present application, before the above step 102b5, the image processing method provided by the embodiment of the present application may also include the following steps A to D.

[0133] Step A: The electronic device acquires a first image group.

[0134] The first image group may include a third image of the image subject posture to be adjusted, a target posture skeleton image corresponding to the third image, and a target posture image corresponding to the third image.

[0135] In some embodiments of the present application, the first image group may be one of a plurality of image groups used to train a preset model.

[0136] In some embodiments of the present application, each of the above-mentioned multiple image groups may include an image of the image subject posture to be adjusted, a target posture skeleton image corresponding to the image, and a target posture image corresponding to the image.

[0137] In some embodiments of the present application, the above-mentioned multiple image groups may be generated by the electronic device itself, or may be image groups generated by other devices. This embodiment of the present application does not specifically limit this.

[0138] In some embodiments of the present application, the electronic device may select an image with higher image quality as the first image in the image group to train a preset model to obtain a better training effect.

[0139] Exemplarily, the electronic device may select an image with a resolution greater than 1024x1024 as the third image in the image group to train the preset model.

[0140] In some embodiments of the present application, the third image may be the same as or different from the first image.

[0141] In some embodiments of the present application, the target posture image may be a manually annotated, desired final image.

[0142] In some embodiments of the present application, the above-mentioned target pose skeleton image can be obtained by extracting the key points of the image body through an Effective Whole-body Pose Estimation with Two-stages Distillation (DWPose) model.

[0143] Step B: The electronic device inputs the first image group into a preset model, and obtains a fourth image based on the third image and the target posture skeleton image.

[0144] Among them, the above-mentioned fourth image is obtained by adjusting the image body of the third image based on the target posture skeleton image by a preset model.

[0145] In some embodiments of the present application, the above-mentioned preset models may include a VAE model, a U-shaped network (UNet) model and a pose encoding (PoseEncoder) model.

[0146] In some embodiments of the present application, during the training process of the above-mentioned preset model, the electronic device may input the target posture image and the third image into the VAE model after preprocessing the target posture image and the third image, and obtain the target posture image feature vector TargetImageLatent and the third image feature vector ReferenceImageLatent represented in the latent space respectively. Then, the electronic device may add Gaussian noise to the target posture image feature vector TargetImageLatent and concatenate it with the third image feature vector ReferenceImageLatent to obtain the input vector ImageLatent of the UNet model.

[0147] In addition, the electronic device can also input the target posture skeleton image into the PoseEncoder model to obtain the encoded target posture skeleton image feature vector PoseEmbedding. It can be understood that the target posture skeleton image feature vector PoseEmbedding can be used to represent the semantic information of the target posture.

[0148] The electronic device can then input the vector ImageLatent and the target pose skeleton image feature vector PoseEmbedding into the UNet model, use the vector ImageLatent as Q in the UNet model to provide background and image subject feature information in the third image, and use the target pose skeleton image feature vector PoseEmbedding as K and V in the UNet model to provide pose-related feature information, and finally output the fourth image.

[0149] It is understandable that the background information of the third image, the background information of the fourth image, and the background information of the target posture image are almost the same. That is, when the electronic device generates the fourth image, the background information in the third image is not processed, that is, the electronic device can retain the background information of the third image to the greatest extent.

[0150] Step C: The electronic device calculates a first loss value based on the fourth image and the target posture image.

[0151] Step D: The electronic device iteratively updates the parameters of the preset model according to the first loss value until the preset model converges to obtain a trained first model.

[0152] In some embodiments of the present application, the first loss value may be used to represent the difference between the fourth image and the target pose image. It is understood that the difference between the fourth image and the target pose image may be the difference between the image subject in the fourth image and the image subject in the target pose image.

[0153] In some embodiments of the present application, the electronic device may iteratively update the parameters in the preset model using a back propagation algorithm according to the first loss value until the preset model converges, thereby obtaining a trained first model. In other words, all parameters in the preset model may be updated by back propagation, or some parameters in the preset model may be updated by back propagation, and some parameters may be updated by iteration. The embodiments of the present application are not specifically limited.

[0154] It should be noted that the back propagation algorithm is an optimization algorithm used to train neural network models. It updates the parameters by calculating the gradient of the loss value to the parameters of the neural network model, thereby minimizing the loss value. In other words, the back propagation algorithm is an optimization algorithm based on gradient descent, which can effectively optimize the parameters in the multi-layer neural network model and enable the model to have learning ability.

[0155] In some embodiments of the present application, the parameters of the above-mentioned preset model may include but are not limited to: weight parameters and bias parameters.

[0156] In some embodiments of the present application, training the preset model until convergence can be understood as the first loss value approaching an infinitesimal state.

[0157] In this way, during the training process of the preset model, the electronic device can improve the effect of the first model adjusting the image subject posture based on the text through the loss value between the fourth image and the target posture image.

[0158] Each of the above-mentioned method embodiments, or various possible implementation methods in each method embodiment can be executed separately, or any two or more of them can be executed in combination with each other. The specific implementation method can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0159] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0160] Figure 7 A possible structural diagram of an image processing device involved in an embodiment of the present application is shown. Figure 7 As shown, the image processing device 70 may include: an acquisition module 71 and a processing module 72 .

[0161] Among them, the acquisition module 71 is used to acquire a first image and a first text, the first image includes an image subject, and the first text is used to describe the expected image subject posture; the processing module 72 is used to adjust the image subject posture in the first image according to the first text acquired by the acquisition module 71 to obtain a second image.

[0162] In a possible implementation, the processing module 72 is specifically configured to:

[0163] Based on the first text and the first image, perform image subject pose estimation to obtain a first pose skeleton image;

[0164] Based on the first posture skeleton image, the posture of the image subject in the first image is adjusted to obtain a second image.

[0165] In a possible implementation, the acquisition module 71 is further used to acquire the first posture feature information of the first text and the second posture feature information of the first image;

[0166] The processing module 72 is specifically used for:

[0167] Based on the first posture feature information and the second posture feature information, determine N key points, where the N key points are used to determine the posture of the image subject after posture adjustment, where N is a positive integer;

[0168] Connect N key points to obtain the first posture skeleton image.

[0169] In a possible implementation, the processing module 72 is specifically configured to:

[0170] Inputting the first posture skeleton image and the first image into a first model, the first model comprising a feature acquisition module and a posture adjustment module;

[0171] Based on the first posture skeleton image, the first image and the feature acquisition module, acquire third posture feature information of the first posture skeleton image, second posture feature information of the first image and background feature information of the first image;

[0172] Based on the third posture feature information, the second posture feature information, the background feature information and the posture adjustment module, the random noise image is denoised to adjust the posture of the image subject in the first image to obtain the second image.

[0173] In a possible implementation, the acquisition module 71 is further used to acquire a first image group before the processing module 72 inputs the first posture skeleton image and the first image into the first model to obtain the second image, wherein the first image group includes a third image of the image subject posture to be adjusted, a target posture skeleton image corresponding to the third image, and a target posture image corresponding to the third image;

[0174] The above-mentioned processing module 72 is also used to input the first image group into the preset model, and obtain a fourth image based on the third image and the target posture skeleton image, where the fourth image is obtained by adjusting the image body of the third image based on the target posture skeleton image by the preset model; and, based on the fourth image and the target posture image, calculate the first loss value; and, according to the first loss value, iteratively update the parameters of the preset model until the preset model converges to obtain the trained first model.

[0175] In a possible implementation, the processing module 72 is specifically configured to obtain the second image based on the reference posture image, the first posture skeleton image and the first image.

[0176] An embodiment of the present application provides an image processing device that can directly adjust the posture of the image subject in an image according to the description of the desired image subject posture in the text, without the need for the user to manually process the image multiple times, so as to obtain an image containing the image subject posture that meets the user's needs, thereby simplifying the image processing operation process and improving the image processing effect.

[0177] The image processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0178] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0179] The image processing device provided in the embodiment of the present application can implement the various processes implemented in the above-mentioned image processing method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0180] Alternatively, if Figure 8 As shown, an embodiment of the present application further provides an electronic device 800, including a processor 801 and a memory 802, wherein the memory 802 stores programs or instructions that can be executed on the processor 801, and when the program or instructions are executed by the processor 801, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0181] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0182] Fig. 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0183] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0184] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0185] Among them, the processor 1010 is used to obtain a first image and a first text, the first image includes an image subject, and the first text is used to describe the expected image subject posture; and is used to adjust the image subject posture in the first image according to the first text obtained by the processor 1010 to obtain a second image.

[0186] In a possible implementation manner, the processor 1010 is specifically configured to:

[0187] Based on the first text and the first image, perform image subject pose estimation to obtain a first pose skeleton image;

[0188] Based on the first posture skeleton image, the posture of the image subject in the first image is adjusted to obtain a second image.

[0189] In a possible implementation, the processor 1010 is further configured to obtain first posture feature information of the first text and second posture feature information of the first image;

[0190] The processor 1010 is specifically configured to:

[0191] Based on the first posture feature information and the second posture feature information, determine N key points, where the N key points are used to determine the posture of the image subject after posture adjustment, where N is a positive integer;

[0192] Connect N key points to obtain the first posture skeleton image.

[0193] In a possible implementation manner, the processor 1010 is specifically configured to:

[0194] Inputting the first posture skeleton image and the first image into a first model, the first model comprising a feature acquisition module and a posture adjustment module;

[0195] Based on the first posture skeleton image, the first image and the feature acquisition module, acquire third posture feature information of the first posture skeleton image, second posture feature information of the first image and background feature information of the first image;

[0196] Based on the third posture feature information, the second posture feature information, the background feature information and the posture adjustment module, the random noise image is denoised to adjust the posture of the image subject in the first image to obtain the second image.

[0197] In a possible implementation, the processor 1010 is further configured to obtain a first image group before the processor 1010 inputs the first posture skeleton image and the first image into the first model to obtain the second image, wherein the first image group includes a third image of the image subject posture to be adjusted, a target posture skeleton image corresponding to the third image, and a target posture image corresponding to the third image;

[0198] The above-mentioned processor 1010 is also used to input the first image group into the preset model, and obtain a fourth image based on the third image and the target posture skeleton image, where the fourth image is obtained by adjusting the image body of the third image based on the target posture skeleton image by the preset model; and, based on the fourth image and the target posture image, calculate the first loss value; and, according to the first loss value, iteratively update the parameters of the preset model until the preset model converges to obtain the trained first model.

[0199] In a possible implementation, the processor 1010 is specifically configured to obtain a second image based on a reference posture image, a first posture skeleton image, and a first image.

[0200] An embodiment of the present application provides an electronic device that can directly adjust the posture of the image subject in an image according to the description of the desired image subject posture in the text, without the need for the user to manually process the image multiple times, so as to obtain an image containing the image subject posture that meets the user's needs, thereby simplifying the image processing operation process and improving the image processing effect.

[0201] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0202] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0203] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0204] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0205] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0206] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0207] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0208] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0209] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0210] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0211] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first image and a first text, wherein the first image includes an image subject, and the first text is used to describe an expected posture of the image subject; According to the first text, the posture of the image subject in the first image is adjusted to obtain a second image.

2. The method according to claim 1, characterized in that The step of adjusting the posture of the image subject in the first image according to the first text to obtain the second image includes: Based on the first text and the first image, perform image subject posture estimation to obtain a first posture skeleton image; Based on the first posture skeleton image, the posture of the image subject in the first image is adjusted to obtain a second image.

3. The method according to claim 2, characterized in that The step of estimating the posture of the image subject based on the first text and the first image to obtain a first posture skeleton image includes: Acquire first posture feature information of the first text and second posture feature information of the first image; Based on the first posture feature information and the second posture feature information, determine N key points, wherein the N key points are used to determine the posture of the image subject after posture adjustment, where N is a positive integer; The N key points are connected to obtain the first posture skeleton image.

4. The method according to claim 2, characterized in that: The step of adjusting the posture of the image subject in the first image based on the first posture skeleton image to obtain a second image includes: Inputting the first posture skeleton image and the first image into a first model, wherein the first model includes a feature acquisition module and a posture adjustment module; Based on the first posture skeleton image, the first image and the feature acquisition module, acquire third posture feature information of the first posture skeleton image, second posture feature information of the first image and background feature information of the first image; Based on the third posture feature information, the second posture feature information, the background feature information and the posture adjustment module, the random noise image is denoised to adjust the posture of the image subject in the first image to obtain the second image.

5. The method according to claim 4, characterized in that Before inputting the first posture skeleton image and the first image into the first model to obtain the second image, the method further includes: Acquire a first image group, the first image group comprising a third image of a subject posture to be adjusted, a target posture skeleton image corresponding to the third image, and a target posture image corresponding to the third image; Inputting the first image group into a preset model, and obtaining a fourth image based on the third image and the target posture skeleton image, wherein the fourth image is obtained by adjusting the image body of the third image by the preset model based on the target posture skeleton image; Calculating a first loss value based on the fourth image and the target pose image; According to the first loss value, the parameters of the preset model are iteratively updated until the preset model converges to obtain a trained first model.

6. The method according to claim 2 or 3, characterized in that: The step of adjusting the posture of the image subject in the first image based on the first posture skeleton image to obtain a second image includes: A second image is obtained based on a reference posture image, the first posture skeleton image and the first image.

7. An image processing device, characterized in that: The device comprises: an acquisition module and a processing module; The acquisition module is used to acquire a first image and a first text, wherein the first image includes an image subject, and the first text is used to describe a desired posture of the image subject; The processing module is used to adjust the posture of the image subject in the first image according to the first text acquired by the acquisition module to obtain a second image.

8. The device according to claim 7, characterized in that The processing module is specifically used for: Based on the first text and the first image, perform image subject posture estimation to obtain a first posture skeleton image; Based on the first posture skeleton image, the posture of the image subject in the first image is adjusted to obtain a second image.

9. The device according to claim 8, characterized in that The acquisition module is further used to acquire first posture feature information of the first text and second posture feature information of the first image; The processing module is specifically used for: Based on the first posture feature information and the second posture feature information, determine N key points, wherein the N key points are used to determine the posture of the image subject after posture adjustment, where N is a positive integer; The N key points are connected to obtain the first posture skeleton image.

10. The device according to claim 8, characterized in that The processing module is specifically used for: Inputting the first posture skeleton image and the first image into a first model, wherein the first model includes a feature acquisition module and a posture adjustment module; Based on the first posture skeleton image, the first image and the feature acquisition module, acquire third posture feature information of the first posture skeleton image, second posture feature information of the first image and background feature information of the first image; Based on the third posture feature information, the second posture feature information, the background feature information and the posture adjustment module, the random noise image is denoised to adjust the posture of the image subject in the first image to obtain the second image.

11. The device according to claim 10, characterized in that The acquisition module is further used to acquire a first image group before the processing module inputs the first posture skeleton image and the first image into the first model to obtain the second image, wherein the first image group includes a third image of the image subject posture to be adjusted, a target posture skeleton image corresponding to the third image, and a target posture image corresponding to the third image; The processing module is further used to input the first image group into a preset model, and obtain a fourth image based on the third image and the target posture skeleton image, wherein the fourth image is obtained by adjusting the image body of the third image by the preset model based on the target posture skeleton image; The processing module is further used to calculate a first loss value based on the fourth image and the target posture image; The processing module is also used to iteratively update the parameters of the preset model according to the first loss value until the preset model converges to obtain a trained first model.

12. The device according to claim 8 or 9, characterized in that The processing module is specifically used to obtain a second image based on a reference posture image, the first posture skeleton image and the first image.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 6 are implemented.

14. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 6 are implemented.

15. A computer program product, wherein the computer program product is stored in a storage medium, and wherein the computer program product is executed by at least one processor to implement the steps of the image processing method according to any one of claims 1 to 6.

Images