Image processing method, device and storage medium

Abstract

The present application discloses an image processing method, device and storage medium, which first obtains the head image and its head area and head coordinates in the image of the target object, and obtains the action posture description information, and then obtains the target body action posture image and its corresponding head area and head area coordinates from the action posture library according to the action posture description information and the head orientation of the head image, and then adjusts the head image according to the head area and the head area to obtain a head adjustment image equal to the head area, and then matches the coordinates of the head adjustment image to the head area coordinates to synthesize the head adjustment image and the target body action posture image to obtain a synthesized image. The embodiment of the present application can efficiently generate images that meet the requirements. The embodiment of the present application can be applied to various image generation application scenarios.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device and storage medium. Background Art

[0002] In the current image production process of generating corresponding images based on text content, the common method is to manually draw images based on the text content, or to use a neural network model to generate images based on the text content. Among them, the manually drawn image production method has the problem of low efficiency; when using a neural network model for image generation, it is necessary to rely on the text semantic understanding ability of the neural network model itself to understand the description information of the text content, and then generate the corresponding image based on the semantics of the understood description information. However, when the description information of the text content is relatively complex, the neural network model often cannot accurately understand the semantics of the description information of the text content, and thus easily generates images that do not meet the requirements or have distorted images. Summary of the invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The embodiments of the present application provide an image processing method, device, and storage medium, which can efficiently generate images that meet the requirements.

[0005] On the one hand, an embodiment of the present application provides an image processing method, comprising the following steps:

[0006] Acquire a head image in the image of the target object, as well as the head area and head coordinates of the head image;

[0007] Acquire action posture description information, and acquire a target body action posture image, and a head region area and head region coordinates corresponding to the target body action posture image from an action posture library according to the action posture description information and the head orientation of the head image;

[0008] performing image adjustment on the head image according to the head area and the head region area to obtain a head adjusted image, wherein after the image adjustment, the head coordinates are adjusted to the coordinates of the head adjusted image, and the area of ​​the head adjusted image is equal to the area of ​​the head region;

[0009] The coordinates of the head adjustment image are matched to the head region coordinates to synthesize the head adjustment image and the target body motion posture image to obtain a synthesized image.

[0010] On the other hand, an embodiment of the present application further provides an image processing device, including:

[0011] A first image acquisition unit, used to acquire a head image in the image of the target object, as well as a head area and head coordinates of the head image;

[0012] a second image acquisition unit, configured to acquire action posture description information, and acquire a target body action posture image, and a head region area and head region coordinates corresponding to the target body action posture image from an action posture library according to the action posture description information and the head orientation of the head image;

[0013] an image adjustment unit, configured to perform image adjustment on the head image according to the head area and the head region area to obtain a head adjustment image, wherein after the image adjustment, the head coordinates are adjusted to the coordinates of the head adjustment image, and the area of ​​the head adjustment image is equal to the area of ​​the head region;

[0014] The image synthesis unit is used to match the coordinates of the head adjustment image to the head region coordinates, so as to synthesize the head adjustment image and the target body motion posture image to obtain a synthesized image.

[0015] Optionally, the coordinates of the head adjustment image include the coordinates of the center point of the head, and the head area coordinates include the coordinates of the center point of the head area; and the image synthesis unit is further used for:

[0016] The center point coordinates of the head are aligned with the center point coordinates of the head region to synthesize the head adjustment image and the target body motion posture image to obtain a synthesized image.

[0017] Optionally, the image synthesis unit is further used for:

[0018] Aligning the coordinates of the center point of the head with the coordinates of the center point of the head area;

[0019] Determine whether there is a gap between the head adjustment image and the target body motion posture image after the head center point coordinates are aligned with the head area center point coordinates;

[0020] When there is no gap between the head adjustment image and the target body motion posture image, a combination of the head adjustment image and the target body motion posture image after aligning the head center point coordinates with the head area center point coordinates is used as a composite image.

[0021] Optionally, the coordinates of the head adjustment image further include the coordinates of the head chin point, and the head region coordinates include the coordinates of the head region chin point; and the image synthesis unit is further used for:

[0022] When there is a gap between the head adjustment image and the target body motion posture image, the head chin point coordinates are aligned with the head area chin point coordinates to synthesize the head adjustment image and the target body motion posture image to obtain a synthesized image.

[0023] Optionally, the image processing device further comprises an action posture library construction unit, and the action posture library construction unit is used to:

[0024] Get action posture labels;

[0025] Acquire a body motion posture image according to the motion posture label, and a head region area and head region coordinates corresponding to the body motion posture image;

[0026] The action posture library is constructed according to the action posture label, the body action posture image, and the head region area and head region coordinates corresponding to the body action posture image.

[0027] Optionally, the action posture library construction unit is further used for:

[0028] Acquire a human body action posture image according to the action posture label;

[0029] Performing image semantic segmentation on the human body motion posture image to obtain a body motion posture image and a head region area and head region coordinates corresponding to the body motion posture image.

[0030] Optionally, the action posture library construction unit is further used for:

[0031] Performing image semantic segmentation on the human body action posture image to obtain a head region image and pixel coordinates of the head region image;

[0032] Calculating the area of ​​the head region image and the region coordinates of the head region image according to the pixel coordinates of the head region image;

[0033] Obtaining a body motion posture image according to the human body motion posture image and the head region image;

[0034] The area of ​​the head region image is used as the head region area corresponding to the body action posture image, and the region coordinates of the head region image are used as the head region coordinates corresponding to the body action posture image.

[0035] Optionally, the first image acquisition unit is further used for:

[0036] Obtaining an image of a target object;

[0037] Perform image semantic segmentation on the image image to obtain a head image in the image image and a head area and head coordinates of the head image.

[0038] Optionally, the first image acquisition unit is further used for:

[0039] Performing image semantic segmentation on the image to obtain a head image in the image and pixel coordinates of the head image;

[0040] The head area and the head coordinates of the head image are calculated according to the pixel coordinates of the head image.

[0041] Optionally, the image processing device further includes:

[0042] An information acquisition unit, used to acquire style conversion guidance information and background description information;

[0043] An image generating unit is used to generate a target style image according to the style conversion guiding information, the background description information and the synthesized image.

[0044] Optionally, the image generating unit is further used for:

[0045] Performing style conversion on the synthesized image according to the style conversion guidance information to obtain a style converted image;

[0046] Generate a background image according to the background description information, wherein the background image matches the style of the style conversion image;

[0047] The style conversion image is fused with the background image to obtain a target style image.

[0048] On the other hand, an embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned image processing method when executing the computer program.

[0049] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the above-mentioned image processing method.

[0050] On the other hand, the embodiment of the present application further provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned image processing method.

[0051] The embodiments of the present application include at least the following beneficial effects: firstly, a head image in an image image of a target object, as well as the head area and head coordinates of the head image, and action posture description information are obtained; then, according to the action posture description information and the head orientation of the head image, a target body action posture image, as well as the head region area and head region coordinates corresponding to the target body action posture image are obtained from an action posture library; since the target body action posture image is obtained from the action posture library according to the action posture description information and the head orientation of the head image, the target body action posture image can not only correspond to the action posture description information, but also match the head image, and there will be no problem of conflict between the head orientation and the body orientation; then, the head image is adjusted according to the head area and the head region area to obtain a head adjusted image, wherein, after the image adjustment, The head coordinates are adjusted to the coordinates of the head adjustment image, and the area of ​​the head adjustment image is equal to the area of ​​the head region. By adjusting the head image according to the head area and the head region area, the area of ​​the head adjustment image obtained after adjustment is equal to the area of ​​the head region, which can be beneficial to improving the harmony and coordination when the head adjustment image is synthesized with the target body movement posture image; then, the coordinates of the head adjustment image are matched to the head region coordinates to synthesize the head adjustment image and the target body movement posture image to obtain a synthesized image. By matching the coordinates of the head adjustment image to the head region coordinates, the head adjustment image can be accurately positioned at the head region corresponding to the target body movement posture image, thereby avoiding a gap between the head adjustment image and the target body movement posture image, and further improving the coordination and rationality of the synthesized image. During image processing, the head image in the target object's image image is synthesized with the target body motion posture image obtained from the motion posture library according to the motion posture description information. Therefore, the obtained synthesized image not only has the required motion posture, but also has the appearance characteristics of the target object, thereby increasing the sense of participation and fun of the target object in making personalized images by himself. In addition, since neither manual drawing nor image generation using a neural network model is required during the entire image processing process, images that meet the requirements can be efficiently generated.

[0052] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained through the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0054] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0055] Figure 2 is a schematic diagram of another implementation environment provided by an embodiment of the present application;

[0056] Figure 3 is a flow chart of an image processing method provided by an embodiment of the present application;

[0057] Figure 4 It is a schematic diagram of the overall architecture of a BiSenet model provided in an embodiment of the present application;

[0058] Figure 5 It is a detailed network architecture diagram of a BiSenet model provided in an embodiment of the present application;

[0059] Figure 6 It is a structural diagram of an ARM module provided in an embodiment of the present application;

[0060] Figure 7 It is a structural diagram of a feature fusion module provided in an embodiment of the present application;

[0061] Figure 8 This is a schematic diagram of the effect of extracting a user's head image using the BiSenet model provided in an embodiment of the present application;

[0062] Fig. 9 is a schematic diagram of a basic three-dimensional human body provided in an embodiment of the present application;

[0063] Fig.10 is a schematic diagram of adjusting parameters of a basic three-dimensional human body provided in an embodiment of the present application;

[0064] Fig.11 It is a schematic diagram of a publicly available and free human body picture with corresponding action postures provided in an embodiment of the present application;

[0065] Fig.12 is a schematic diagram of simulating the actions of a virtual human provided in an embodiment of the present application;

[0066] Fig.13 It is a schematic diagram of a process of image adjustment of a head image provided by an embodiment of the present application;

[0067] Fig.14is a schematic diagram of a head adjustment image and a target body movement posture image provided by an embodiment of the present application;

[0068] Fig.15 is a schematic diagram of matching the coordinates of a head adjustment image to the coordinates of a head region provided by an embodiment of the present application;

[0069] Fig.16 is a schematic diagram of a training image for fine-tuning and training a Stable Diffusion model provided in an embodiment of the present application;

[0070] Fig.17 is a schematic diagram of a process for generating a target style image provided by an embodiment of the present application;

[0071] Fig.18 It is a system flow diagram of an image processing method provided by an embodiment of the present application;

[0072] Fig.19 is a processing flow chart of an image synthesis part of adaptive proportional adjustment provided by an embodiment of the present application;

[0073] Fig. 20 This is a specific flow chart of generating a customized novel martial arts image in ink-and-wash style provided by an embodiment of the present application;

[0074] Fig.21 is a specific flow chart of performing image synthesis based on a user's image provided by an embodiment of the present application;

[0075] Fig. 22 is a detailed flow chart of an image processing method provided by a specific example of the present application;

[0076] Fig.23 is a schematic diagram of an image processing device provided in an embodiment of the present application;

[0077] Fig.24 A partial structural block diagram of a user terminal provided in an embodiment of the present application;

[0078] Fig.25 A partial structural block diagram of the server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The present application is further described below in conjunction with the accompanying drawings and specific embodiments. The described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0080] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0081] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0082] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0083] 1) Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction system, mechatronics, etc. Among them, the pre-trained model is also called the large model or basic model. After fine-tuning, it can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0084] 2) Stable Diffusion model is a variant of the Diffusion Model. The purpose of the Diffusion Model is to eliminate the Gaussian noise that is continuously applied to the training images. It can be regarded as a series of denoising autoencoders. The Stable Diffusion Model consists of 3 parts: Variational Autoencoder (VAE), U-Net, and a text encoder.

[0085] 3) U-Net is an algorithm that uses a fully convolutional network for semantic segmentation. Its structure is a symmetrical U-shaped structure consisting of a compression path and an expansion path.

[0086] 4) DreamBooth is a fine-tuning technology that can be applied to the fine-tuning training of personalized text-to-image models (such as the StableDiffusion model). Given several subject images, by customizing the model training, realistic images of the same subject can be generated during the inference phase.

[0087] 5) Bilateral Real-time Semantic Segmentation Network (BiSenet), which can perform low-cost and high-precision semantic segmentation of human images. The BiSenet model designs a spatial path with a small step size to retain spatial position information to generate high-resolution feature maps, and also designs a semantic path with a fast downsampling strategy to obtain an objective perceptual range. In addition, on top of the spatial path and the semantic path, the BiSenet model also introduces a new feature fusion module to fuse the feature maps of the two to achieve a balance between speed and accuracy.

[0088] When generating images corresponding to text content, the commonly used image generation method is to manually draw images according to the text content, or to generate images according to the text content using a neural network model. For example, in the process of producing novel illustrations on the current common online novel platforms, a large number of illustrators are generally chosen to perform manual painting. The illustrator will first read the novel, and according to the description of the character in the specified relevant fragment, complete the painting of the character image to obtain the character image illustration, and then insert the character image illustration into the novel. This image production method can make the generated image and style very consistent with the design requirements, but this artificial method of generating illustrations will result in extremely slow efficiency of the entire image generation, especially when the number of illustrations in the novel is relatively large, which will cause the entire illustration painting project to be time-consuming and the cost to double. Moreover, each illustrator will have different subjective understandings of the same paragraph of text description, which will lead to different illustrators' understanding of the same character's painting image, which will lead to different illustrations of the same character in the same novel. In the case of using a neural network model to generate images according to text content, for example, a Stable Diffusion model can be used to generate a character image with a specified action or posture. By inputting the text describing the action posture of the character into the Stable Diffusion model, the Stable Diffusion model is allowed to understand the action posture of the character by itself, and then the image of the corresponding action posture is generated according to the understanding of the Stable Diffusion model. This image production method requires the use of the text semantic understanding ability of the Stable Diffusion model itself. Although it can complete the image generation of relatively simple action postures to a certain extent, when the action posture to be generated is relatively complex, it is often easy to generate images that do not meet the requirements or have distorted images because the Stable Diffusion model cannot accurately understand the semantics of the description information of the complex text content of the action posture, which will affect the user's perception.

[0089] In the related art, in addition to the image production methods in the above two related technologies, the Stable Diffusion model fine-tuned and trained by plug-in modules such as lora and controlnet can also be used to generate the required character image. In this way, you can choose to use the controlnet plug-in module to control the action posture of the character image generated by the Stable Diffusion model, and use plug-in modules such as lora and DreamBooth to fine-tune the image performance and style performance of the Stable Diffusion model to generate the required character image. Although plug-in modules such as lora and controlnet that can be personalized for training can solve the generation effect of action posture and user image features to a certain extent, plug-in modules such as lora and controlnet will conflict with the original image generation ability of the Stable Diffusion model, which will affect the generalization of the entire model. Moreover, due to the addition of plug-in modules such as lora and controlnet, the stylized processing of the image by the Stable Diffusion model will be affected, which will cause the style of the generated image to differ from the target style, and then cause the final generated image to fail to meet the user's design requirements.

[0090] In order to efficiently generate an image that meets the requirements, the embodiments of the present application provide an image processing method, an image processing device, an electronic device, a computer-readable storage medium, and a computer program product. First, a head image in an image of a target object, as well as the head area and head coordinates of the head image, are obtained, and action posture description information is obtained. Then, based on the action posture description information and the head orientation of the head image, a target body action posture image, as well as the head region area and head region coordinates corresponding to the target body action posture image are obtained from an action posture library. Since the target body action posture image is obtained from the action posture library based on the action posture description information and the head orientation of the head image, the target body action posture image can not only correspond to the action posture description information, but also match the head image, and there will be no problem of conflict between the head orientation and the body orientation. Then, the head image is calibrated based on the head area and the head region area. Image adjustment is performed to obtain a head adjustment image, wherein after the image adjustment, the head coordinates are adjusted to the coordinates of the head adjustment image, and the area of ​​the head adjustment image is equal to the area of ​​the head region. By performing image adjustment on the head image according to the head area and the head region area, the area of ​​the head adjustment image obtained after adjustment is equal to the area of ​​the head region, which can be beneficial to improving the harmony and coordination when the head adjustment image is synthesized with the target body movement posture image; then, the coordinates of the head adjustment image are matched to the head region coordinates to synthesize the head adjustment image and the target body movement posture image to obtain a synthesized image. By matching the coordinates of the head adjustment image to the head region coordinates, the head adjustment image can be accurately positioned at the head region corresponding to the target body movement posture image, thereby avoiding a gap between the head adjustment image and the target body movement posture image, and further improving the coordination and rationality of the synthesized image. During image processing, the head image in the target object's image image is synthesized with the target body motion posture image obtained from the motion posture library according to the motion posture description information. Therefore, the obtained synthesized image not only has the required motion posture, but also has the appearance characteristics of the target object, thereby increasing the sense of participation and fun of the target object in making personalized images by himself. In addition, since neither manual drawing nor image generation using a neural network model is required during the entire image processing process, images that meet the requirements can be efficiently generated.

[0091] Figure 1 Schematic diagram of an implementation environment provided by an embodiment of the present application. Figure 1, the implementation environment includes a first user terminal 101 and a first server 102. The first user terminal 101 and the first server 102 are directly or indirectly connected through wired or wireless communication, wherein the first user terminal 101 and the first server 102 can be nodes in the blockchain, which is not specifically limited in this embodiment.

[0092] The first user terminal 101 may include, but is not limited to, smart phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft and other smart devices. Optionally, the first user terminal 101 may be installed with an image processing client, which may generate a stylized image with the user's appearance characteristics.

[0093] The first server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network) and big data and artificial intelligence platforms.

[0094] In one embodiment, the first server 102 has at least functions such as image acquisition, image adjustment, and image synthesis. For example, the first server 102 can obtain the head image in the image uploaded by the user, as well as the head area and head coordinates of the head image, and can obtain the action posture description information input by the user, and then obtain the target body action posture image, and the head area and head area coordinates corresponding to the target body action posture image from the local action posture library according to the action posture description information and the head orientation of the head image, and then adjust the head image according to the head area and the head area to obtain the head adjustment image, wherein after the image adjustment, the head coordinates are adjusted to the coordinates of the head adjustment image, and the area of ​​the head adjustment image is equal to the area of ​​the head area, and then the coordinates of the head adjustment image are matched to the head area coordinates to synthesize the head adjustment image and the target body action posture image to obtain a synthesized image; after obtaining the synthesized image, the first server 102 can send the synthesized image to the first user terminal 101 for image display, or can further perform style conversion on the synthesized image to obtain the style image desired by the user, and then send the style image to the first user terminal 101 for image display.

[0095] Reference Figure 1As shown, in an application scenario, it is assumed that the first user terminal 101 is a smart phone, and the first user terminal 101 is installed with an image processing client. In the process of the user making an image through the image processing client in the first user terminal 101 to obtain a style image with the user's own appearance characteristics, in response to the user inputting an image making instruction through the image processing client, the first user terminal 101 sends the image making instruction to the first server 102; in response to receiving the image making instruction, the first server 102 obtains the image uploaded by the user, the action posture description information, the style conversion guidance information and the background description information, and then performs image semantic segmentation on the image image to obtain the head image in the image image and the head area and head coordinates of the head image, and then obtains the target body action posture image and the head area and head area coordinates corresponding to the target body action posture image from the local action posture library according to the action posture description information and the head orientation of the head image; at this time, the first server 102 obtains the head area and head area according to the head area and head area The first server 102 performs image adjustment on the head image according to the domain area to obtain the head adjustment image, wherein, after the image adjustment, the head coordinates will be adjusted to the coordinates of the head adjustment image, and the area of ​​the head adjustment image will be equal to the area of ​​the head region, and then, the first server 102 matches the coordinates of the head adjustment image to the head region coordinates to synthesize the head adjustment image and the target body movement posture image to obtain a synthesized image; after obtaining the synthesized image, the first server 102 performs style conversion on the synthesized image according to the style conversion guidance information to obtain a style conversion image, and generates a background image matching the style of the style conversion image according to the background description information, and then fuses the style conversion image with the background image to obtain a target style image having the user's own appearance characteristics; after obtaining the target style image, the first server 102 sends the target style image to the first user terminal 101 for image display.

[0096] Figure 2 Schematic diagram of another implementation environment provided by the embodiment of the present application. Figure 2 , the implementation environment includes a second user terminal 201, a second server 202 and a database server 203. The second user terminal 201 and the second server 202 are directly or indirectly connected through wired or wireless communication, and the database server 203 can be set separately, or integrated on the second server 202, or integrated on other devices, wherein the second user terminal 201, the second server 202 and the database server 203 can all be nodes in the blockchain, and this embodiment does not specifically limit this.

[0097] The second user terminal 201 may include, but is not limited to, smart phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft and other smart devices. Optionally, the second user terminal 201 may be installed with an image processing client, which may generate a stylized image with the user's appearance characteristics.

[0098] The second server 202 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN networks, and big data and artificial intelligence platforms. Among them, a motion posture library may be pre-deployed in the database server 203, and a plurality of different body motion posture images, as well as the head area area and head area coordinates corresponding to each body motion posture image are pre-stored in the motion posture library. The second server 202 may obtain a target body motion posture image from the plurality of body motion posture images in the database server 203, as well as the head area area and head area coordinates corresponding to the target body motion posture image, so that image synthesis that meets user needs may be performed according to the target body motion posture image.

[0099] In one embodiment, the second server 202 has at least functions such as image acquisition, image adjustment, and image synthesis. For example, the second server 202 can obtain the head image in the image uploaded by the user, as well as the head area and head coordinates of the head image, and can obtain the action posture description information input by the user, and then obtain the corresponding target body action posture image, and the head area and head area coordinates corresponding to the target body action posture image from the database server 203 according to the action posture description information and the head orientation of the head image, and then perform image adjustment on the head image according to the head area and the head area to obtain the head adjustment image, wherein after the image adjustment, the head coordinates are adjusted to the coordinates of the head adjustment image, and the area of ​​the head adjustment image is equal to the area of ​​the head area, and then the coordinates of the head adjustment image are matched to the head area coordinates to perform image synthesis on the head adjustment image and the target body action posture image to obtain a synthesized image; after obtaining the synthesized image, the second server 202 can send the synthesized image to the second user terminal 201 for image display, or can further perform style conversion on the synthesized image to obtain the style image desired by the user, and then send the style image to the second user terminal 201 for image display.

[0100] Reference Figure 2As shown, in another application scenario, it is assumed that the second user terminal 201 is a computer, and the second user terminal 201 is installed with an image processing client. In the process of the user making an image through the image processing client in the second user terminal 201 to obtain a style image with the user's own appearance characteristics, in response to the user inputting an image making instruction through the image processing client, the second user terminal 201 sends the image making instruction to the second server 202; in response to receiving the image making instruction, the second server 202 obtains the image image, action posture description information, style conversion guidance information and background description information uploaded by the user, and then performs image semantic segmentation on the image image to obtain the head image in the image image and the head area and head coordinates of the head image, and then sends the action posture description information and the head orientation information of the head image to the database server 203; in response to receiving the action posture description information and the head orientation information of the head image, the database server 203 obtains the target body action posture image, the head area and head area coordinates corresponding to the target body action posture image from the action posture library according to the action posture description information and the head orientation information of the head image, and then sends the target body action posture image and the head area and head area corresponding to the target body action posture image to the database server 203. The head region coordinates are sent to the second server 202; in response to receiving the target body motion posture image and the head region area and head region coordinates corresponding to the target body motion posture image, the second server 202 adjusts the head image according to the head area and the head region area to obtain a head adjustment image, wherein after the image adjustment, the head coordinates will be adjusted to the coordinates of the head adjustment image, and the area of ​​the head adjustment image will be equal to the head region area, and then, the second server 202 matches the coordinates of the head adjustment image to the head region coordinates to perform image synthesis on the head adjustment image and the target body motion posture image to obtain a synthesized image; after obtaining the synthesized image, the second server 202 performs style conversion on the synthesized image according to the style conversion guidance information to obtain a style conversion image, and generates a background image matching the style conversion image according to the background description information, and then fuses the style conversion image with the background image to obtain a target style image having the user's own appearance characteristics; after obtaining the target style image, the second server 202 sends the target style image to the second user terminal 201 for image display.

[0101] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on the attribute information or attribute information set of the target object (such as a user, etc.) and other data related to the characteristics of the target object, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain the attribute information of the target object, the separate permission or separate consent of the target object will be obtained through a pop-up window or jump to a confirmation page. After clearly obtaining the separate permission or separate consent of the target object, the relevant data of the target object necessary for the normal operation of the embodiment of the present application is obtained.

[0102] Figure 3 : is a flowchart of an image processing method provided in an embodiment of the present application. The image processing method can be executed by a server, or by a user terminal, or by a user terminal and a server together. In the embodiment of the present application, the method is described by taking the method executed by a server as an example. Figure 3 , the image processing method includes but is not limited to steps 310 to 340.

[0103] Step 310: Acquire the head image in the image of the target object, as well as the head area and head coordinates of the head image.

[0104] In one embodiment, the image of the target object may be an image including only the head, or may be an image including the head and the body, which is not specifically limited here. In addition, the image of the target object may be pre-uploaded by the target object, for example, the image of the target object may be an account avatar uploaded by the target object when registering an account, or may be uploaded by the target object when the execution of the image processing method is triggered, for example, a selfie image uploaded by the target object when the execution of the image processing method is triggered by the image processing client, which is not specifically limited here. In one embodiment, the target object may be the user who triggers the execution of the image processing method.

[0105] In one embodiment, the head area of ​​the head image refers to the area of ​​the head image in the image image, and the head coordinates of the head image refer to the coordinates of the head image in the image image, wherein the head coordinates of the head image may include the contour coordinates of the head image, and may also include the coordinates of key positions in the head image, such as the head center point coordinates, the head chin point coordinates or the eye position coordinates, etc., which are not specifically limited here.

[0106] In one embodiment, in the process of obtaining the head image in the image of the target object and the head area and head coordinates of the head image, the image image of the target object can be first obtained, and then the image image can be semantically segmented to obtain the head image in the image image and the head area and head coordinates of the head image. Among them, when obtaining the image image of the target object, the image image of the target object can be obtained by obtaining the account avatar uploaded by the target object when registering an account, or by obtaining the selfie image currently uploaded by the target object. The image image of the target object can be appropriately selected according to the actual application situation, and is not specifically limited here. In addition, when performing image semantic segmentation on the image image to obtain the head image in the image image and the head area and head coordinates of the head image, the image image can be semantically segmented to obtain the head image in the image image and the pixel coordinates of the head image, and then the head area and head coordinates of the head image can be calculated based on the pixel coordinates of the head image.

[0107] In one embodiment, the pixel coordinates of the head image may include the coordinates of all the pixels in the head image. Therefore, when calculating the head area of ​​the head image according to the pixel coordinates of the head image, the number of all the pixels included in the head image may be counted according to the pixel coordinates, and then the head area of ​​the head image may be calculated according to the number of the pixels. In addition, when the head coordinates of the head image are coordinates of different positions, there may be different implementation methods in the process of calculating the head coordinates of the head image according to the pixel coordinates of the head image, which may be appropriately selected according to the actual situation and is not specifically limited here. For example, when the head coordinates of a head image include the contour coordinates of the head image, the contour pixels of the head image can be first calculated based on the size relationship of all pixel coordinates, and then the pixel coordinates of the contour pixels are used as the head coordinates of the head image; for another example, when the head coordinates of a head image include the head center point coordinates of the head image, the center pixel of the head image can be first calculated based on the size relationship of all pixel coordinates, and then the pixel coordinates of the center pixel are used as the head coordinates of the head image; for another example, when the head coordinates of a head image include the chin point coordinates of the head image, the chin pixel of the head image can be first calculated based on the size relationship of all pixel coordinates, and then the pixel coordinates of the chin pixel are used as the head coordinates of the head image.

[0108] In one embodiment, during the process of performing semantic segmentation on the image, a semantic segmentation model may be called to perform semantic segmentation on the head image in the image. For example, a BiSenet model may be called to perform detection calculations on the image to obtain the head image in the image and its head area and head coordinates. Then, the head image is cropped based on the calculated results, and the cropped head image and the calculated head area and head coordinates are stored in a preset action posture library. Figure 4 As shown, Figure 4 This is a schematic diagram of the overall architecture of the BiSenet model. Figure 4 It can be seen that the BiSenet model mainly includes two parts: the spatial path part 410 and the semantic path part 420. Among them, the Spatial Path part 410 can solve the problem of spatial information loss after channel reduction. At the same time, the Spatial Path part 410 also has a certain degree of downsampling, which can reduce the size of internal processing features, thereby improving the overall calculation speed and reducing processing time. In addition, the ContextPath part 420 can solve the problem of the perception range of image content, and the ContextPath part 420 also has a certain degree of downsampling. Through the joint action of these two parts, the calculation time of the overall model can be effectively reduced to achieve a real-time effect.

[0109] refer to Figure 5 As shown, Figure 5 This is a detailed network architecture diagram of the BiSenet model. Figure 5In the figure, the BiSenet model includes a Spatial Path part 510 and a Context Path part 520, wherein the Context Path part 520 has an Attention Refinment Module (ARM) 521, and the output results of the Spatial Path part 510 and the Context Path part 520 are input into a Feature Fusion Module (FFM) 530 for feature fusion to obtain fused features, and then the fused features are upsampled to obtain image semantic segmentation results. Among them, the BiSenet model uses global pooling to improve the perception range and reduce the number of pooling branches at the same time. The Context Path part 520 in the BiSenet model is implemented using a U-shaped architecture, wherein the Context Path part 520 first downsamples the input features to reduce the size of the input features to improve the perception range of the BiSenet model. In the Context Path part 520, there are a first downsampling branch with a sampling rate of 1 / 16 and a second downsampling branch with a sampling rate of 1 / 32, and both the first downsampling branch and the second downsampling branch are connected to an ARM module 521, and these two ARM modules 521 are used to enhance the perception range of the BiSenet model for image content.

[0110] refer to Figure 6 As shown, Figure 6 This is a schematic diagram of the structure of the ARM module. Figure 6 In the embodiment, the ARM module 521 includes a feature branch 610 and an attention branch 620, wherein the input features of the attention branch 620 are resized features. In the attention branch 620, the resized input features are first globally pooled, and then the globally pooled input features are sequentially subjected to convolution processing with a convolution kernel of 1x1 and regularization processing, and then the sigmoid activation function is used to normalize all the calculated feature pixel values, wherein the result obtained after normalization is the attention correlation, and then the attention correlation is matrix multiplied with the original features of the feature branch 610 that have not been resized to obtain the output result of the ARM module 521.

[0111] At the level of feature representation, the semantics of the features output by the Spatial Path part and the Context Path part are different. The spatial information captured by the Spatial Path part encodes most of the rich detail information, so the output features of the Spatial Path part are low-level, while the output features of the Context Path part mainly encode context information, so the output features of the Context Path part are high-level. Therefore, a feature fusion module can be used to fuse the output features of the Spatial Path part and the output features of the Context Path part. Figure 7 As shown, Figure 7 It is a structural diagram of the feature fusion module. Figure 7 In the embodiment, the feature fusion module 530 will first concatenate the output features of the Spatial Path part and the output features of the Context Path part on different channels to obtain the concatenated features on different channels, and then successively perform convolution processing, regularization processing and activation processing to fuse the concatenated features on different channels, so that the low-level features of the Spatial Path part and the high-level features of the Context Path part can be fully fused to obtain a unified feature, and then input the unified feature into a feature branch 710 and an attention branch 720, wherein, in the attention branch 720, the unified feature will first be subjected to global pooling, 1x1 convolution processing, activation processing and 1x1 convolution processing in sequence, and the calculated eigenvalues ​​will be normalized using the sigmoid activation function to obtain the attention relevance of the feature, and then, the attention relevance of the feature will be matrix multiplied with the unified feature in the feature branch 710, and the result after the matrix multiplication will be matrix added with the unified feature in the feature branch 710 to obtain the output result of the feature fusion module.

[0112] After the feature fusion processing of the feature fusion module, the BiSenet model has extracted the detection information of each semantic element in the entire image, and then can upsample the fused features according to the size of the input original image, so that the size of the output feature is consistent with the input original image. After the upsampling is completed, a specific semantic judgment can be made for each pixel. By aggregating the pixels of the same category, the distribution of different semantic elements in the entire image can be calculated, so that the boundary between different elements can be located, and then the required semantic elements can be extracted. At the same time, the relevant pixel coordinates of the extracted area and the total area of ​​the extracted area can also be obtained. Reference Figure 8 As shown, Figure 8The following is an example of how to use the BiSenet model to extract the head image from the user's image. Figure 8 In the figure, the left picture is the user's image 810. After the user's image 810 is input into the BiSenet model for image semantic segmentation, a head image 820 as shown in the right picture can be obtained. At the same time, the area a of the head image 820 and the coordinate value of the head image 820 in the coordinate system can be obtained.

[0113] Step 320: Acquire the action posture description information, and acquire the target body action posture image, and the head region area and head region coordinates corresponding to the target body action posture image from the action posture library according to the action posture description information and the head orientation of the head image.

[0114] In one embodiment, the action posture description information may be text information with action posture description input by the target object himself, or may be text information with action posture description obtained by extracting content from a text uploaded by the target object (such as an article, a novel chapter, etc.), or may be text information with action posture description selected by the target object through an option, etc., and is not specifically limited here.

[0115] In one embodiment, after obtaining the head image in the image of the target object, the head orientation of the head image can be detected (for example, the head is facing left, the head is facing middle, etc.), and then the corresponding target body action posture image and the head area and head area coordinates corresponding thereto are obtained from a pre-prepared action posture library according to the obtained action posture description information and the detected head orientation, so that in the subsequent steps, the head adjustment image and the target body action posture image can be synthesized according to the head area and head coordinates of the head image and the head area and head area coordinates corresponding to the target body action posture image to obtain a synthesized image having the appearance characteristics of the target object. Since the target body action posture image is obtained in the action posture library according to the action posture description information and the head orientation of the head image, the target body action posture image can not only correspond to the action posture description information, but also match the head image, and there will be no problem of conflict between the head orientation and the body orientation.

[0116] In one embodiment, when detecting the head orientation of a head image, the nose position (or mouth position), the number of ears and the ear positions in the head image may be detected first, and then the head orientation of the head image may be determined based on the nose position (or mouth position), the number of ears and the ear positions. For example, when determining the head orientation of a head image based on the nose position (or mouth position), the number of ears and the ear positions, if the number of ears is 2 and the nose position (or mouth position) is between the two ear positions, the head orientation of the head image may be considered to be toward the center; if the number of ears is 1 and the nose position (or mouth position) is to the left of the ear position, the head orientation of the head image may be considered to be toward the left; if the number of ears is 1 and the nose position (or mouth position) is to the right of the ear position, the head orientation of the head image may be considered to be toward the right.

[0117] In one embodiment, the action posture library may store various action posture labels, body action posture images, and head area areas and head area coordinates corresponding to the body action posture images, wherein the action posture labels correspond to the body action posture images one by one. Therefore, in the process of obtaining the target body action posture image and the head area area and head area coordinates corresponding to it from the action posture library according to the action posture description information and the head orientation of the head image, the similarity between the action posture description information and each action posture label may be calculated first, and then the body action posture images corresponding to the multiple action posture labels with higher similarity may be used as candidate body action posture images, and then the target body action posture image whose body orientation matches the head orientation may be selected from the candidate body action posture images according to the head orientation of the head image, and the head area area and head area coordinates corresponding to the target body action posture image may be obtained from the action posture library. Since the body orientation of the target body action posture image matches the head orientation of the head image, and the target body action posture image is screened out from candidate body action posture images corresponding to multiple action posture labels having high similarity to the action posture description information, the target body action posture image can not only correspond to the action posture description information, but also match the head image, and there will be no problem of conflict between the head orientation and the body orientation.

[0118] In one embodiment, when constructing an action posture library, the action posture label can be first obtained, and then the body action posture image, the head area and the head area coordinates corresponding to the body action posture image are obtained according to the action posture label, and then the action posture library is constructed according to the action posture label, the body action posture image, the head area and the head area coordinates corresponding to the body action posture image. Wherein, when obtaining the action posture label, the action posture type that needs to be included in the entire action posture library can be determined first, and then the text description information including these action posture types (such as the action posture description content of the character in the article, novel, etc.) can be obtained, and then the text description information is segmented and the part of speech analysis is processed, and the description words for the human body action posture in the text description information are extracted. At this time, the extracted description words can be further subjected to word frequency statistics, and the target description words with higher word frequency can be screened out according to the number of occurrences of these description words obtained after the statistics, and then the screened target description words are used as action posture labels. In addition, in the process of obtaining the body motion posture image and its corresponding head region area and head region coordinates according to the motion posture label, the human body motion posture image can be first obtained according to the motion posture label, and then the human body motion posture image can be semantically segmented to obtain the body motion posture image and its corresponding head region area and head region coordinates.

[0119] In one embodiment, when obtaining a human body motion posture image according to the motion posture label, there may be a variety of different implementation methods, which may be appropriately selected according to the actual application situation, and are not specifically limited here. In one embodiment, the human body motion posture image may be obtained by making a three-dimensional human body. For example, the following may be made first: Fig. 9 The basic 3D human body is shown in FIG. 1 , and then various parameters of the basic 3D human body are adjusted according to the action posture label (such as Fig.10 As shown in the figure, a target 3D human body with a corresponding action posture is obtained, and then the target 3D human body is saved to obtain a human body action posture image. In addition, a human body action posture image can also be obtained by collecting human body pictures of related actions. For example, a public and free human body picture with a corresponding action posture can be obtained from the Internet according to the action posture label (such as Fig.11 Then, the action posture label is manually or model-recognized to bind the corresponding human body image to obtain the human body action posture image. In addition, the human body action posture image can also be obtained by simulating the action of a virtual human. For example, a virtual human can be generated by a virtual human tool, and then the action of the virtual human matching the action posture label can be simulated (as shown in FIG. Fig.12As shown), then the virtual human that has performed the action simulation is screenshotted to obtain the corresponding human action posture image. It should be noted that when the human action posture image is obtained by collecting human body pictures of relevant actions, since the action posture label needs to be bound to the corresponding human body picture manually or by model recognition, it will cause label binding error to a certain extent. In order to reduce the label binding error and improve the accuracy of obtaining the human action posture image according to the action posture label, in one embodiment, it is also possible to request to obtain a more accurate human action posture image from a relevant professional action human model website according to the action posture label, or the label binding error can be reduced by increasing the data ratio of the human action posture image obtained by making a three-dimensional human body and the human action posture image obtained by simulating the action of the virtual human.

[0120] In one embodiment, in the process of performing image semantic segmentation on a human body motion posture image to obtain a body motion posture image and the head region area and head region coordinates corresponding to the body motion posture image, the human body motion posture image can be firstly subjected to image semantic segmentation to obtain a head region image and pixel point coordinates of the head region image, and then the area of ​​the head region image and the region coordinates of the head region image are calculated based on the pixel point coordinates of the head region image, and then the body motion posture image is obtained based on the human body motion posture image and the head region image, and then the area of ​​the head region image is used as the head region area corresponding to the body motion posture image, and the region coordinates of the head region image are used as the head region coordinates corresponding to the body motion posture image. Wherein, when obtaining the body motion posture image based on the human body motion posture image and the head region image, the position of the head region image can be determined in the human body motion posture image, and then the head region image can be image segmented in the human body motion posture image based on the position of the head region image, and after the head region image in the human body motion posture image is segmented, the remaining part is the body motion posture image. In addition, in the process of performing image semantic segmentation on human action posture images, a semantic segmentation model such as the BiSenet model may also be called to perform image semantic segmentation on the head region in the human action posture images. It should be noted that the principle and process of calling a semantic segmentation model such as the BiSenet model to perform image semantic segmentation on the head region in the human action posture images can refer to the principle and process of calling a semantic segmentation model such as the BiSenet model to perform image semantic segmentation on the head image in the image, which will not be repeated here to avoid redundant content.

[0121] In one embodiment, the pixel coordinates of the head region image may include the coordinates of all the pixels in the head region image. Therefore, when calculating the area of ​​the head region image according to the pixel coordinates of the head region image, the number of all the pixels included in the head region image may be counted according to the pixel coordinates, and then the area of ​​the head region image may be calculated according to the number of the pixels. In addition, when the region coordinates of the head region image are coordinates of different positions, there may be different implementation methods in the process of calculating the region coordinates of the head region image according to the pixel coordinates of the head region image, which may be appropriately selected according to the actual situation and is not specifically limited here. For example, when the regional coordinates of the head region image include the contour coordinates of the head region image, the contour pixels of the head region image can be first calculated according to the size relationship of all pixel coordinates, and then the pixel coordinates of the contour pixels are used as the regional coordinates of the head region image; for another example, when the regional coordinates of the head region image include the center point coordinates of the head region of the head region image, the center pixel point of the head region of the head region image can be first calculated according to the size relationship of all pixel coordinates, and then the pixel coordinates of the center pixel point of the head region are used as the regional coordinates of the head region image; for another example, when the regional coordinates of the head region image include the chin point coordinates of the head region of the head region image, the chin pixel point of the head region of the head region image can be first calculated according to the size relationship of all pixel coordinates, and then the pixel coordinates of the chin pixel point of the head region are used as the regional coordinates of the head region image.

[0122] Step 330: performing image adjustment on the head image according to the head area and the head region area to obtain a head adjusted image, wherein after the image adjustment, the head coordinates are adjusted to the coordinates of the head adjusted image, and the area of ​​the head adjusted image is equal to the area of ​​the head region.

[0123] In one embodiment, after obtaining the head image and the target body action posture image in the image image of the target object, since the size of the head image is not necessarily compatible with the size of the target body action posture image, if the head image and the target body action posture image are directly subjected to image synthesis, the resulting synthetic image may have a situation where the head is large and the body is small, or the head is small and the body is large, which will seriously affect the coordination and rationality of the synthetic image. In order to solve this problem, the head image can be image-adjusted according to the head area and the head region area to obtain a head-adjusted image whose area is equal to the head region area. By image-adjusting the head image according to the head area and the head region area, so that the area of ​​the head-adjusted image obtained after the adjustment is equal to the head region area, it can be beneficial to improve the harmony and coordination when the head-adjusted image and the target body action posture image are subjected to image synthesis.

[0124] In one embodiment, when adjusting the head image according to the head area and the head region area, if the head area is larger than the head region area, the head image needs to be reduced so that the reduced head area (i.e., the area of ​​the head adjustment image) is equal to the head region area; if the head area is smaller than the head region area, the head image needs to be enlarged so that the enlarged head area (i.e., the area of ​​the head adjustment image) is equal to the head region area. In addition, if the head area is equal to the head region area, there is no need to adjust the head image, and the head coordinates and head area of ​​the head image remain unchanged.

[0125] The process of adjusting the head image according to the head area and the head region area is described in detail below using a specific example.

[0126] For example Fig.13 As shown, Fig.13 1 is a schematic diagram of a process of adjusting a head image provided by an embodiment of the present application. Fig.13 In the figure, assuming that the head area of ​​the head image 1310 is S1, and the head region area corresponding to the target body movement posture image 1320 is S2, then when adjusting the head image according to the head area and the head region area, first compare the head area S1 with the head region area S2. Assuming that the head area S1 is larger than the head region area S2, the head region area S2 is used as a reference to reduce the head image 1310, so that the head area S1 is reduced accordingly. When the reduced head area S1 is equal to the head region area S2, the head adjustment image 1330 can be obtained. At this time, the original head coordinates of the head image 1310 will be adjusted to the current coordinates of the head adjustment image 1330.

[0127] Step 340: Match the coordinates of the head adjustment image to the head region coordinates to synthesize the head adjustment image and the target body motion posture image to obtain a synthesized image.

[0128] In one embodiment, after obtaining the head adjustment image, since the area of ​​the head adjustment image is equal to the area of ​​the head region, the coordinates of the head adjustment image can be matched to the coordinates of the head region, so as to synthesize the head adjustment image and the target body motion posture image, and obtain a synthetic image that has both the required motion posture and the appearance characteristics of the target object. By matching the coordinates of the head adjustment image to the coordinates of the head region, the head adjustment image can be accurately positioned at the head region corresponding to the target body motion posture image, thereby avoiding a gap between the head adjustment image and the target body motion posture image, and further improving the coordination and rationality of the synthetic image.

[0129] In one embodiment, when the coordinates of the head adjustment image include the coordinates of the center point of the head, and the coordinates of the head region include the coordinates of the center point of the head region, in the process of matching the coordinates of the head adjustment image to the coordinates of the head region to synthesize the head adjustment image and the target body motion posture image to obtain a synthesized image, the coordinates of the center point of the head can be aligned with the coordinates of the center point of the head region to synthesize the head adjustment image and the target body motion posture image to obtain a synthesized image. Wherein, when aligning the coordinates of the center point of the head with the coordinates of the center point of the head region to synthesize the head adjustment image and the target body motion posture image, the head adjustment image can be first copied to the layer where the target body motion posture image is located, so that the head adjustment image and the target body motion posture image are in the same coordinate system, and then the coordinates of the center point of the head of the head adjustment image are aligned to the coordinates of the center point of the head region of the target body motion posture image. At this time, the head adjustment image will move to the head region position of the target body motion posture image as the coordinates of the center point of the head are aligned with the coordinates of the center point of the head region, thereby realizing image synthesis between the head adjustment image and the target body motion posture image.

[0130] It should be noted that, since there is a difference between the shape of the head image and the shape of the head corresponding to the target body motion posture image, even if the head image is adjusted to a head adjustment image with an area equal to that of the head region, and the coordinates of the head adjustment image are matched to the coordinates of the head region corresponding to the target body motion posture image, there may still be a gap between the head adjustment image and the target body motion posture image, which will lead to disharmony between the entire head and the human body in the composite image. Therefore, after aligning the head center point coordinates with the head region center point coordinates, it is also necessary to determine whether there is a gap between the head adjustment image and the target body motion posture image after the head center point coordinates are aligned with the head region center point coordinates. If there is no gap between the head adjustment image and the target body motion posture image, it means that the head adjustment image and the target body motion posture image after the head center point coordinates are aligned with the head region center point coordinates are harmonious and reasonable. At this time, the combination of the head adjustment image and the target body motion posture image after the head center point coordinates are aligned with the head region center point coordinates can be used as a composite image. In addition, if there is a gap between the head adjustment image and the target body motion posture image, it means that the head adjustment image and the target body motion posture image need to be further adjusted so that there is no gap between the adjusted head adjustment image and the target body motion posture image. At this time, the head chin point coordinates of the head adjustment image can be aligned with the head area chin point coordinates of the target body motion posture image. In this way, the head chin position of the head adjustment image will be connected with the neck position of the target body motion posture image. In other words, there is no gap between the head adjustment image and the target body motion posture image at this time, so the combination of the head adjustment image and the target body motion posture image at this time can be used as a composite image.

[0131] The following describes in detail a process of achieving image synthesis of the head adjustment image and the target body motion posture image by matching the coordinates of the head adjustment image to the head region coordinates, using a specific example.

[0132] For example Fig.14 As shown in FIG. 1 , assuming that the coordinates of the head adjustment image include the head center point coordinates A11 and the head chin point coordinates A12, and the head region coordinates corresponding to the target body motion posture image include the head region center point coordinates A21 and the head region chin point coordinates A22, then when matching the coordinates of the head adjustment image to the head region coordinates to achieve image synthesis of the head adjustment image and the target body motion posture image, the head center point coordinates A11 can be matched to the head region center point coordinates A21 first. At this time, the synthesis result between the head adjustment image and the target body motion posture image will be as shown in FIG. Fig.15 As shown in the left figure or the middle figure. Fig.15In the figure on the left, after matching the head center point coordinate A11 to the head area center point coordinate A21, the head chin point coordinate A12 is below the head area chin point coordinate A22, that is, the head adjustment image can cover the head area chin point corresponding to the target body motion posture image, and will not cause a gap between the head adjustment image and the target body motion posture image. At this time, the image is a composite image obtained by synthesizing the head adjustment image and the target body motion posture image. Fig.15 In the middle figure, after the head center point coordinate A11 is matched to the head region center point coordinate A21, the head chin point coordinate A12 is above the head region chin point coordinate A22, that is, there is a gap between the head adjustment image and the target body action posture image. Fig.15 As shown in the figure on the right, the head chin point coordinate A12 can be adjusted to match the head area chin point coordinate A22. Since the head chin point coordinate A12 overlaps with the head area chin point coordinate A22, there will be no gap between the head adjustment image and the target body movement posture image. At this time, the image is a composite image obtained by synthesizing the head adjustment image and the target body movement posture image.

[0133] In this embodiment, by using the image processing method including the previous steps 310 to 340, the head image in the image image of the target object, as well as the head area and head coordinates of the head image, and the action posture description information are first obtained, and then the target body action posture image, as well as the head area and head area coordinates corresponding to the target body action posture image are obtained from the action posture library according to the action posture description information and the head orientation of the head image. Since the target body action posture image is obtained from the action posture library according to the action posture description information and the head orientation of the head image, the target body action posture image can not only correspond to the action posture description information, but also match the head image, and there will be no problem of conflict between the head orientation and the body orientation. Then, the head image is adjusted according to the head area and the head area to obtain a head adjusted image, wherein the adjustment is performed. After the row image is adjusted, the head coordinates are adjusted to the coordinates of the head adjustment image, and the area of ​​the head adjustment image is equal to the area of ​​the head region. By adjusting the head image according to the head area and the head region area, the area of ​​the head adjustment image obtained after adjustment is equal to the area of ​​the head region, which can be beneficial to improving the harmony and coordination when the head adjustment image is synthesized with the target body movement posture image; then, the coordinates of the head adjustment image are matched to the head region coordinates to synthesize the head adjustment image and the target body movement posture image to obtain a synthesized image. By matching the coordinates of the head adjustment image to the head region coordinates, the head adjustment image can be accurately positioned at the head region corresponding to the target body movement posture image, thereby avoiding a gap between the head adjustment image and the target body movement posture image, and further improving the coordination and rationality of the synthesized image. During image processing, the head image in the target object's image image is synthesized with the target body motion posture image obtained from the motion posture library according to the motion posture description information. Therefore, the obtained synthesized image not only has the required motion posture, but also has the appearance characteristics of the target object, thereby increasing the sense of participation and fun of the target object in making personalized images by himself. In addition, since neither manual drawing nor image generation using a neural network model is required during the entire image processing process, images that meet the requirements can be efficiently generated.

[0134] In one embodiment, after obtaining the composite image, the target style image that meets the requirements of the target object can be generated in combination with the composite image according to the acquired style conversion guidance information and background description information. In the process of generating the target style image according to the style conversion guidance information, the background description information and the composite image, the style conversion image can be firstly style-converted according to the style conversion guidance information to obtain the style conversion image, and then the background image can be generated according to the background description information, wherein the background image matches the style conversion image, and then the style conversion image and the background image are image-fused to obtain the target style image.

[0135] In one embodiment, the style conversion guide information and the background description information may be obtained by the server at the same time as the action posture description information is obtained, that is, when the server obtains the action posture description information, the style conversion guide information and the background description information may be obtained at the same time; in addition, the style conversion guide information and the background description information may also be obtained by the server when the target object triggers the execution of generating the target style image, that is, when the target object triggers the execution of style conversion of the composite image and provides the style conversion guide information and the background description information to the server, the server obtains the style conversion guide information and the background description information. In addition, similar to the action posture description information, the style conversion guide information and the background description information may also be text information input by the target object itself, or may be text information obtained by extracting the content of the text uploaded by the target object, or may be text information selected by the target object through options, etc., which are not specifically limited here.

[0136] In one embodiment, when performing style conversion on the synthetic image according to the style conversion guidance information, the style conversion model may be called to perform style conversion on the synthetic image according to the style conversion guidance information. In addition, when generating a background image that matches the style of the style conversion image according to the background description information, the style conversion model may be called to generate the background image according to the background description information, or another trained image generation model may be called to generate a background image that matches the style of the style conversion image according to the background description information, which is not specifically limited here.

[0137] In one embodiment, the style conversion guidance information is a text fragment or prompt information provided to start the style conversion model. The style conversion guidance information can be a word, a phrase, a sentence, a paragraph, or a whole article, which is not specifically limited here. After the style conversion model receives the style conversion guidance information, the style conversion model can perform style conversion on the synthesized image according to the style conversion guidance information, so that the style of the obtained style conversion image can conform to the style type described in the style conversion guidance information (for example, ink style type, stick figure style type, future technology style type, etc.).

[0138] In one embodiment, the style conversion model can be an image generation model, such as a StableDiffusion model. When it is necessary to convert a synthetic image into a specific style (such as an ink painting style), the Stable Diffusion model can be converted into a specific style using training data of the corresponding style (such as ink painting style training images) in advance. When training the Stable Diffusion model, the DreamBooth training strategy can be used to use a certain amount of training data of the corresponding style (such as ink painting style training images) to fine-tune the entire StableDiffusion model as a whole. Fig.16 As shown, Fig.16 An ink-and-wash style training image for fine-tuning the ink-and-wash style training of the StableDiffusion model is given as an example. Fig.16 Fine-tuning the Stable Diffusion model with a certain number of ink-style training images can effectively improve the generation effect of the Stable Diffusion model for ink-style images. It should be noted that DreamBooth can fine-tune the existing pre-trained style transfer model to achieve the generation of images of different scenes of the same subject through style transfer guidance information control.

[0139] The following is a detailed description of the process of generating a target style image based on style transfer guidance information, background description information and a synthesized image using a specific example.

[0140] Reference Fig.17 As shown, Fig.17 It is a flowchart of generating a target style image based on style transfer guidance information, background description information and synthetic image. Fig.17In the process, a certain amount of ink painting image data 1710 (e.g., about 100 images) is first used to perform a style transfer model 1720 (e.g., Stable Diffusion model) is fine-tuned and trained. After the fine-tuning training of the style transfer model 1720 is completed, when a composite image 1730 having the user's appearance characteristics is obtained, the composite image 1730 can be input into the style transfer model 1720, so that the style transfer model 1720 performs style conversion on the composite image 1730 according to the style conversion guidance information input by the user for guiding the style transfer model 1720 to perform ink style conversion, and obtains a style conversion image 1740 with ink style and the user's appearance characteristics. At this time, background description information can also be input into the style transfer model 1720, so that the style transfer model 1720 generates a background image with the same ink style as the style conversion image 1740 according to the background description information. Then, the style conversion image 1740 and the background image can be fused to obtain a target style image 1750 with both an ink style background and the user's appearance characteristics.

[0141] The image processing method provided in the embodiment of the present application is described in detail below with specific examples.

[0142] Reference Fig.18 As shown, Fig.18 is a system flow chart of an image processing method provided by a specific example. Fig.18 In the embodiment, the image processing method may include two parts: an image synthesis part 1810 with adaptive scaling and a style conversion part 1820.

[0143] In the adaptive proportional adjustment image synthesis part 1810, the input information is the action posture description information (such as the action posture description information input by the user or the action posture description information of the selected novel character image) and the user's own image (such as the image currently input by the user or the image pre-uploaded by the user). After receiving the action posture description information and the user's own image, a synthetic image with the specified action posture and the user's own image characteristics can be generated using the action posture description information and the user's own image.

[0144] Before generating a synthetic image, a motion posture library can be prepared in advance, and various motion posture labels can be collected, and then various corresponding human motion posture images can be obtained according to these motion posture labels, wherein these human motion posture images can be obtained by making a three-dimensional human body, by simulating virtual human motion, or by obtaining human motion posture images disclosed on the Internet, and can be appropriately selected according to the actual application situation, and are not specifically limited here. After obtaining these human motion posture images, these human motion posture images can be resized so that the sizes of all human motion posture images can be unified, and then for each human motion posture image, the semantic segmentation model can be called to segment the head and the body, and the area and coordinates of the head can be calculated to obtain the body motion posture image and its corresponding head area area, head area coordinates, and then all body motion posture images and their corresponding head area area, head area coordinates are saved in the motion posture library.

[0145] After receiving the action posture description information and the user's own image, the semantic segmentation model can be called to perform head detection and segmentation on the user's own image, and the area and coordinates of the head can be calculated to obtain the user's own head image and its head area and head coordinates; in addition, according to the action posture description information and the head orientation of the user's own head image, a suitable target body action posture image and its corresponding head area and head area coordinates can be screened out from the action posture library; then, according to the head area and the head area, the user's own head image can be adaptively adjusted in proportion to obtain a head-adjusted image, wherein, after the adaptively adjusted image is performed, the head coordinates of the user's own head image will be adjusted to the coordinates of the head adjustment image, and the area of ​​the head adjustment image will be equal to the head area; then, the coordinates of the head adjustment image can be matched to the head area coordinates to synthesize the head adjustment image and the target body action posture image to obtain a synthesized image with the user's image and in accordance with the action posture description information.

[0146] In the style conversion part 1820, the style conversion can be performed on the composite image output by the image synthesis part 1810 with adaptive scale adjustment. The operation of the style conversion part 1820 is mainly performed by the style conversion model. Before calling the style conversion model to perform style conversion on the composite image, the style conversion model can be pre-trained with style customization using the training data of the corresponding style, so that the style conversion model can fully learn the representation ability of the style. After receiving the composite image output by the image synthesis part 1810 with adaptive scale adjustment, the composite image can be style converted according to the style conversion guidance information to obtain a target style image that meets the style requirements of the user and has the user's appearance characteristics, and at the same time meets the action posture description information. The style conversion guidance information can be the description information input by the user to guide the style conversion model to perform a specific style conversion on the composite image, or it can be the style category information to be converted selected by the user, which is not specifically limited here.

[0147] In this example, the user's image is integrated into the final generated target style image by using an image synthesis method with adaptive proportion adjustment for the head, so that the user only needs to upload one image of his or her own image to complete the final image generation. This method not only gets rid of the traditional reliance on user appearance image data when simply using the Stable Diffusion model, but also does not require secondary fine-tuning of the model, thereby avoiding conflicts with the original style concept parameters of the model, and thus speeding up the processing speed of the entire image synthesis, thereby improving the user experience.

[0148] Reference Fig.19 As shown, Fig.19 This is a processing flow chart of the image synthesis part of adaptive ratio adjustment. Fig.19 In the embodiment, the image synthesis part of the adaptive proportional adjustment may include two sub-parts: an action posture library construction sub-part 1910 and an adaptive adjustment image synthesis sub-part 1920 .

[0149] In the action posture library construction sub-section 1910, various action posture labels can be collected first, and based on these action posture labels, various corresponding human action posture images can be obtained by making a three-dimensional human body, simulating virtual human actions, or obtaining human action posture pictures publicly available on the Internet. Then, the obtained human action posture images are resized so that the sizes of all human action posture images can be unified. Then, for each human action posture image, the semantic segmentation model is called to segment the head and body, and the area and coordinates of the head are calculated to obtain the body action posture image and its corresponding head area area, head area coordinates. At this time, all body action posture images and their corresponding head area area, head area coordinates are saved in the action posture library.

[0150] In the adaptive adjustment image synthesis sub-section 1920, after receiving the image and action posture description information input by the user, the semantic segmentation model is first called to perform image semantic segmentation on the image of the user to obtain the user's head image and its head area and head coordinates; at the same time, according to the action posture description information and the head orientation of the user's head image, a suitable target body action posture image and its corresponding head area and head area coordinates are screened from the action posture library; then, the user's head image and the target body action posture image are adaptively adjusted and synthesized to obtain a synthetic image with the user's image and in accordance with the action posture description information. In the process of adaptively adjusting and synthesizing the user's head image and the target body action posture image, the user's head image can be adaptively adjusted according to the head area and the head area to obtain a head adjustment image, wherein after the adaptive adjustment, the head coordinates of the user's head image will be adjusted to the coordinates of the head adjustment image, and the area of ​​the head adjustment image will be equal to the area of ​​the head area, and then, the coordinates of the head adjustment image are matched to the head area coordinates to perform image synthesis on the head adjustment image and the target body action posture image.

[0151] Reference Fig. 20 As shown, in Fig. 20 In this paper, the specific process of the entire image processing method is explained by taking the generation of customized martial arts images in ink painting style as an example. Fig. 20In the process shown, when the user wants to generate a customized novel illustration of a martial arts image in ink style for a certain content description in the novel text, the user can input a self-portrait image through the image processing client, and provide this section of the novel content description to the image processing client by uploading or selecting an option. At this time, the image processing client will send the image image and this section of the novel content description to the server; after the server receives the image image and this section of the novel content description, the server can call the semantic segmentation model 2010 to perform head detection and segmentation on the image image, and calculate the area and coordinates of the head to obtain the user's own head image and its head area and head coordinates; at the same time, the server can also detect action posture words for this section of the novel content description, and obtain action posture prompt words by detecting the entity words and relationship words in this section of the novel content description, and then retrieve the corresponding action posture description information in the pre-set action posture prompt word library according to the action posture prompt word, and then select the appropriate target body action of the novel martial arts image from the action posture library according to the action posture description information and the head orientation of the user's own head image. Posture image and its corresponding head region area, head region coordinates; at this time, the server can perform adaptive proportional image adjustment on the user's own head image according to the head area and the head region area, and obtain a head adjustment image with an area equal to the head region area, and match the coordinates of the head adjustment image to the head region coordinates, so as to synthesize the head adjustment image and the target body movement posture image to obtain a synthetic image of a novel martial arts image with the user's appearance characteristics; after obtaining the synthetic image, the server can call the style conversion model 2020 to perform style conversion on the synthetic image according to the ink style conversion guidance information selected by the user (for example, the ink style information pre-selected by the user or the description information for ink style conversion currently input by the user), and obtain an ink style novel martial arts image image (ie, style conversion image); at this time, the server can also call the style conversion model 2020 to generate an ink style background image according to the background description information selected by the user, and then fuse the ink style novel martial arts image image and the background image to obtain the customized ink style novel martial arts image required by the user The novel illustration (ie, the target style image) is required.

[0152] When the user uploads the novel content description to the image processing client, for example Fig.21As shown, after the image processing client sends the user's image and the novel content description to the server, the server can first call the semantic segmentation model 2110 to perform image semantic segmentation on the image to obtain the user's own head image and its head area and head coordinates; then, the server selects the corresponding action posture description information according to the novel content description, and determines the type of body action posture to be selected according to the action posture description information and the head orientation of the user's own head image, and then selects a suitable target body action posture image and its corresponding head area and head area coordinates in the action posture library 2120 according to the body action posture type; at this time, the server can adaptively adjust and synthesize the user's own head image and the target body action posture image to obtain a synthetic image with the user's image and conforming to the action posture description information. Among them, in the process of adaptively adjusting and synthesizing the head image and the target body movement posture image, the server can first perform adaptive proportional image adjustment on the head image according to the head area and the head region area to obtain a head adjustment image, wherein after performing the adaptive proportional image adjustment, the head coordinates of the head image will be adjusted to the coordinates of the head adjustment image, and the area of ​​the head adjustment image will be equal to the area of ​​the head region. At this time, the server will match the coordinates of the head adjustment image to the head region coordinates to perform image synthesis of the head adjustment image and the target body movement posture image.

[0153] Reference Fig. 22 As shown, Fig. 22 This is a detailed flowchart of an image processing method provided by a specific example. The image processing method can be executed by a server, or by a user terminal, or by a user terminal and a server together. In this specific example, the method is executed by a server as an example. Fig. 22 In the embodiment, the image processing method may include but is not limited to steps 2201 to 2217.

[0154] Step 2201: Acquire action posture labels, and acquire human action posture images according to the action posture labels.

[0155] Step 2202: performing image semantic segmentation on the human body action posture image to obtain a head region image and pixel coordinates of the head region image;

[0156] Step 2203: Calculate the area of ​​the head region image and the region coordinates of the head region image according to the pixel coordinates of the head region image;

[0157] Step 2204: obtaining a body motion posture image according to the human body motion posture image and the head region image;

[0158] Step 2205: Use the area of ​​the head region image as the head region area corresponding to the body motion posture image, and use the region coordinates of the head region image as the head region coordinates corresponding to the body motion posture image, and then construct a motion posture library based on the motion posture label, the body motion posture image and its corresponding head region area and head region coordinates.

[0159] Step 2206: Acquire the image of the target object, and perform image semantic segmentation on the image to obtain the head image and the pixel coordinates of the head image in the image.

[0160] Step 2207: Calculate the head area and head coordinates of the head image based on the pixel coordinates of the head image.

[0161] Step 2208: Obtain the action posture description information, and obtain the target body action posture image, as well as the head region area and head region coordinates corresponding to the target body action posture image from the action posture library according to the action posture description information and the head orientation of the head image.

[0162] Step 2209: Perform image adjustment on the head image according to the head area and the head region area to obtain a head adjusted image, wherein after the image adjustment, the head coordinates are adjusted to the coordinates of the head adjusted image, and the area of ​​the head adjusted image is equal to the area of ​​the head region.

[0163] In one embodiment, the coordinates of the head adjustment image may include the coordinates of the head center point and the coordinates of the head chin point, and the head region coordinates include the coordinates of the head region center point and the coordinates of the head region chin point.

[0164] Step 2210: Align the coordinates of the center point of the head with the coordinates of the center point of the head area.

[0165] Step 2211: Determine whether there is a gap between the head adjustment image after the head center point coordinates are aligned with the head area center point coordinates and the target body movement posture image. If not, execute step 2212; if so, execute step 2213.

[0166] Step 2212: The combination of the head adjustment image after the head center point coordinates are aligned with the head area center point coordinates and the target body action posture image is used as a composite image, and then step 2214 is executed.

[0167] Step 2213: Align the coordinates of the chin point of the head with the coordinates of the chin point of the head region to synthesize the head adjustment image and the target body motion posture image to obtain a synthesized image, and then execute step 2214.

[0168] Step 2214: Obtain style transfer guidance information and background description information.

[0169] Step 2215: Perform style conversion on the synthesized image according to the style conversion guidance information to obtain a style converted image.

[0170] Step 2216: Generate a background image based on the background description information, wherein the background image matches the style of the style transfer image.

[0171] Step 2217: Fuse the style conversion image with the background image to obtain a target style image.

[0172] In this embodiment, through the image processing method of steps 2201 to 2217, the head image in the image image of the target object, the head area and the head coordinates of the head image are first obtained, and the action posture description information is obtained. Then, according to the action posture description information and the head orientation of the head image, the target body action posture image, and the head region area and the head region coordinates corresponding to the target body action posture image are obtained from the action posture library. Since the target body action posture image is obtained from the action posture library according to the action posture description information and the head orientation of the head image, the target body action posture image can not only correspond to the action posture description information, but also match the head image, and there will be no problem of conflict between the head orientation and the body orientation. Then, the head image is adjusted according to the head area and the head region area to obtain a head adjusted image, wherein After image adjustment, the head coordinates are adjusted to the coordinates of the head adjustment image, and the area of ​​the head adjustment image is equal to the area of ​​the head region. By adjusting the head image according to the head area and the head region area, the area of ​​the head adjustment image obtained after adjustment is equal to the area of ​​the head region, which can be beneficial to improving the harmony and coordination when the head adjustment image and the target body movement posture image are synthesized; then, the coordinates of the head adjustment image are matched to the head region coordinates to synthesize the head adjustment image and the target body movement posture image to obtain a synthesized image. By matching the coordinates of the head adjustment image to the head region coordinates, the head adjustment image can be accurately positioned at the head region corresponding to the target body movement posture image, avoiding a gap between the head adjustment image and the target body movement posture image, thereby improving the coordination and rationality of the synthesized image. During image processing, the head image in the target object's image image is synthesized with the target body motion posture image obtained from the motion posture library according to the motion posture description information. Therefore, the obtained synthesized image not only has the required motion posture, but also has the appearance characteristics of the target object, thereby increasing the sense of participation and fun of the target object in making personalized images by himself. In addition, since neither manual drawing nor image generation using a neural network model is required during the entire image processing process, images that meet the requirements can be efficiently generated.

[0173] The following are some practical examples to illustrate the application scenarios of the embodiments of the present application.

[0174] It should be noted that the image processing method provided in the embodiment of the present application can be applied to different application scenarios such as novel illustration generation, customized comic generation or game character production on an online novel platform. The following will be explained using the novel illustration generation scenario, customized comic generation scenario and game character production scenario on an online novel platform as examples.

[0175] Scene 1

[0176] The image processing method provided in the embodiment of the present application can be applied to the novel illustration generation scenario of the online novel platform. For example, when the reader of the online novel platform reads a key chapter or a storyline and hopes to be able to match the novel character illustration with the reader's own appearance characteristics to improve the reading experience, the reader can trigger the generation operation for the novel character illustration. At this time, the online novel platform can obtain the reader's image (for example, the image uploaded by the reader when registering an account or the image uploaded by the reader when triggering the illustration generation operation) and the novel chapter content selected by the reader for which the novel character illustration needs to be generated, and obtain the style conversion guidance information and background description information determined by the reader, and then perform image semantic segmentation on the reader's image to obtain the reader's head image and the head area and head coordinates of the head image, and then determine the action posture description information of the key character according to the novel chapter content selected by the reader, and obtain the target body action posture image and the head area and head area coordinates corresponding to the target body action posture image from a pre-prepared action posture library according to the action posture description information and the head orientation of the head image. ; At this time, the online novel platform can adjust the head image according to the head area and the head region area to obtain a head adjustment image, wherein, after the image adjustment, the head coordinates will be adjusted to the coordinates of the head adjustment image, and the area of ​​the head adjustment image will be equal to the area of ​​the head region, and then, the online novel platform matches the coordinates of the head adjustment image to the head region coordinates to synthesize the head adjustment image and the target body movement posture image to obtain a synthesized image; after obtaining the synthesized image, the online novel platform performs style conversion on the synthesized image according to the style conversion guidance information to obtain a style conversion image, and generates a background image matching the style conversion image according to the background description information, and then fuses the style conversion image with the background image to obtain a target style image with the reader's own appearance characteristics; after obtaining the target style image, the online novel platform adds the target style image as a novel illustration to the novel chapter content selected by the reader, and displays the novel chapter content and the target style image to the reader together, so that the reader can more intuitively feel the image of the character in the text when reading the novel chapter content, thereby improving the overall reading experience.

[0177] Scene 2

[0178] The image processing method provided in the embodiment of the present application can also be applied to customized comic generation scenarios. For example, when a user wants to use a comic generator to create a customized comic strip with himself as the protagonist, the user can upload an image of his own image and the story description content that the user wants to the comic generator; after receiving the image image and story description content uploaded by the user, the comic generator can perform image semantic segmentation on the image of the user to obtain the user's head image and the head area and head coordinates of the head image, and then determine the action posture description information according to the story description content uploaded by the user, and obtain the target from a pre-prepared action posture library according to the action posture description information and the head orientation of the head image. body action posture image, and the head area area and head area coordinates corresponding to the target body action posture image; at this time, the comic generator can adjust the head image according to the head area and the head area area to obtain the head adjustment image, wherein after the image adjustment, the head coordinates will be adjusted to the coordinates of the head adjustment image, and the area of ​​the head adjustment image will be equal to the head area, and then, the comic generator matches the coordinates of the head adjustment image to the head area coordinates to synthesize the head adjustment image and the target body action posture image to obtain a synthesized image with the user's own appearance characteristics, and at this time, by combining the lines module of the comic generator, a customized comic strip with the user as the protagonist can be automatically generated. This method can greatly reduce the difficulty of users creating comics, thereby enhancing the user's creative interest. In addition, after obtaining a customized comic strip with the user himself as the protagonist, it is also possible to obtain style conversion guidance information determined by the user, and then perform style conversion on each composite image in the customized comic strip according to the style conversion guidance information to obtain customized comic strips of different styles. For example, a customized comic strip with a two-dimensional character style can be converted into a customized comic strip with a three-dimensional character style, or a customized comic strip with a modern style can be converted into a customized comic strip with an ancient style, etc., thereby improving the user's flexibility in creating comics.

[0179] Scene 3

[0180] The image processing method provided in the embodiment of the present application can also be applied to game character production scenarios. For example, when a player is performing a game character generation operation, the player can trigger the generation of a personalized game character. When the player chooses to generate a personalized game character, the game server can provide the player with different character types (such as attacking character types, auxiliary character types, etc.) and their associated character descriptions. After the player selects a target character type among these character types, the game server can obtain the player's own image (such as the image uploaded by the player when registering a game account or the image uploaded by the player when choosing to generate a personalized game character), and then perform image semantic segmentation on the player's image to obtain the player's head image and the head area and head coordinates of the head image. Then, based on the character description of the target character type selected by the player and the head orientation of the head image, select the player from the pre-prepared action posture library. Obtain the target character's body action posture image (i.e., the target body action posture image) and its corresponding head region area and head region coordinates; at this time, the game server can adjust the head image according to the head area and the head region area to obtain a head adjustment image, wherein, after the image adjustment, the head coordinates will be adjusted to the coordinates of the head adjustment image, and the area of ​​the head adjustment image will be equal to the head region area, and then, the game server matches the coordinates of the head adjustment image to the head region coordinates to perform image synthesis on the head adjustment image and the target character's body action posture image to obtain a character image with the player's own appearance characteristics, and then, the game server can generate a corresponding game character based on the character image. Since the player's game character in the game has the player's own appearance characteristics, the player's sense of participation in the game can be improved, thereby improving the player's gaming experience.

[0181] It is to be understood that, although the steps in the above-mentioned various flow charts are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in the present embodiment, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above-mentioned flow charts can include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0182] Reference Fig.23 The present application also discloses an image processing device. The image processing device 2300 can implement the image processing method in the previous embodiment. The image processing device 2300 includes:

[0183] The first image acquisition unit 2310 is used to acquire the head image, head area and head coordinates of the head image in the image of the target object;

[0184] The second image acquisition unit 2320 is used to acquire the action posture description information, and acquire the target body action posture image, and the head region area and head region coordinates corresponding to the target body action posture image from the action posture library according to the action posture description information and the head orientation of the head image;

[0185] An image adjustment unit 2330 is used to perform image adjustment on the head image according to the head area and the head region area to obtain a head adjusted image, wherein after the image adjustment, the head coordinates are adjusted to the coordinates of the head adjusted image, and the area of ​​the head adjusted image is equal to the area of ​​the head region;

[0186] The image synthesis unit 2340 is used to match the coordinates of the head adjustment image to the head region coordinates, so as to synthesize the head adjustment image and the target body motion posture image to obtain a synthesized image.

[0187] In one embodiment, the coordinates of the head adjustment image include the coordinates of the center point of the head, and the coordinates of the head region include the coordinates of the center point of the head region; the image synthesis unit 2340 is further used to:

[0188] The coordinates of the center point of the head are aligned with the coordinates of the center point of the head region to synthesize the head adjustment image and the target body motion posture image to obtain a synthesized image.

[0189] In one embodiment, the image synthesis unit 2340 is further configured to:

[0190] Align the coordinates of the center point of the head with the coordinates of the center point of the head area;

[0191] Determine whether there is a gap between the head adjustment image after the center point coordinates of the head are aligned with the center point coordinates of the head area and the target body action posture image;

[0192] When there is no gap between the head adjustment image and the target body motion posture image, a combination of the head adjustment image and the target body motion posture image after aligning the coordinates of the center point of the head with the coordinates of the center point of the head region is used as a composite image.

[0193] In one embodiment, the coordinates of the head adjustment image further include the coordinates of the head chin point, and the head region coordinates include the coordinates of the head region chin point; the image synthesis unit 2340 is further configured to:

[0194] When there is a gap between the head adjustment image and the target body motion posture image, the coordinates of the head chin point are aligned with the coordinates of the head region chin point to synthesize the head adjustment image and the target body motion posture image to obtain a synthesized image.

[0195] In one embodiment, the image processing device 2300 further includes a motion gesture library construction unit, which is used to:

[0196] Get action posture labels;

[0197] Obtaining a body action posture image, and a head region area and head region coordinates corresponding to the body action posture image according to the action posture label;

[0198] An action posture library is constructed according to the action posture labels, the body action posture images, the head area and the head area coordinates corresponding to the body action posture images.

[0199] In one embodiment, the action posture library construction unit is further used for:

[0200] Acquire human body action posture images according to action posture labels;

[0201] Perform image semantic segmentation on the human body motion posture image to obtain the body motion posture image and the head region area and head region coordinates corresponding to the body motion posture image.

[0202] In one embodiment, the action posture library construction unit is further used for:

[0203] Perform image semantic segmentation on the human body action posture image to obtain the head area image and the pixel coordinates of the head area image;

[0204] According to the pixel coordinates of the head region image, the area of ​​the head region image and the region coordinates of the head region image are calculated;

[0205] Obtaining a body action posture image according to the human action posture image and the head region image;

[0206] The area of ​​the head region image is used as the head region area corresponding to the body action posture image, and the region coordinates of the head region image are used as the head region coordinates corresponding to the body action posture image.

[0207] In one embodiment, the first image acquisition unit 2310 is further configured to:

[0208] Obtaining an image of a target object;

[0209] The image semantic segmentation is performed on the image to obtain the head image in the image and the head area and head coordinates of the head image.

[0210] In one embodiment, the first image acquisition unit 2310 is further configured to:

[0211] Performing image semantic segmentation on the image to obtain the head image and the pixel coordinates of the head image in the image;

[0212] According to the pixel coordinates of the head image, the head area and the head coordinates of the head image are calculated.

[0213] In one embodiment, the image processing apparatus 2300 further includes:

[0214] An information acquisition unit, used to acquire style conversion guidance information and background description information;

[0215] The image generation unit is used to generate a target style image according to the style conversion guidance information, the background description information and the synthesized image.

[0216] In one embodiment, the image generation unit is further configured to:

[0217] Performing style conversion on the synthesized image according to the style conversion guidance information to obtain a style conversion image;

[0218] Generate a background image according to the background description information, wherein the background image matches the style of the style transfer image;

[0219] The style transfer image is fused with the background image to obtain the target style image.

[0220] It should be noted that since the image processing device 2300 of this embodiment can implement the image processing method of the previous embodiment, the image processing device 2300 of this embodiment and the image processing method of the previous embodiment have the same technical principles and the same beneficial effects. In order to avoid repetition, they will not be repeated here.

[0221] The electronic device for executing the above-mentioned image processing method provided in the embodiment of the present application may be a user terminal, referring to Fig.24 , Fig.24 This is a partial structural block diagram of a user terminal provided in an embodiment of the present application. The user terminal includes: a camera assembly 2410, a first memory 2420, an input unit 2430, a display unit 2440, a sensor 2450, an audio circuit 2460, a wireless fidelity (WiFi) module 2470, a first processor 2480, and a first power supply 2490. Those skilled in the art can understand that Fig.24 The user terminal structure shown in the figure does not constitute a limitation on the user terminal, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0222] The camera assembly 2410 can be used to collect images or videos. Optionally, the camera assembly 2410 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the user terminal, and the rear camera is arranged on the back of the user terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions.

[0223] The first memory 2420 may be used to store software programs and modules. The first processor 2480 executes various functional applications and data processing of the user terminal by running the software programs and modules stored in the first memory 2420 .

[0224] The input unit 2430 may be used to receive input digital or character information and generate key signal input related to the setting and function control of the user terminal. Specifically, the input unit 2430 may include a touch panel 2424 and other input devices 2432 .

[0225] The display unit 2440 may be used to display input information or provided information and various menus of the user terminal. The display unit 2440 may include a display panel 2441.

[0226] The audio circuit 2460, the speaker 2461, and the microphone 2462 may provide an audio interface.

[0227] The first power source 2490 may be alternating current, direct current, a disposable battery, or a rechargeable battery.

[0228] The number of sensors 2450 may be one or more, and the one or more sensors 2450 include but are not limited to: acceleration sensors, gyroscope sensors, pressure sensors, optical sensors, etc. Among them:

[0229] The acceleration sensor can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the user terminal. For example, the acceleration sensor can be used to detect the components of gravity acceleration on the three coordinate axes. The first processor 2480 can control the display unit 2440 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor. The acceleration sensor can also be used for collecting game or user motion data.

[0230] The gyroscope sensor can detect the body direction and rotation angle of the user terminal, and the gyroscope sensor can cooperate with the acceleration sensor to collect the user's 3D action on the user terminal. The first processor 2480 can implement the following functions based on the data collected by the gyroscope sensor: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0231] The pressure sensor can be set in the side frame of the user terminal and / or the lower layer of the display unit 2440. When the pressure sensor is set in the side frame of the user terminal, the user's holding signal of the user terminal can be detected, and the first processor 2480 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor. When the pressure sensor is set in the lower layer of the display unit 2440, the first processor 2480 controls the operability controls on the UI interface according to the user's pressure operation on the display unit 2440. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0232] The optical sensor is used to collect the ambient light intensity. In one embodiment, the first processor 2480 can control the display brightness of the display unit 2440 according to the ambient light intensity collected by the optical sensor. Specifically, when the ambient light intensity is high, the display brightness of the display unit 2440 is increased; when the ambient light intensity is low, the display brightness of the display unit 2440 is decreased. In another embodiment, the first processor 2480 can also dynamically adjust the shooting parameters of the camera assembly 2410 according to the ambient light intensity collected by the optical sensor.

[0233] In this embodiment, the first processor 2480 included in the user terminal can execute the image processing method of the previous embodiment.

[0234] The electronic device for executing the above-mentioned image processing method provided in the embodiment of the present application may also be a server, referring to Fig.25 , Fig.25Partial structural block diagram of a server provided in an embodiment of the present application, the server 2500 may have relatively large differences due to different configurations or performances, and may include one or more second processors 2522 and a second memory 2532, and one or more storage media 2530 (e.g., one or more mass storage devices) storing application programs 2542 or data 2544. Among them, the second memory 2532 and the storage medium 2530 may be short-term storage or permanent storage. The program stored in the storage medium 2530 may include one or more modules (not shown in the figure), and each module may include a series of command operations in the server 2500. Furthermore, the second processor 2522 may be configured to communicate with the storage medium 2530 and execute a series of command operations in the storage medium 2530 on the server 2500.

[0235] The server 2500 may also include one or more second power supplies 2526, one or more wired or wireless network interfaces 2550, one or more input and output interfaces 2558, and / or one or more operating systems 2541, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0236] The processor in the server 2500 may be used to execute the image processing method.

[0237] An embodiment of the present application further provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the image processing method of each of the aforementioned embodiments.

[0238] The embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned image processing method.

[0239] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0240] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0241] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0242] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0243] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0244] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0245] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0246] The step numbers in the above method embodiment are only provided for the convenience of explanation and description, and no limitation is imposed on the order of the steps. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

Claims

1. An image processing method, characterized in that: The following steps are involved: Acquire a head image in the image of the target object, as well as the head area and head coordinates of the head image; Acquire action posture description information, and acquire a target body action posture image, and a head region area and head region coordinates corresponding to the target body action posture image from an action posture library according to the action posture description information and the head orientation of the head image; performing image adjustment on the head image according to the head area and the head region area to obtain a head adjusted image, wherein after the image adjustment, the head coordinates are adjusted to the coordinates of the head adjusted image, and the area of ​​the head adjusted image is equal to the area of ​​the head region; The coordinates of the head adjustment image are matched to the head region coordinates to synthesize the head adjustment image and the target body motion posture image to obtain a synthesized image.

2. The method according to claim 1, characterized in that The coordinates of the head adjustment image include the coordinates of the center point of the head, and the coordinates of the head region include the coordinates of the center point of the head region; The step of matching the coordinates of the head adjustment image to the head region coordinates to synthesize the head adjustment image and the target body motion posture image to obtain a synthesized image includes: The center point coordinates of the head are aligned with the center point coordinates of the head region to synthesize the head adjustment image and the target body motion posture image to obtain a synthesized image.

3. The method according to claim 2, characterized in that The step of aligning the center point coordinates of the head with the center point coordinates of the head region to synthesize the head adjustment image and the target body motion posture image to obtain a synthesized image includes: Aligning the coordinates of the center point of the head with the coordinates of the center point of the head area; Determine whether there is a gap between the head adjustment image and the target body motion posture image after the head center point coordinates are aligned with the head area center point coordinates; When there is no gap between the head adjustment image and the target body motion posture image, a combination of the head adjustment image and the target body motion posture image after aligning the head center point coordinates with the head area center point coordinates is used as a composite image.

4. The method according to claim 3, characterized in that The coordinates of the head adjustment image further include the coordinates of the head chin point, and the head region coordinates include the coordinates of the head region chin point; The step of aligning the coordinates of the center point of the head with the coordinates of the center point of the head region to synthesize the head adjustment image and the target body motion posture image to obtain a synthesized image further includes: When there is a gap between the head adjustment image and the target body motion posture image, the head chin point coordinates are aligned with the head area chin point coordinates to synthesize the head adjustment image and the target body motion posture image to obtain a synthesized image.

5. The method according to claim 1, characterized in that The action posture library is constructed by the following steps: Get action posture labels; Acquire a body motion posture image according to the motion posture label, and a head region area and head region coordinates corresponding to the body motion posture image; The action posture library is constructed according to the action posture label, the body action posture image, and the head region area and head region coordinates corresponding to the body action posture image.

6. The method according to claim 5, characterized in that The step of acquiring a body motion posture image according to the motion posture label, and a head region area and head region coordinates corresponding to the body motion posture image, comprises: Acquire a human body action posture image according to the action posture label; Performing image semantic segmentation on the human body motion posture image to obtain a body motion posture image and a head region area and head region coordinates corresponding to the body motion posture image.

7. The method according to claim 6, characterized in that The performing image semantic segmentation on the human body motion posture image to obtain the body motion posture image and the head region area and head region coordinates corresponding to the body motion posture image includes: Performing image semantic segmentation on the human body action posture image to obtain a head region image and pixel coordinates of the head region image; Calculating the area of ​​the head region image and the region coordinates of the head region image according to the pixel coordinates of the head region image; Obtaining a body motion posture image according to the human body motion posture image and the head region image; The area of ​​the head region image is used as the head region area corresponding to the body action posture image, and the region coordinates of the head region image are used as the head region coordinates corresponding to the body action posture image.

8. The method according to claim 1, characterized in that The step of acquiring the head image in the image of the target object, and the head area and head coordinates of the head image, includes: Obtaining an image of a target object; Perform image semantic segmentation on the image image to obtain a head image in the image image and a head area and head coordinates of the head image.

9. The method according to claim 8, characterized in that The performing image semantic segmentation on the image to obtain the head image in the image and the head area and head coordinates of the head image includes: Performing image semantic segmentation on the image to obtain a head image in the image and pixel coordinates of the head image; The head area and the head coordinates of the head image are calculated according to the pixel coordinates of the head image.

10. The method according to claim 1, characterized in that The method further comprises: Obtain style transfer guidance information and background description information; A target style image is generated according to the style conversion guidance information, the background description information and the synthesized image.

11. The method according to claim 10, characterized in that The step of generating a target style image according to the style conversion guidance information, the background description information and the synthesized image includes: Performing style conversion on the synthesized image according to the style conversion guidance information to obtain a style converted image; Generate a background image according to the background description information, wherein the background image matches the style of the style conversion image; The style conversion image is fused with the background image to obtain a target style image.

12. An image processing device, characterized in that: include: A first image acquisition unit, used to acquire a head image in the image of the target object, as well as a head area and head coordinates of the head image; a second image acquisition unit, configured to acquire action posture description information, and acquire a target body action posture image, and a head region area and head region coordinates corresponding to the target body action posture image from an action posture library according to the action posture description information and the head orientation of the head image; an image adjustment unit, configured to perform image adjustment on the head image according to the head area and the head region area to obtain a head adjustment image, wherein after the image adjustment, the head coordinates are adjusted to the coordinates of the head adjustment image, and the area of ​​the head adjustment image is equal to the area of ​​the head region; The image synthesis unit is used to match the coordinates of the head adjustment image to the head region coordinates, so as to synthesize the head adjustment image and the target body motion posture image to obtain a synthesized image.

13. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the image processing method according to any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 11 is implemented.

Images