Image generation method and device, computer program product and electronic equipment

Abstract

The invention relates to the technical field of computers, in particular to an image generation method and device, a computer program product and electronic equipment. The image generation method comprises the steps that an original image and a style image of a role object are acquired, role feature points in the original image are identified and marked, and mark point information is obtained; performing diffusion processing on the original image through a diffusion model based on the style image to obtain a first reconstructed image with a target style; and binding the basic skeleton structure with the first reconstructed image based on the marking point information to obtain a second reconstructed image, and generating an image with the target action based on the second reconstructed image. According to the invention, automatic generation of role images of various postures can be realized, and the generation efficiency and quality of the role images are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to an image generating method, an image generating device, a computer program product, and an electronic device. Background Art

[0002] With the development of computer technology, the fields of games, movies and virtual reality have developed rapidly, and the demand for the generation of high-precision character images is getting higher and higher. For example, the KV (Key Visual) image of the character in the game is used to display and convey the image and characteristics of the game character, highlight the characteristics of the game character and the story background, etc., so as to attract the attention of users. However, the current method of generating character images requires the provision of more high-quality character materials, otherwise it is difficult to generate high-quality character images of various postures, which increases the processing flow and investment cost of image generation.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The purpose of the present disclosure is to provide an image generation method and device, a computer program product and an electronic device, thereby enabling the automatic generation of character images in various postures to at least a certain extent, thereby improving the generation efficiency and quality of character images.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to one aspect of the present disclosure, there is provided an image generation method, comprising: acquiring an original image and a style image of a character object, identifying and annotating the character feature points in the original image to obtain annotated point information; based on the style image, performing diffusion processing on the original image through a diffusion model to obtain a first reconstructed image with a target style; based on the annotated point information, binding a basic skeletal structure with the first reconstructed image to obtain a second reconstructed image, so as to generate an image with a target action based on the second reconstructed image.

[0007] In an exemplary embodiment of the present disclosure, based on a style image, an original image is diffused through a diffusion model to obtain a first reconstructed image with a target style, including: determining a target network layer from a network structure of the diffusion model; inputting the style image and the original image into the diffusion model respectively to obtain a first feature map of the style image in the target network layer, and obtaining a second feature map of the original image in the target network layer; performing style transfer on the second feature map according to the first feature map to obtain a target feature map, and injecting the target feature map as the output of the target network layer into the diffusion model for processing to obtain a first reconstructed image.

[0008] In an exemplary embodiment of the present disclosure, a target network layer includes at least one first subnetwork and at least one second subnetwork; style transfer is performed on the second feature map according to the first feature map to obtain a target feature map, and the target feature map is injected into the diffusion model as the output of the target network layer for processing to obtain a first reconstructed image, including: based on the first feature map extracted from the style image by the first subnetwork, style transfer is performed on the second feature map extracted from the original image by the first subnetwork to obtain a first reference feature map; based on the first feature map extracted from the style image by the second subnetwork, style transfer is performed on the second feature map extracted from the original image by the second subnetwork to obtain a second reference feature map; the first reference feature map and the second reference feature map are injected into the diffusion model as target feature maps for processing to obtain a first reconstructed image.

[0009] In an exemplary embodiment of the present disclosure, a basic skeletal structure is bound to a first reconstructed image based on annotation point information to obtain a second reconstructed image, including: data alignment of the annotation point information with the basic skeletal structure; and binding the data-aligned basic skeletal structure to character feature points in the first reconstructed image according to the annotation point information to obtain a second reconstructed image and skeletal control points in the second reconstructed image.

[0010] In an exemplary embodiment of the present disclosure, an image with a target action is generated based on a second reconstructed image, including: adjusting the posture of the character object in the second reconstructed image based on the skeleton control points; and generating an image with the target action based on the second reconstructed image after the posture is adjusted.

[0011] In an exemplary embodiment of the present disclosure, generating an image with a target action based on the second reconstructed image also includes: extracting a face from the image with the target action; and performing face-changing processing on the extracted face based on a face image corresponding to the original image.

[0012] In an exemplary embodiment of the present disclosure, before obtaining the original image and style image of the character object, identifying the character feature points in the original image for annotation, and obtaining the annotation point information, the method also includes: obtaining at least one initial content image of the character object; performing super-resolution processing on the initial content image, and performing feature similarity evaluation on the initial content image after super-resolution processing and the initial content image; and determining the original image from the initial content image based on the feature similarity evaluation result.

[0013] According to one aspect of the present disclosure, there is provided an image generating device, comprising: an image acquiring module, for acquiring an original image and a style image of a character object, identifying and annotating the character feature points in the original image, and obtaining annotated point information; an image generating module, for performing diffusion processing on the original image through a diffusion model based on the style image, and obtaining a first reconstructed image having a target style; and an image processing module, for binding a basic skeletal structure with the first reconstructed image based on the annotated point information, and obtaining a second reconstructed image, so as to generate an image having a target action based on the second reconstructed image.

[0014] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, any one of the above methods is implemented.

[0015] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above methods by executing the executable instructions.

[0016] The image generation method in the exemplary embodiment of the present disclosure obtains the original image and style image of the character object, identifies the character feature points in the original image and annotates them to obtain the annotated point information, and then based on the style image, diffuses the original image through a diffusion model to obtain a first reconstructed image with a target style, and binds the basic skeletal structure to the first reconstructed image based on the annotated point information to obtain a second reconstructed image, so as to generate an image with a target action based on the second reconstructed image. This process can generate an image with a target style and a target action through an original image and a style image of a character object, thereby reducing the processing flow chart for generating the character image and improving the generation efficiency. In addition, since the basic skeletal structure is bound to the first reconstructed image based on the annotated point information, when generating an image based on the second reconstructed image, the posture of the character object can be adjusted based on the basic skeletal structure, so as to quickly obtain a variety of postures of the character object, and then efficiently generate a variety of images with different target actions.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present disclosure are shown in an exemplary and non-limiting manner.

[0019] Figure 1 An application environment according to an exemplary embodiment of the present disclosure is shown.

[0020] Figure 2 A flowchart of an image generating method according to an exemplary embodiment of the present disclosure is shown.

[0021] Figure 3 A flowchart of an implementation method for acquiring a first reconstructed image with a target style according to an exemplary embodiment of the present disclosure is shown.

[0022] Figure 4 A flowchart is shown of another implementation method of acquiring a first reconstructed image having a target style according to an exemplary embodiment of the present disclosure.

[0023] Figure 5 A flowchart of an implementation method of generating an image with a target action according to an exemplary embodiment of the present disclosure is shown.

[0024] Figure 6 A schematic diagram of adjusting the action of a character object based on skeleton control points according to an exemplary embodiment of the present disclosure is shown.

[0025] Figure 7 A flowchart of an implementation method of acquiring an original image according to an exemplary embodiment of the present disclosure is shown.

[0026] Figure 8 A schematic diagram showing the composition of an image generating device according to an exemplary embodiment of the present disclosure is shown.

[0027] Fig. 9 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown.

[0028] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0029] The exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and the concepts of the exemplary embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the figures represent the same or similar structures, and thus their detailed description will be omitted.

[0030] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known structures, methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or these functional entities or parts of functional entities may be implemented in one or more software hardened modules, or these functional entities may be implemented in different networks and / or processor devices and / or microcontroller devices.

[0032] At present, the demand for high-precision realistic character image generation is getting higher and higher. For example, the character KV images currently generated are mostly for two-dimensional characters, with relatively simple costumes, and the characters are usually only shown from a single frontal perspective. When generating character images, if there are only a small amount of low-quality materials for the character, and the character complexity is extremely high, it is impossible to automatically generate images of the character in multiple postures, and there is a disadvantage of low image quality. Otherwise, it will take a long time and cost to achieve. Taking the traditional generation of character KV images as an example, it generally requires demand analysis, sketching, character design, scene design, character and scene refinement and adjustment, etc., and each step takes a lot of time. Although AI technology can improve the efficiency of image generation, there are still high requirements for the quantity and quality of materials, which affects the generation efficiency and quality of character images to a certain extent. Therefore, it is urgent to provide a method for automatically generating high-quality character images based on a small amount of materials.

[0033] It should be noted that the image generation method and device provided by the exemplary embodiments of the present disclosure can be used in the field of games, such as various scenes that require the generation of character images. The image generation method and device provided by the exemplary embodiments of the present disclosure can also be used in any field other than the field of games, such as in the fields of virtual reality VR, augmented display AR, and film and television production. The application field of the image generation method and device provided by the exemplary embodiments of the present disclosure is not limited, and the following description is taken as an example of generating character images in the field of games.

[0034] The image generation method provided by the exemplary embodiment of the present disclosure can be applied to Figure 1 In the application environment shown, the terminal 101 communicates with the server 102 via a network. The data storage system can store data that the server 102 needs to process. The data storage system can be integrated on the server 102, or placed on the cloud or other network servers.

[0035] In an exemplary embodiment, the image generation method provided by the exemplary embodiment of the present disclosure may be executed by the server 102, and the corresponding image generation device is disposed in the server 102. Correspondingly, in this mode of execution by the server 102, the server 102 may start to execute the steps in the technical solution of the exemplary embodiment of the present disclosure in response to a trigger command, wherein the trigger command may be sent by a terminal used by a user, or may be triggered locally by the server in response to some automation events.

[0036] The server 102 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server 102 may perform background tasks.

[0037] Furthermore, in another exemplary embodiment, the terminal 101 may also have similar functions to the server 102 , so as to execute the image generating method provided by the exemplary embodiment of the present disclosure.

[0038] The terminal 101 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an IoT device, and a portable wearable device. The IoT device may be a smart speaker, a smart TV, a smart air conditioner, and a smart vehicle-mounted device, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The terminal 101 may also be referred to as a mobile terminal, a terminal device, a mobile device, etc. The exemplary embodiments of the present disclosure do not limit the type of the terminal 101.

[0039] In addition, the technical solution of the exemplary embodiment of the present disclosure can also be executed by the terminal 101 and the server 102 in collaboration. In this way of collaborative execution by the terminal 101 and the server 102, some steps in the technical solution provided by the exemplary embodiment of the present disclosure are executed by the terminal 101, and other steps are executed by the server 102.

[0040] It should be noted that in this mode of collaborative execution by the terminal 101 and the server 102, the steps respectively executed by the terminal 101 and the server 102 can be dynamically adjusted according to actual conditions, and there is no special restriction on this.

[0041] The terminal 101 and the server 102 may be connected directly or indirectly via wireless communication, and the exemplary embodiments of the present disclosure do not impose any special limitation thereto.

[0042] In an exemplary embodiment of the present disclosure, a method for generating an image is provided, referring to Figure 2 FIG. 1 is a flow chart of an image generation method according to an exemplary embodiment of the present disclosure. Figure 2 As shown, the image generation method includes steps S210 to S230, which are specifically as follows:

[0043] In step S210, an original image and a style image of a character object are obtained, and the character feature points in the original image are identified and annotated to obtain annotated point information.

[0044] In the exemplary embodiments of the present disclosure, the original image refers to the content image of the character object, which provides content information for generating an image with a target action, and may include facial parts, body parts, clothing and accessories, etc., without limitation. Among them, if the original image also includes a background, the background can be removed by data annotation. Of course, if the same background is to be retained in the final generated image, the background may not be removed. The original image can be collected by shooting a model of the character object, such as shooting an image or video, or can be obtained by downloading from the Internet, etc., without limitation.

[0045] The style image refers to an image that provides style information, that is, the final generated image will have the same style as the style image, such as cartoon style, natural style, documentary style, etc. The style image can also be obtained by shooting, downloading, etc., and there is no restriction on this.

[0046] Identifying the character feature points in the original image can be done by processing the original image through image recognition technology, and identifying the character feature points in the original image, including facial feature points, skeleton key points, and clothing and accessories key points, and annotating the character feature points to obtain annotated point information. It is necessary to ensure the consistency of the annotated character feature points in different original images of the same character object.

[0047] In step S220, based on the style image, the original image is diffused by a diffusion model to obtain a first reconstructed image with a target style.

[0048] In an exemplary embodiment of the present disclosure, the diffusion model works by simulating a step-by-step diffusion process, which first gradually converts data into unstructured random noise and then reverses the process to recover meaningful data from the noise. The exemplary embodiment of the present disclosure can select a diffusion model according to demand, and there is no specific limitation on the type of diffusion model.

[0049] Among them, the diffusion model needs to be trained before this. Specifically, training samples can be obtained first, and the samples can be enhanced by data enhancement (such as flipping, scaling, rotation), etc. to increase the diversity of samples. Then, a noise image is obtained by adding different degrees of random noise to the training samples, and during the training process, the diffusion model is used to restore the original image from the noise image at different noise levels, so that the model selects the details and features of the image. After multiple rounds of training and parameter optimization, the image generation ability and image restoration accuracy of the model are improved. Optionally, during the training process, the labeled data can be used to distinguish between the character object and the background to ensure that the model can focus on generating the image of the character object and ignore the changes in the background. The exemplary embodiments of the present disclosure do not specifically limit the training process of the model.

[0050] It should be understood that based on the style image, the original image is diffused through the diffusion model, so that the first reconstructed image obtained through the diffusion model not only restores the original image with high quality, but also has the style type of the style image, thereby obtaining a high-quality image that meets the required style.

[0051] In step S230, the basic skeleton structure is bound to the first reconstructed image based on the annotation point information to obtain a second reconstructed image, so as to generate an image with the target action based on the second reconstructed image.

[0052] In an exemplary embodiment of the present disclosure, the basic skeletal structure is a predefined skeleton structure composed of interconnected "bones", and different postures or actions are obtained by changing the orientation or position of the bones. The basic skeletal structure may include human or animal body bones, including facial bones, hand bones, and trunk bones. For example, the basic skeletal structure of a human body may be defined based on the human body structure, or the basic skeletal structure of an animal may be defined based on the animal body structure, without limitation.

[0053] The same basic skeleton structure can be used for the same character object, and of course, the same basic skeleton structure can also be used for multiple different character objects according to needs. The skeleton posture of the basic skeleton structure can indicate an action or posture, such as putting hands on hips, waving, standing, bending over, etc.

[0054] The basic skeleton structure is bound to the first reconstructed image based on the annotation point information, and the position or orientation of each part in the image can be accurately controlled by the basic skeleton structure to obtain different postures or actions, thereby rendering and generating an image with a target action.

[0055] The image generation method in the exemplary embodiment of the present disclosure can generate an image with a target style and a target action through an original image of a character object and a style image, thereby reducing the processing flow chart for generating the character image and improving the generation efficiency. In addition, since the basic skeletal structure is bound to the first reconstructed image based on the annotation point information, when generating an image based on the second reconstructed image, the posture of the character object can be adjusted based on the basic skeletal structure, thereby quickly obtaining a variety of postures of the character object, and then efficiently generating a variety of images with different target actions.

[0056] In an exemplary embodiment, a method for obtaining a first reconstructed image having a target style is provided. Figure 3 As shown, based on the style image, performing diffusion processing on the original image through a diffusion model to obtain a first reconstructed image with a target style may include:

[0057] Step S310: determining a target network layer from the network structure of the diffusion model.

[0058] A network layer suitable for style transfer can be selected from the network structure of the diffusion model, and an intermediate layer in the hierarchical structure of the diffusion model can be selected, the intermediate layer containing high-level semantic information and local details of the image. For example, the l1th layer and the l2th layer of the diffusion model can be selected for style transfer.

[0059] Step S320: Input the style image and the original image into the diffusion model respectively, obtain the first feature map of the style image at the target network layer, and obtain the second feature map of the original image at the target network layer.

[0060] A first feature map of the style image at the target network layer can be extracted through a pre-trained neural network (such as a convolutional neural network), and a second feature map of the original image at the target network layer can be extracted.

[0061] Step S330: Perform style transfer on the second feature map according to the first feature map to obtain a target feature map, and inject the target feature map as the output of the target network layer into the diffusion model for processing to obtain a first reconstructed image.

[0062] Style transfer is to combine the content features of the original image (content image) with the style features of the style image to generate a content image with the style of the style image. It can achieve style transfer by calculating the feature difference between the generated image and the original image at the target network layer, and calculating the difference between the generated image and the style image at the target network layer, and optimizing the generated image by gradient descent. Among them, the mean and variance (based on pixel values) of the original image can be used to adjust the feature map of the style image so that it has the structural features of the content image and the style features of the style image, and an image with content features and style features is generated through back propagation.

[0063] Specifically, the target network layer may include at least one first subnetwork and at least one second subnetwork, such as Figure 4 As shown, performing style transfer on the second feature map according to the first feature map to obtain a target feature map, and injecting the target feature map as the output of the target network layer into the diffusion model for processing, to obtain a first reconstructed image may include:

[0064] Step S410: Based on the first feature map extracted from the style image by the first sub-network, style transfer is performed on the second feature map extracted from the original image by the first sub-network to obtain a first reference feature map; Step S420: Based on the first feature map extracted from the style image by the second sub-network, style transfer is performed on the second feature map extracted from the original image by the second sub-network to obtain a second reference feature map; Step S430: The first reference feature map and the second reference feature map are injected into the diffusion model as target feature maps for processing to obtain a first reconstructed image.

[0065] Exemplarily, the target network layer is selected from the l1 layer and the l2 layer of the diffusion model as an example for explanation. The first feature map (Fs1) of the style image in the l1 layer can be extracted from the diffusion model, and the first feature map (Fs2) of the style image in the l2 layer can be extracted, and then the second feature map (Fc1) of the original image in the l1 layer can be extracted from the diffusion model, and the second feature map (Fc2) of the original image in the l2 layer can be extracted, and then the style migration algorithm is used to perform style migration processing on the l1 layer and the l2 layer, that is, based on the first feature map (Fs1), the second feature map (Fc1) is style migrated to obtain a new feature map, that is, the first reference feature map (F'c1), and based on the first feature map (Fs2), the second feature map (Fc2) is style migrated to obtain a new feature map, that is, the second reference feature map (F'c2), and finally the first reference feature map and the second reference feature map are re-injected into the diffusion model, so as to generate an optimized image, that is, the first reconstructed image, through the decoder part of the diffusion model. The first reconstructed image contains the style features of the style image and the content features of the content image. By using the multi-layer network structure of the diffusion model to achieve style transfer during the diffusion process, style transfer can be performed on image features with high-level semantic information and local details, which is conducive to capturing rich image features to achieve style transfer and improve the accuracy of style transfer.

[0066] The diffusion model can introduce noise according to the preset random seed and denoising strength by adding noise to the original image and denoising the original image. This process will erase the details in the original image. The random seed is used to determine the randomness of the generation process in order to obtain reproducible results. The seed determines the process of noise generation in the algorithm. If the number of seeds is consistent with other input conditions, multiple generations will generally produce almost identical results. The random seed is input by the user or generated randomly, and there is no restriction on this.

[0067] During denoising, the added noise is removed under the guidance of the diffusion model condition information, and new image details are generated in the place where the noise is removed. That is, this process is equivalent to guiding the diffusion model to reconstruct and generate details in a partial area of ​​the image to obtain an image generation processing result, so that the image generates a high-quality processing result. The condition information can be set according to needs, and there is no restriction on this. In addition, when the exemplary embodiment of the present disclosure performs diffusion processing on the original image, it extracts the features of the target network layer and performs style transfer, and injects the new features obtained after the style transfer into the diffusion model for further processing, so that the diffusion model not only performs diffusion processing on the original image according to the condition information, but also realizes style transfer, thereby improving the efficiency of generating high-quality character images with the target style.

[0068] In an exemplary embodiment, a method for obtaining a second reconstructed image is provided. Binding the basic bone structure with the first reconstructed image based on the annotation point information to obtain the second reconstructed image may include:

[0069] The annotation point information is data-aligned with the basic skeleton structure, and the basic skeleton structure after data alignment is bound to the character feature points in the first reconstructed image according to the annotation point information to obtain a second reconstructed image and the skeleton control points in the second reconstructed image.

[0070] The annotation point information includes the character feature point information of the character object. The basic skeleton structure is first aligned with the character feature point information to achieve skeleton initialization so that the skeleton can be bound later. The principle of skeleton binding is that the movement of the skeleton object drives the movement of the grid points, and the grid points move with the movement of the skeleton. By binding the basic skeleton structure after data alignment with the character feature points in the first reconstructed image, the character feature points (parts) in the image can be controlled by controlling the basic skeleton structure to adjust different postures or actions.

[0071] Because the first reconstructed image is obtained by diffusion processing based on the original image, the character feature points in the first reconstructed image are consistent with the character feature points in the original image, and there is no need to determine them repeatedly. The basic bone structure after data alignment is bound to the first reconstructed image. In the second reconstructed image obtained, the original character feature points and the corresponding grid structure in the basic bone structure have a mapping relationship, and the character feature points can move with the movement of the grid points in the basic bone structure, so that the character feature points bound to the grid structure are determined as bone control points.

[0072] By binding the first reconstructed image to the data-aligned basic skeleton structure, the posture or action of the character object in the image can be adjusted according to the drive of the basic skeleton structure.

[0073] In an exemplary embodiment, a method for generating an image with a target action is also provided, such as Figure 5 As shown, generating an image with a target motion based on the second reconstructed image may include:

[0074] Step S510: adjusting the posture of the character object in the second reconstructed image based on the skeleton control points.

[0075] The posture change of the character object in the second reconstructed image is driven by the skeleton control points, and can be converted from the original posture of the character object to other postures through interpolation, deformation and other technologies.

[0076] Among them, each bone control point has a weight, which determines the degree of deformation of the bone control point when the bone moves. By adjusting these weights, fine control of the deformation of the character object can be achieved to achieve the desired deformation effect. In skeletal animation, key frames record the position and posture of the bones at a specific time. The interpolation algorithm calculates the bone position of the intermediate frame based on the data of these key frames, so that the action changes look more natural and smooth. The interpolation algorithm includes linear interpolation, spline interpolation, etc., which can be selected according to actual needs. The exemplary embodiments of the present disclosure do not limit the specific process of interpolation and deformation.

[0077] Step S520: Generate an image with the target motion according to the second reconstructed image after the posture is adjusted.

[0078] The image rendering generation process can be performed according to the second reconstructed image after the posture is adjusted, and there is no specific limitation on this, such as Figure 6 The figure shows a schematic diagram of adjusting the action of a character object based on the bone control points. After adjustment, the character object is adjusted from the "simple standing" action to the "preparing to punch" action. Figure 6 Just as an example.

[0079] Based on this, diverse postures of character objects can be quickly generated, which improves the efficiency of generating character objects with different postures.

[0080] In an exemplary embodiment, a method for generating an image with a target action is also provided. Considering that the images of different actions generated based on limited materials need to ensure that the face is fully restored, generating an image with a target action based on the second reconstructed image may include:

[0081] The face is extracted from the image with the target action, and then the extracted face is replaced based on the face image corresponding to the original image.

[0082] Among them, face extraction can be performed through face recognition technology, and the face of an image with a target action can be extracted through a pre-trained face extraction model, and there is no restriction on the extraction method. The face image corresponding to the original image refers to the face image of the character object, which can be an image redrawn in high definition or an original face image provided by the user, and there is no restriction on this.

[0083] The face-changing method for extracting the face based on the face image can refer to the face-changing method of the related art, and is not limited to this. By performing face-changing processing on the face, the face restoration degree of the character object in the image with different actions is made consistent.

[0084] In some optional embodiments, the image with the target action may be optimized in image details to ensure that facial features, matching clothing, etc. are fully restored. For example, high-definition redrawing technology may be used to optimize details and improve image quality.

[0085] In an exemplary embodiment, a method for obtaining an original image is provided. Figure 7 As shown, before obtaining the original image and style image of the character object, identifying the character feature points in the original image for annotation, and obtaining the annotation point information, it also includes:

[0086] Step S710: Acquire at least one initial content image of a character object.

[0087] As can be seen from the above, at least one initial content image can be obtained by photographing a model of a character object or downloading it from the Internet.

[0088] Step S720: performing super-resolution processing on the initial content image, and performing feature similarity evaluation on the initial content image after super-resolution processing and the initial content image.

[0089] The initial content image can be enlarged a second time through super-resolution technology to improve image resolution and detail quality. Super-resolution technology can be, for example, SRCNN (Super-Resolution Convolutional Neural Network, a super-resolution method based on neural network), ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks, a generative adversarial network for image super-resolution), etc., and there is no limitation to this.

[0090] Among them, a preset automated inspection task can be used to perform a consistency check on the initial content image after super-resolution processing and the initial content image to achieve feature similarity assessment. Image consistency check refers to the process of verifying whether the image remains consistent under different conditions or different processing processes by comparing and analyzing various parameters and features of the image. For example, an image similarity measurement method can be used to evaluate the image consistency of images before and after super-resolution processing, such as the SSIM (Structural Similarity) method, the PSNR (Peak Signal to Noise Ratio) method, etc., without limitation.

[0091] Step S730: determining an original image from the initial content images according to the feature similarity evaluation result.

[0092] In order to provide high-quality original images for the subsequent image generation process, an initial content image that passes consistency verification may be selected from the initial content images as the original image according to the feature similarity evaluation result.

[0093] Based on this, even if only low-quality original materials exist for the character object, a high-quality original image can be obtained through super-resolution processing, thereby lowering the application threshold of the exemplary image generator of the present disclosure.

[0094] The image generation method in the exemplary embodiment of the present disclosure obtains the original image and style image of the character object, identifies the character feature points in the original image and annotates them to obtain the annotated point information, and then based on the style image, the original image is diffused through the diffusion model to obtain a first reconstructed image with a target style, and the basic skeleton structure is bound to the first reconstructed image based on the annotated point information to obtain a second reconstructed image, so as to generate an image with a target action based on the second reconstructed image. This process can generate an image with a target style and a target action through an original image and a style image of a character object, thereby reducing the processing flow chart for generating the character image and improving the generation efficiency. In addition, since the basic skeleton structure is bound to the first reconstructed image based on the annotated point information, when generating an image based on the second reconstructed image, the posture of the character object can be adjusted based on the basic skeleton structure, so as to quickly obtain a variety of postures of the character object, and then efficiently generate a variety of images with different target actions. Through the exemplary embodiments of the present disclosure, even in extreme cases, such as when the character objects all use a small amount of low-quality materials, character images with a variety of different actions can be obtained through the automated process of data acquisition, diffusion model, skeleton binding and style transfer, detail optimization and face-changing processing, thereby improving the generation efficiency and quality of the character image.

[0095] In an exemplary embodiment of the present disclosure, an image generating device is also provided. Figure 8 As shown, the image generating device 800 may include an image acquiring module 810, an image generating module 820 and an image processing module 830. Specifically:

[0096] The image acquisition module 810 is used to acquire the original image and style image of the character object, identify the character feature points in the original image and mark them to obtain the marked point information; the image generation module 820 is used to diffuse the original image through a diffusion model based on the style image to obtain a first reconstructed image with a target style; the image processing module 830 is used to bind the basic bone structure with the first reconstructed image based on the marked point information to obtain a second reconstructed image, so as to generate an image with a target action based on the second reconstructed image.

[0097] In an exemplary embodiment of the present disclosure, the image generation module 820 is configured to perform: determining a target network layer from the network structure of a diffusion model; inputting a style image and an original image into the diffusion model respectively, obtaining a first feature map of the style image at the target network layer, and obtaining a second feature map of the original image at the target network layer; performing style migration on the second feature map according to the first feature map to obtain a target feature map, and injecting the target feature map as the output of the target network layer into the diffusion model for processing to obtain a first reconstructed image.

[0098] In an exemplary embodiment of the present disclosure, the target network layer includes at least one first subnetwork and at least one second subnetwork; the image generation module 820 is configured to execute: based on the first feature map extracted from the style image by the first subnetwork, style transfer is performed on the second feature map extracted from the original image by the first subnetwork to obtain a first reference feature map; based on the first feature map extracted from the style image by the second subnetwork, style transfer is performed on the second feature map extracted from the original image by the second subnetwork to obtain a second reference feature map; the first reference feature map and the second reference feature map are injected into the diffusion model as target feature maps for processing to obtain a first reconstructed image.

[0099] In an exemplary embodiment of the present disclosure, the image processing module 830 is configured to perform: data alignment of the annotation point information with the basic skeleton structure; based on the annotation point information, binding the data-aligned basic skeleton structure with the character feature points in the first reconstructed image to obtain a second reconstructed image and the skeleton control points in the second reconstructed image.

[0100] In an exemplary embodiment of the present disclosure, the image processing module 830 is configured to perform: adjusting the posture of the character object in the second reconstructed image based on the skeleton control points; and generating an image with a target action according to the second reconstructed image after the posture is adjusted.

[0101] In an exemplary embodiment of the present disclosure, the image processing module 830 is further configured to execute: extracting a face from an image having a target action; and performing face-changing processing on the extracted face based on a face image corresponding to the original image.

[0102] In an exemplary embodiment of the present disclosure, the image acquisition module 810 is further configured to execute: before acquiring the original image and style image of the character object, identifying the character feature points in the original image for annotation, and obtaining the annotation point information, acquiring at least one initial content image of the character object; performing super-resolution processing on the initial content image, and performing feature similarity evaluation on the initial content image after super-resolution processing and the initial content image; and determining the original image from the initial content image based on the feature similarity evaluation result.

[0103] Since the details of the various functional modules of the image generating device of the exemplary embodiment of the present disclosure have been described in the exemplary embodiment of the above-mentioned image generating method, they will not be repeated here.

[0104] It should be noted that although several modules or units of the image generating device are mentioned in the above detailed description, such division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0105] The exemplary embodiments of the present disclosure further provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the above-mentioned image generation method is implemented.

[0106] In one embodiment, the computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory, and the like.

[0107] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0108] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0109] Computer programs can be carried or transmitted through electrical, magnetic, optical, electromagnetic, infrared and other signals. Electronic devices can convert signals carrying computer programs into digital signals to run computer programs. When a computer program runs on an electronic device, its code is used to enable the electronic device to execute (more specifically, the processor of the electronic device can execute) the method steps of various exemplary embodiments of the present disclosure, such as the above-mentioned image generation method, which includes the following steps: obtaining the original image and style image of the character object, identifying the character feature points in the original image for annotation, and obtaining the annotation point information; based on the style image, the original image is diffused through a diffusion model to obtain a first reconstructed image with a target style; based on the annotation point information, the basic skeletal structure is bound to the first reconstructed image to obtain a second reconstructed image, so as to generate an image with a target action based on the second reconstructed image.

[0110] By executing the above method steps through a computer program, an image with a target style and a target action can be generated through an original image of a character object and a style image, which reduces the processing flow chart for generating the character image and improves the generation efficiency. In addition, since the basic skeletal structure is bound to the first reconstructed image based on the annotation point information, when generating an image based on the second reconstructed image, the posture of the character object can be adjusted based on the basic skeletal structure, so as to quickly obtain a variety of postures of the character object, and then efficiently generate a variety of images with different target actions. Through the exemplary embodiments of the present disclosure, even in extreme cases, such as when the character objects all use a small amount of low-quality materials, it is possible to obtain character images with a variety of different actions through the automated process of data acquisition, diffusion model, skeletal binding and style transfer, detail optimization and face-changing processing, thereby improving the generation efficiency and quality of the character images.

[0111] In an exemplary embodiment of the present disclosure, based on a style image, an original image is diffused through a diffusion model to obtain a first reconstructed image with a target style, including: determining a target network layer from a network structure of the diffusion model; inputting the style image and the original image into the diffusion model respectively to obtain a first feature map of the style image in the target network layer, and obtaining a second feature map of the original image in the target network layer; performing style transfer on the second feature map according to the first feature map to obtain a target feature map, and injecting the target feature map as the output of the target network layer into the diffusion model for processing to obtain a first reconstructed image.

[0112] In an exemplary embodiment of the present disclosure, a target network layer includes at least one first subnetwork and at least one second subnetwork; style transfer is performed on the second feature map according to the first feature map to obtain a target feature map, and the target feature map is injected into the diffusion model as the output of the target network layer for processing to obtain a first reconstructed image, including: based on the first feature map extracted from the style image by the first subnetwork, style transfer is performed on the second feature map extracted from the original image by the first subnetwork to obtain a first reference feature map; based on the first feature map extracted from the style image by the second subnetwork, style transfer is performed on the second feature map extracted from the original image by the second subnetwork to obtain a second reference feature map; the first reference feature map and the second reference feature map are injected into the diffusion model as target feature maps for processing to obtain a first reconstructed image.

[0113] In an exemplary embodiment of the present disclosure, a basic skeletal structure is bound to a first reconstructed image based on annotation point information to obtain a second reconstructed image, including: data alignment of the annotation point information with the basic skeletal structure; and binding the data-aligned basic skeletal structure to character feature points in the first reconstructed image according to the annotation point information to obtain a second reconstructed image and skeletal control points in the second reconstructed image.

[0114] In an exemplary embodiment of the present disclosure, an image with a target action is generated based on a second reconstructed image, including: adjusting the posture of the character object in the second reconstructed image based on the skeleton control points; and generating an image with the target action based on the second reconstructed image after the posture is adjusted.

[0115] In an exemplary embodiment of the present disclosure, generating an image with a target action based on the second reconstructed image also includes: extracting a face from the image with the target action; and performing face-changing processing on the extracted face based on a face image corresponding to the original image.

[0116] In an exemplary embodiment of the present disclosure, before obtaining the original image and style image of the character object, identifying the character feature points in the original image for annotation, and obtaining the annotation point information, the method also includes: obtaining at least one initial content image of the character object; performing super-resolution processing on the initial content image, and performing feature similarity evaluation on the initial content image after super-resolution processing and the initial content image; determining the original image from the initial content image according to the feature similarity evaluation result. In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided. It can be understood by those skilled in the art that various aspects of the present disclosure can be implemented as a system, method or program product. Therefore, various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuit", "module" or "system" here.

[0117] Refer to the following Fig. 9 hereinafter describes an electronic device 900 according to such an embodiment of the present disclosure. Fig. 9 The electronic device 900 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0118] like Fig. 9 As shown, the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 may include, but are not limited to: the at least one processing unit 910, the at least one storage unit 920, a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910), and a display unit 940.

[0119] The storage unit stores a program code, and the program code can be executed by the processing unit 910, so that the processing unit 910 executes the steps described in the "Exemplary Method" section of the specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 910 can execute the following steps: obtaining an original image and a style image of a character object, identifying the character feature points in the original image for annotation, and obtaining annotation point information; based on the style image, the original image is diffused through a diffusion model to obtain a first reconstructed image with a target style; based on the annotation point information, the basic bone structure is bound to the first reconstructed image to obtain a second reconstructed image, so as to generate an image with a target action based on the second reconstructed image.

[0120] In an exemplary embodiment of the present disclosure, based on a style image, an original image is diffused through a diffusion model to obtain a first reconstructed image with a target style, including: determining a target network layer from a network structure of the diffusion model; inputting the style image and the original image into the diffusion model respectively to obtain a first feature map of the style image in the target network layer, and obtaining a second feature map of the original image in the target network layer; performing style transfer on the second feature map according to the first feature map to obtain a target feature map, and injecting the target feature map as the output of the target network layer into the diffusion model for processing to obtain a first reconstructed image.

[0121] In an exemplary embodiment of the present disclosure, a target network layer includes at least one first subnetwork and at least one second subnetwork; style transfer is performed on the second feature map according to the first feature map to obtain a target feature map, and the target feature map is injected into the diffusion model as the output of the target network layer for processing to obtain a first reconstructed image, including: based on the first feature map extracted from the style image by the first subnetwork, style transfer is performed on the second feature map extracted from the original image by the first subnetwork to obtain a first reference feature map; based on the first feature map extracted from the style image by the second subnetwork, style transfer is performed on the second feature map extracted from the original image by the second subnetwork to obtain a second reference feature map; the first reference feature map and the second reference feature map are injected into the diffusion model as target feature maps for processing to obtain a first reconstructed image.

[0122] In an exemplary embodiment of the present disclosure, a basic skeletal structure is bound to a first reconstructed image based on annotation point information to obtain a second reconstructed image, including: data alignment of the annotation point information with the basic skeletal structure; and binding the data-aligned basic skeletal structure to character feature points in the first reconstructed image according to the annotation point information to obtain a second reconstructed image and skeletal control points in the second reconstructed image.

[0123] In an exemplary embodiment of the present disclosure, an image with a target action is generated based on a second reconstructed image, including: adjusting the posture of the character object in the second reconstructed image based on the skeleton control points; and generating an image with the target action based on the second reconstructed image after the posture is adjusted.

[0124] In an exemplary embodiment of the present disclosure, generating an image with a target action based on the second reconstructed image also includes: extracting a face from the image with the target action; and performing face-changing processing on the extracted face based on a face image corresponding to the original image.

[0125] In an exemplary embodiment of the present disclosure, before obtaining the original image and style image of the character object, identifying the character feature points in the original image for annotation, and obtaining the annotation point information, the method also includes: obtaining at least one initial content image of the character object; performing super-resolution processing on the initial content image, and performing feature similarity evaluation on the initial content image after super-resolution processing and the initial content image; and determining the original image from the initial content image based on the feature similarity evaluation result.

[0126] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 921 and / or a cache memory unit 922 , and may further include a read-only memory unit (ROM) 923 .

[0127] The storage unit 920 may also include a program / utility 924 having a set (at least one) of program modules 925, such program modules 925 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0128] Bus 930 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0129] The electronic device 900 may also communicate with one or more external devices 1000 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 900, and / or communicate with any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 950. Furthermore, the electronic device 900 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 960. As shown, the network adapter 960 communicates with other modules of the electronic device 900 via a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0130] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0131] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0132] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be considered merely exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. An image generation method, characterized in that: include: Acquire an original image and a style image of a character object, identify and annotate the character feature points in the original image, and obtain annotated point information; Based on the style image, the original image is diffused by a diffusion model to obtain a first reconstructed image with a target style; The basic skeleton structure is bound to the first reconstructed image based on the annotation point information to obtain a second reconstructed image, so as to generate an image with the target action based on the second reconstructed image.

2. The method according to claim 1, characterized in that The step of performing diffusion processing on the original image through a diffusion model based on the style image to obtain a first reconstructed image with a target style includes: determining a target network layer from the network structure of the diffusion model; Input the style image and the original image into the diffusion model respectively, obtain a first feature map of the style image at the target network layer, and obtain a second feature map of the original image at the target network layer; Style transfer is performed on the second feature map according to the first feature map to obtain a target feature map, and the target feature map is injected into the diffusion model as an output of the target network layer for processing to obtain the first reconstructed image.

3. The method according to claim 2, characterized in that The target network layer includes at least one first subnetwork and at least one second subnetwork; The performing style transfer on the second feature map according to the first feature map to obtain a target feature map, and injecting the target feature map as the output of the target network layer into the diffusion model for processing to obtain the first reconstructed image, comprises: Based on the first feature map extracted from the style image by the first sub-network, performing style migration on the second feature map extracted from the original image by the first sub-network to obtain a first reference feature map; Performing style migration on a second feature map extracted from the original image by the second sub-network based on a first feature map extracted from the style image by the second sub-network to obtain a second reference feature map; The first reference feature map and the second reference feature map are injected into the diffusion model as the target feature maps for processing to obtain the first reconstructed image.

4. The method according to claim 1, characterized in that: The step of binding the basic bone structure to the first reconstructed image based on the annotation point information to obtain a second reconstructed image includes: Performing data alignment of the annotation point information and the basic bone structure; According to the annotation point information, the basic skeleton structure after data alignment is bound to the character feature points in the first reconstructed image to obtain the second reconstructed image and the skeleton control points in the second reconstructed image.

5. The method according to claim 4, characterized in that The step of generating an image having a target action based on the second reconstructed image comprises: adjusting the posture of the character object in the second reconstructed image based on the skeleton control points; An image having a target motion is generated based on the second reconstructed image after the posture is adjusted.

6. The method according to any one of claims 1 to 5, characterized in that The step of generating an image having a target action based on the second reconstructed image further includes: Performing face extraction on the image with the target action; The extracted face is subjected to face-changing processing based on the face image corresponding to the original image.

7. The method according to claim 6, characterized in that Before obtaining the original image and the style image of the character object, identifying the character feature points in the original image for annotation, and obtaining the annotation point information, the method further includes: Acquire at least one initial content image of the character object; Performing super-resolution processing on the initial content image, and performing feature similarity evaluation on the initial content image after the super-resolution processing and the initial content image; The original image is determined from the initial content images according to the feature similarity evaluation result.

8. An image generating device, characterized in that: The device comprises: An image acquisition module is used to acquire an original image and a style image of a character object, identify and annotate the character feature points in the original image, and obtain annotated point information; An image generation module, configured to perform diffusion processing on the original image through a diffusion model based on the style image to obtain a first reconstructed image with a target style; An image processing module is used to bind the basic bone structure with the first reconstructed image based on the annotation point information to obtain a second reconstructed image, so as to generate an image with a target action based on the second reconstructed image.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

Images