Image generation method, device and non-volatile computer readable storage medium

Abstract

The disclosure relates to a method and device for generating an image and a nonvolatile computer readable storage medium, and relates to the technical field of computers. The method comprises the following steps: extracting a plurality of first contour feature points of a target in a first image; adjusting the plurality of first contour feature points according to the action of the target to be generated, and determining a plurality of second contour feature points; and generating a second image by using an image generation model according to the plurality of second contour feature points. The technical scheme of the disclosure can improve the quality of the generated image.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a method for generating an image, an apparatus for generating an image and a nonvolatile computer readable storage medium. BACKGROUND

[0002] The customization of interactive digital human images pursues authenticity and individualization. Under the requirement of a photograph-level super-realistic, every detail of the digital human image will be concerned by the user. This puts forward higher requirements for the model when recording the image material.

[0003] However, the model is not a robot after all, and cannot completely match the interactive scene used by the image in terms of time and action positioning. Therefore, the action editing technology can be used to generate the digital human action matching the interactive scene.

[0004] In the related art, action editing can be implemented based on 3D character modeling or 2D image editing to generate an image containing a target action. SUMMARY

[0005] The present inventors have found that the above-mentioned related art has the following problems: the action in the generated image is not accurate, and phenomena such as deformation and distortion easily occur, resulting in a decrease in the quality of the generated image.

[0006] In view of this, the present disclosure provides a technical scheme for generating an image, which can improve the quality of the generated image.

[0007] According to some embodiments of the present disclosure, a method for generating an image is provided, including: extracting a plurality of first contour feature points of a target in a first image; adjusting the plurality of first contour feature points according to an action of the target required to be generated, to determine a plurality of second contour feature points; and generating a second image by using an image generation model according to the plurality of second contour feature points.

[0008] In some embodiments, the extracting of the plurality of first contour feature points of the target in the first image includes: extracting a contour line of the target in the first image and a plurality of key points on the target; and determining the plurality of first contour feature points on the contour line according to the contour line and the plurality of key points.

[0009] In some embodiments, the determining of the plurality of first contour feature points on the contour line according to the contour line and the plurality of key points includes: connecting the plurality of key points to determine a plurality of key connecting lines according to structure information of the target; and determining the plurality of first contour feature points according to the plurality of key connecting lines and the contour line.

[0010] In some embodiments, the determining of the plurality of first contour feature points according to the plurality of key connecting lines and the contour line includes: determining the plurality of first contour feature points according to the intersection points of the perpendicular lines of the plurality of key connecting lines and the contour line.

[0011] In some embodiments, the extracting the contour line of the target in the first image and the plurality of key points on the target comprises: extracting the contour line of the target in the first image by using a semantic segmentation network model; and extracting the plurality of key points on the target by using a target detection network model.

[0012] In some embodiments, the generating the second image according to the plurality of second contour feature points by using the image generation model comprises: extracting feature information of a neighborhood of each first contour feature point in the first image by using a feature extraction model; and generating the second image according to the feature information and the plurality of second contour feature points by using the image generation model.

[0013] In some embodiments, the feature extraction model is a convolution kernel model.

[0014] In some embodiments, the first image is a frame image in a first dynamic video, the plurality of second contour feature points are determined by adjusting the plurality of first contour feature points according to actions of the target to be generated, and the generating the second image according to the plurality of second contour feature points by using the image generation model comprises: generating a plurality of second images as a plurality of key frame images by using the image generation model according to a plurality of second contour feature point groups, each of which comprises a plurality of second contour feature points; the method further comprises: generating a plurality of transition frame images between the plurality of key frame images by using a video interpolation algorithm; and generating a second dynamic video comprising the plurality of key frame images and the plurality of transition frame images.

[0015] In some embodiments, the generating the plurality of transition frame images between the plurality of key frame images by using the video interpolation algorithm comprises: determining a first key frame image as a starting frame image of a current iteration and a second key frame image as an ending frame image of the current iteration in the plurality of key frame images; generating transition frame images of the current iteration between the starting frame image of the current iteration and the ending frame image of the current iteration by using the video interpolation algorithm; determining the transition frame images of the current iteration as an ending frame image of a next iteration, generating transition frame images of the next iteration between the starting frame image of the current iteration and the ending frame image of the next iteration by using the video interpolation algorithm, and repeating the step until an iteration condition is met; determining the transition frame images of the current iteration as a starting frame image of the next iteration, generating transition frame images of the next iteration between the starting frame image of the next iteration and the ending frame image of the current iteration by using the video interpolation algorithm, and repeating the step until the iteration condition is met.

[0016] In some embodiments, the generating method further comprises: generating, according to the plurality of contour feature point samples of the first image sample, a second image sample by using the image generation model; and calculating a generation loss function according to a difference between the second image sample and the first image sample; and training the image generation model by using the generation loss function.

[0017] In some embodiments, the generating method further comprises: extracting, by using a feature extraction model, a feature information sample of a neighborhood of each contour feature point sample in the first image sample; and training the image generation model by using the generation loss function comprises: calculating a contrast loss function according to the feature information sample; and training the image generation model, the feature extraction model and an adversarial network model by using the contrast loss function and the generation loss function, the adversarial network model being configured to distinguish between the first image sample and the second image sample.

[0018] According to some other embodiments of the present disclosure, there is provided a generating device of an image, comprising: an extraction unit configured to extract a plurality of first contour feature points of a target in a first image; a determination unit configured to adjust the plurality of first contour feature points according to an action of a target to be generated, and determine a plurality of second contour feature points; and a generation unit configured to generate a second image by using an image generation model according to the plurality of second contour feature points.

[0019] In some embodiments, the generation unit generates a second image sample by using the image generation model according to the plurality of contour feature point samples of the first image sample; and the generating device further comprises a training unit configured to calculate a generation loss function according to a difference between the second image sample and the first image sample, and train the image generation model by using the generation loss function.

[0020] In some embodiments, the extraction unit extracts a feature information sample of a neighborhood of each contour feature point sample in the first image sample by using a feature extraction model; the training unit calculates a contrast loss function according to the feature information sample, and trains the image generation model, the feature extraction model and an adversarial network model by using the contrast loss function and the generation loss function, the adversarial network model being configured to distinguish between the first image sample and the second image sample.

[0021] In some embodiments, the extraction unit extracts a contour line of the target in the first image and a plurality of key points on the target, and determines the plurality of first contour feature points on the contour line according to the contour line and the plurality of key points.

[0022] In some embodiments, the extraction unit connects the plurality of key points to determine a plurality of key connecting lines according to structure information of the target, and determines the plurality of first contour feature points according to the plurality of key connecting lines and the contour line.

[0023] In some embodiments, the extraction unit determines the plurality of first contour feature points according to intersection points of perpendicular lines of the plurality of key connecting lines and the contour line.

[0024] In some embodiments, the extraction unit extracts, by using a semantic segmentation network model, the contour line of the target in the first image; and extracts, by using a target detection network model, a plurality of key points on the target.

[0025] In some embodiments, the extraction unit extracts, by using a feature extraction model, feature information of a neighborhood of each first contour feature point in the first image; and the generation unit generates, by using an image generation model, the second image according to the feature information and the plurality of second contour feature points.

[0026] In some embodiments, the feature extraction model is a convolution kernel model.

[0027] In some embodiments, the determination unit determines a plurality of second contour feature point groups according to a plurality of actions of the target to be generated, each second contour feature point group including a plurality of second contour feature points; the generation unit generates a plurality of second images as a plurality of key frame images respectively by using the image generation model according to the plurality of second contour feature point groups; the generation unit generates a plurality of transition frame images between the plurality of key frame images by using a video interpolation algorithm; and the generation unit generates a second dynamic video including the plurality of key frame images and the plurality of transition frame images.

[0028] In some embodiments, the determination unit determines a first key frame image as a starting frame image of the current iteration and a second key frame image as an ending frame image of the current iteration among the plurality of key frame images; the generation unit generates a transition frame image of the current iteration between the starting frame image of the current iteration and the ending frame image of the current iteration by using the video interpolation algorithm; the determination unit determines the transition frame image of the current iteration as an ending frame image of a next iteration, the generation unit generates a transition frame image of the next iteration between the starting frame image of the current iteration and the ending frame image of the next iteration by using the video interpolation algorithm, and the determination unit and the generation unit repeat the step until an iteration condition is met; the determination unit determines the transition frame image of the current iteration as a starting frame image of the next iteration, the generation unit generates a transition frame image of the next iteration between the starting frame image of the next iteration and the ending frame image of the current iteration by using the video interpolation algorithm, and the determination unit and the generation unit repeat the step until the iteration condition is met.

[0029] According to yet some embodiments of the present disclosure, there is provided an image generation apparatus, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute an image generation method in any one of the above embodiments based on instructions stored in the memory.

[0030] According to still some embodiments of the present disclosure, there is provided a non-volatile computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements an image generation method in any one of the above embodiments.

[0031] In the above embodiments, by adjusting the feature points on the target contour, the action intended to be generated is determined, and then an image containing the action is generated. In this way, the action intended to be generated can be generated more accurately, and phenomena such as deformation and distortion are less likely to occur, thereby improving the quality of the generated image. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings, which constitute a part of this specification, illustrate embodiments of the present disclosure and serve to explain the principles of the present disclosure.

[0033] The present disclosure can be more clearly understood and appreciated from the following detailed description, taken in conjunction with the following drawings of which:

[0034] Figure 1 Flowcharts illustrating some embodiments of the image generation method of the present disclosure;

[0035] Figures 2a-2d Schematic diagrams illustrating some embodiments of the image generation method of the present disclosure;

[0036] Figures 3a-3b Flowcharts illustrating some other embodiments of the image generation method of the present disclosure;

[0037] Figure 4 Block diagrams illustrating some embodiments of the image generation apparatus of the present disclosure;

[0038] Figure 5 Block diagrams illustrating some other embodiments of the image generation apparatus of the present disclosure;

[0039] Figure 6 Block diagrams illustrating some other embodiments of the image generation apparatus of the present disclosure. DETAILED DESCRIPTION

[0040] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. Note that the relative arrangement, numerical expressions, and numerical values of components and steps set forth in these embodiments are not limiting to the scope of the present disclosure unless otherwise specifically stated.

[0041] At the same time, it should be understood that the sizes of the various parts shown in the drawings are not drawn in accordance with the actual proportional relationship for the convenience of description.

[0042] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way limiting to the scope of the present disclosure and its applications or uses.

[0043] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail herein, but should be considered part of the specification, where appropriate.

[0044] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary, and not as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0045] It should be noted that like reference numerals and letters refer to like items throughout the attached drawings, and once an item is defined in one drawing, it is not necessary to discuss it further in subsequent drawings.

[0046] As mentioned above, the final rendering effect of the animation technology based on 3D human modeling cannot reach the level of photographic realism. The 2D image editing technology has the following technical problems.

[0047] In terms of pose generation, the editing capability of the 2D image editing technology is limited. For some states with large adjustments, the body of the character is blocked, and the like, the presentation effect is not good, and distortion and deformation are prone to occur; the level of detail is not enough, the output pixels are low, the edge profile of the character is rough, and details such as fingers and hair are missing and blurred. Manual intervention is required for post-processing.

[0048] In terms of motion video continuity, the effect of the 2D image editing technology is unstable. Small errors in adjusting control points can easily cause larger flaws in the effect, and technical problems such as frame-to-frame jumps, incoherent motion, motion speed not conforming to human behavior rules, and strong mechanical feeling often occur when the motion is connected.

[0049] In terms of algorithm generalization, the 2D image editing technology has poor generalization. Editing the motion of a single image requires a large amount of training data for various motions of the image. As a result, the cost of data collection is high, and the computational cost and cycle of model training are long.

[0050] To solve the above technical problems, the present disclosure controls the generation of character motion by formulating a set of character contour feature points based on a small amount of recorded character motion data. In this way, the following advantages are mainly achieved.

[0051] On the one hand, by adjusting the contour feature points, the pose of the character can be effectively adjusted, and a high-definition (such as a resolution of 1080p) character motion pose image with a character ratio of more than 1 / 4 can be generated. The details such as the body contour and the hand pose are well presented, and deformation or edge blur that violates the rules of human body structure is not prone to occur.

[0052] On the other hand, the generated motion is naturally coherent without obvious jumps or mechanical feeling of the motion. Moreover, uniform speed changes in the position of the contour feature points can make the character motion uniform, and the effect of the technical solution of the present disclosure is stable.

[0053] In addition, for action editing of a single character, the model only needs to perform recording in a specified action according to requirements, and a small number (for example, about 10) of frames are recorded, so as to complete data acquisition. The required training data is only several minutes (for example, 8-10 minutes) of character action videos, which reduces the performance requirements of the model and the professional requirements of photography, thereby reducing the production cost of digital human action.

[0054] For example, the technical solutions of the present disclosure can be implemented through the following embodiments.

[0055] Figure 1 A flowchart showing some embodiments of the image generation method of the present disclosure is shown.

[0056] As shown in step 110, a plurality of first contour feature points of a target in a first image are extracted. Figure 1

[0057] In some embodiments, the first image is a frame image in a first dynamic image, and the first dynamic image can be a video or a dynamic picture (such as a picture in GIF (Graphics Interchange Format) format).

[0058] For example, different application scenarios can be edited according to actions, and the first dynamic image containing corresponding actions is recorded. For a single character, the corresponding actions can be 8-12, and the recording time length can be 8-10 minutes.

[0059] For example, the application scenario is action editing of an interactive digital human for a customer service scenario. The corresponding actions include holding hands in front of the body, laying one hand on one side, waving a hand, making a heart sign, rotating the head around a ring, laying both hands on both sides, etc.; and the recording time length of the first dynamic image is about 8 minutes. Based on the frame images in the dynamic image, most of the action images used in the customer service scenario can be edited by using the technical solutions of the present disclosure.

[0060] In some embodiments, the contour line of the target in the first image and a plurality of key points on the target are extracted; and the plurality of first contour feature points on the contour line are determined according to the contour line and the plurality of key points. For example, a semantic segmentation network model is used to extract the contour line of the target in the first image; and a target detection network model is used to extract the plurality of key points on the target.

[0061] For example, based on the first dynamic image recorded in the above steps, semantic segmentation and human key point detection are performed by using a deep learning related algorithm; and the positions of the plurality of first contour feature points are calculated by using the detected key points and the contour line extracted by human segmentation.

[0062] Figures 2a-2b A schematic diagram showing some embodiments of the image generation method of the present disclosure is shown.​

[0063] As shown in Figure 2a , according to the structure information of the target (such as the human body structure information), a plurality of key points (black dots in the figure) are connected to determine a plurality of key lines (white straight lines in the figure).

[0064] As shown in Figure 2b , according to the plurality of key lines and the contour line, a plurality of first contour feature points (white dots in the figure) are determined. For example, according to the intersection of the perpendicular lines of the plurality of key lines and the contour line, a plurality of first contour feature points are determined.

[0065] For example, the first dynamic image can be subjected to semantic segmentation frame by frame to extract the contour of the human body. The semantic segmentation algorithm based on the neural network can be used; or the method of limiting the shooting environment first and then eliminating the background curtain can be used.

[0066] For example, the key point detection model can be used to extract the key points on the human body; the perpendicular lines of the lines between the key points are drawn and extended to the contour line to obtain the intersection of the perpendicular lines and the contour line to determine the position of the first contour feature point.

[0067] In the above embodiment, taking the customer service scene as an example, 14 pairs of first contour feature points can be obtained through this step. The contour feature points obtained in this way are more sufficient in expressing the edge information and more detailed in generating human body actions, and the effect is better than directly editing the actions with the key points on the human body (such as the key points on the skeleton model).

[0068] In the case where a plurality of first contour feature points are determined, the remaining steps in Figure 1 can be used to generate images.

[0069] In step 120, according to the action of the target to be generated, the plurality of first contour feature points are adjusted to determine a plurality of second contour feature points.

[0070] For example, the embodiment in Figure 2c can be used to implement step 120.

[0071] Figure 2c The schematic diagram shows some embodiments of the image generation method of the present disclosure.

[0072] As shown in Figure 2c , after obtaining a plurality of first contour feature points based on the frame image in any one of the recorded videos, according to the action to be obtained, the positions of the first contour feature points are moved to generate a plurality of second contour feature points Figure 2c(White dots in the middle). For example, the first contour feature points can be moved in pairs, that is, a pair of second contour feature points determined by a key line can be moved simultaneously each time to avoid target stretching and deformation, thereby improving image quality.

[0073] The adjusted second contour feature points are input into the image generation model to generate keyframe images of the desired action. Repeating steps 120 and 130 yields multiple keyframe images of the desired action.

[0074] For example, with Figure 2b Using the first contour feature point of the mid-hip as a reference fixed point, by relatively shifting the three pairs of first contour feature points of the head downwards, the digital human figure can make a head-down movement, forming... Figure 2c The three pairs of second contour feature points in the middle head; Figure 2b The three primary contour feature points of the right hand were moved to the front of the chest and adjusted accordingly. Figure 2b A pair of first contour feature points at the mid-elbow area can be used to generate... Figure 2c The figure in the painting is depicted touching its chest.

[0075] Given multiple second contour feature points, it is possible to... Figure 1 The remaining steps generate the image.

[0076] In step 130, a second image is generated based on multiple second contour feature points using an image generation model. For example, generating... Figure 2c Images of human postures in the image.

[0077] In some embodiments, a feature extraction model is used to extract feature information of the neighborhood of each first contour feature point in the first image; based on the feature information and multiple second contour feature points, an image generation model is used to generate a second image. For example, the feature extraction model is a convolutional kernel model.

[0078] In some embodiments, a neural network model or a convolutional kernel model (such as an 11×11 convolutional kernel) can be used as the feature extraction model. The feature extraction model is used to extract feature information within the neighborhood of each first contour feature point in the first image.

[0079] In some embodiments, the first image is a frame image in the first dynamic image. According to the multiple actions of the target to be generated, multiple second contour feature point groups are determined respectively, each second contour feature point group including multiple second contour feature points. According to the multiple second contour feature point groups, multiple second images are generated as multiple key frame images respectively by using an image generation model. Multiple transition frame images are generated between the multiple key frame images by using a video interpolation algorithm. A second dynamic image including the multiple key frame images and the multiple transition frame images is generated. For example, the second dynamic image can be a video or a dynamic picture (such as a picture in GIF format).

[0080] In some embodiments, in the multiple key frame images, a first key frame image is determined as a starting frame image of the current iteration, and a second key frame image is determined as an ending frame image of the current iteration. Transition frame images of the current iteration between the starting frame image of the current iteration and the ending frame image of the current iteration are generated by using the video interpolation algorithm. The transition frame images of the current iteration are determined as ending frame images of a next iteration, and transition frame images of the next iteration between the starting frame image of the current iteration and the ending frame images of the next iteration are generated by using the video interpolation algorithm. The step is repeated until an iteration condition is met. The transition frame images of the current iteration are determined as starting frame images of the next iteration, and transition frame images of the next iteration between the starting frame images of the next iteration and the ending frame images of the current iteration are generated by using the video interpolation algorithm. The step is repeated until the iteration condition is met.

[0081] For example, the step 130 can be implemented by the embodiments in the method 100. Figure 2d

[0082] Figure 2d The schematic diagram of some embodiments of the image generation method of the present disclosure is shown.

[0083] As shown in the method 100, the adjusted second contour feature points are input into the image generation model to generate a key frame image of the desired action. The steps 120 and 130 are repeated to obtain multiple key frame images (key frame 1-3) of the desired action. Figure 2d

[0084] In some embodiments, for a complete set of actions to be generated, the present disclosure only needs to select several key actions and determine the corresponding second contour feature point groups for generating the corresponding key frame images of the several key actions. The actions between the key frame images are transition frame image parts, which can be generated by action connection processing, and there is no need to generate second contour feature point groups and frame images of all actions.

[0085] In some embodiments, the video interpolation algorithm is used to insert the corresponding transition frame images between two key frame images to complete the image generation of the entire coherent action.​​

[0086] For example, taking the initial starting frame image and the ending frame image as input, a transition frame image is obtained by using a deep learning-based video frame interpolation algorithm (such as DAIN, etc.) for interpolation processing; taking the starting frame image and the transition frame image or the transition frame image and the ending frame image as input, respectively, interpolation processing is continued; such iterative iteration obtains an image sequence after interpolation processing.

[0087] For example, the number of inserted transition frame images is determined by the offset amount of the contour feature points between two frames. If there is no contour feature point with an offset amount greater than a threshold between two frames, interpolation processing is stopped, and a motion video frame sequence is obtained.

[0088] In some embodiments, according to a plurality of contour feature point samples of the first image sample, an image generation model is used to generate a second image sample; a generation loss function is calculated according to the difference between the second image sample and the first image sample; and the image generation model is trained using the generation loss function.

[0089] In some embodiments, a feature extraction model is used to extract feature information samples of the neighborhood of each contour feature point sample in the first image sample; a contrast loss function is calculated according to the feature information samples; and the image generation model, the feature extraction model, and the adversarial network model are trained using the contrast loss function and the generation loss function, and the adversarial network model is used to distinguish between the first image sample and the second image sample.

[0090] For example, for the generation model, domain transfer learning can be performed based on GAN (Generative Adversarial Networks). Through learning of training data, a mapping relationship is established between the position of each first contour feature point, the feature information of the neighborhood, and the image containing the corresponding motion. Thus, the motion editing effect of changing the human posture by moving the contour feature point position is achieved.

[0091] For example, a recorded first image sample containing a desired generated motion and all contour feature point samples obtained based on the first image sample can be taken as a pair of training data. Each contour feature point sample is taken as input of the image generation model, and the first image sample is taken as supervision information (i.e., annotation result).

[0092] The difference between the second image sample generated by the image generation model and the first image sample is calculated pixel by pixel; a generation loss function is calculated according to the difference (pixel to pixel loss); a contrast loss function is calculated according to the feature information in the neighborhood; and the generation adversarial network is trained using the generation loss function and the contrast loss function.

[0093] Thus, by training, an image generation model for obtaining a human pose image containing a desired action can be obtained.

[0094] Figures 3a-3b A flowchart showing another embodiment of the image generation method of the present disclosure.

[0095] As Figures 3a-3b shown, in the character action recording step, the main purpose is to provide training data for the subsequent image generation model, so as to complete the generation of the image of the specified action of the character.

[0096] Different application scenarios can be edited according to the action, and the first dynamic image containing the corresponding action is specified to be recorded. For a single character image, the corresponding action can be 8-12, and the recording time length can be 8-10 minutes.

[0097] For example, the application scenario is the action editing of the interactive digital person for the customer service scene. The corresponding actions include holding hands in front of the body, laying one side of the hand flat, waving a hand, making a heart, rotating the head around the ring, laying both sides of the hands flat, etc.; the time length of the first dynamic image recorded is about 8 minutes. Based on the frame image in the dynamic image, most of the action images used in the customer service scene can be edited by using the technical solution of the present disclosure.

[0098] In the feature point detection step, as Figure 3b shown, semantic segmentation can be performed on the first dynamic image frame by frame to extract the contour of the human body. It can be realized based on the semantic segmentation algorithm of the neural network; it can also be realized by limiting the shooting environment first and then eliminating the background curtain. For example, the key point detection model can be used to extract the key points on the human body; the center lines of the connecting lines between the key points are drawn, and the center lines are extended to the contour lines to obtain the intersection points of the center lines and the contour lines to determine the positions of the first contour feature points.

[0099] In some embodiments, a neural network model can be used as a feature extraction model, and a convolution kernel model (such as an 11x11 convolution kernel), i.e., a descriptor, can also be used as a feature extraction model to express the features of the contour feature points. The feature extraction model is used to extract the feature information of each first contour feature point in the neighborhood of the first image.

[0100] In the above embodiment, taking the customer service scene as an example, 14 pairs of first contour feature points can be obtained through this step. The contour feature points obtained in this way are more sufficient and detailed in expressing the edge information of the human action generation, and the effect is better than directly editing the action with the key points on the human body (such as the key points on the skeleton model).

[0101] In the generative model training process, cross-domain transfer learning can be performed based on GANs (Generative Adversarial Networks). By learning from the training data, a mapping relationship is established between the position of each first contour feature point, the feature information of its neighborhood, and the image containing the corresponding action. This allows for the motion editing effect of changing the human pose simply by moving the contour feature points.

[0102] For example, a recorded first image sample containing the desired action can be used as a pair of training data along with all contour feature point samples obtained based on that first image sample. The contour feature point samples are then used as input to the image generation model, while the first image sample serves as supervision information (i.e., annotation results).

[0103] The difference between the second image sample generated by the image generation model and the first image sample is calculated pixel by pixel; the generation loss function (pixel to pixel loss) is calculated based on the difference; the contrast loss function is calculated based on the feature information in the neighborhood; and the generative adversarial network is trained using the generation loss function and the contrast loss function.

[0104] In this way, through training, an image generation model can be obtained that captures a human pose corresponding to the desired action based on a set of contour feature points.

[0105] In the motion editing, motion splicing, and motion video generation steps, after obtaining multiple first contour feature points based on frame images from any recorded video, the positions of each first contour feature point are moved according to the desired motion to generate multiple second contour feature points. Figure 2c (White dots in the middle). For example, the first contour feature points can be moved in pairs, that is, a pair of second contour feature points determined by a key line can be moved simultaneously each time to avoid target stretching and deformation, thereby improving image quality.

[0106] The adjusted second contour feature points are input into the image generation model to generate keyframe images of the desired action. By repeating the action editing and action stitching steps, multiple keyframe images of the desired action can be obtained.

[0107] For example, with Figure 2b Using the first contour feature point of the mid-hip as a reference fixed point, by relatively shifting the three pairs of first contour feature points of the head downwards, the digital human figure can make a head-down movement, forming... Figure 2c The three pairs of second contour feature points in the middle head; Figure 2b The three primary contour feature points of the right hand were moved to the front of the chest and adjusted accordingly. Figure 2bA pair of first contour feature points in the middle elbow part, i.e. Figure 2c A motion of the middle character figure patting the chest.

[0108] Figure 4 A block diagram showing some embodiments of the image generation apparatus of the present disclosure.

[0109] As Figure 4 shown, the image generation apparatus 4 includes an extraction unit 41 configured to extract a plurality of first contour feature points of a target in a first image; a determination unit 42 configured to determine a plurality of second contour feature points by adjusting the plurality of first contour feature points according to a motion of the target to be generated; and a generation unit 43 configured to generate a second image by using an image generation model according to the plurality of second contour feature points.

[0110] In some embodiments, the generation unit 43 generates a second image sample by using the image generation model according to a plurality of contour feature point samples of a first image sample; and the image generation apparatus 4 further includes a training unit 44 configured to calculate a generation loss function according to a difference between the second image sample and the first image sample, and train the image generation model by using the generation loss function.

[0111] In some embodiments, the extraction unit 41 extracts a feature information sample of a neighborhood of each contour feature point sample in the first image sample by using a feature extraction model; and the training unit 44 calculates a contrast loss function according to the feature information sample, and trains the image generation model, the feature extraction model, and an adversarial network model by using the contrast loss function and the generation loss function, the adversarial network model being configured to distinguish between the first image sample and the second image sample.

[0112] In some embodiments, the extraction unit 41 extracts a contour line of the target in the first image and a plurality of key points on the target, and determines the plurality of first contour feature points on the contour line according to the contour line and the plurality of key points.

[0113] In some embodiments, the extraction unit 41 connects the plurality of key points to determine a plurality of key connecting lines according to structure information of the target, and determines the plurality of first contour feature points according to the plurality of key connecting lines and the contour line.

[0114] In some embodiments, the extraction unit 41 determines the plurality of first contour feature points according to intersection points of perpendicular lines of the plurality of key connecting lines and the contour line.

[0115] In some embodiments, the extraction unit 41 extracts the contour line of the target in the first image by using a semantic segmentation network model, and extracts the plurality of key points on the target by using a target detection network model.

[0116] In some embodiments, the extraction unit 41 extracts feature information of a neighborhood of each first contour feature point in the first image using a feature extraction model; and the generation unit 43 generates the second image according to the feature information and the plurality of second contour feature points using an image generation model.

[0117] In some embodiments, the feature extraction model is a convolution kernel model.

[0118] In some embodiments, the determination unit 42 determines a plurality of second contour feature point groups according to a plurality of actions of a target to be generated, each second contour feature point group including a plurality of second contour feature points; the generation unit 43 generates a plurality of second images as a plurality of key frame images respectively according to the plurality of second contour feature point groups using the image generation model; the generation unit 43 generates a plurality of transition frame images between the plurality of key frame images using a video interpolation algorithm; and the generation unit 43 generates a second dynamic video including the plurality of key frame images and the plurality of transition frame images.

[0119] In some embodiments, the determination unit 42 determines a first key frame image as a starting frame image of a current iteration and a second key frame image as an ending frame image of the current iteration in the plurality of key frame images; the generation unit 43 generates a transition frame image of the current iteration between the starting frame image of the current iteration and the ending frame image of the current iteration using the video interpolation algorithm; the determination unit 42 determines the transition frame image of the current iteration as an ending frame image of a next iteration, the generation unit 43 generates a transition frame image of the next iteration between the starting frame image of the current iteration and the ending frame image of the next iteration using the video interpolation algorithm, and the determination unit 42 and the generation unit 43 repeat the step until an iteration condition is met; the determination unit 42 determines the transition frame image of the current iteration as a starting frame image of the next iteration, the generation unit 43 generates a transition frame image of the next iteration between the starting frame image of the next iteration and the ending frame image of the current iteration using the video interpolation algorithm, and the determination unit 42 and the generation unit 43 repeat the step until the iteration condition is met.

[0120] Figure 5 A block diagram showing another embodiment of the image generation apparatus of the present disclosure.

[0121] As Figure 5 shown, the image generation apparatus 5 of this embodiment includes a memory 51 and a processor 52 coupled to the memory 51, the processor 52 being configured to execute the image generation method in any one of the embodiments of the present disclosure based on instructions stored in the memory 51.

[0122] The memory 51 can include, for example, a system memory, a fixed nonvolatile memory medium, and the like. The system memory stores, for example, an operating system, an application program, a Boot Loader, a database, and other programs.

[0123] Figure 6 A block diagram illustrating still another embodiment of the image generation apparatus of the present disclosure is shown.

[0124] As Figure 6 shown, the image generation apparatus 6 of this embodiment includes a memory 610 and a processor 620 coupled to the memory 610, the processor 620 being configured to execute the image generation method of any one of the preceding embodiments based on instructions stored in the memory 610.

[0125] The memory 610 can include, for example, a system memory, a fixed nonvolatile memory medium, and the like. The system memory stores, for example, an operating system, an application program, a Boot Loader, and other programs.

[0126] The image generation apparatus 6 can further include an input / output interface 630, a network interface 640, a storage interface 650, and the like. These interfaces 630, 640, 650, and the memory 610 and the processor 620 can be connected, for example, through a bus 660. The input / output interface 630 provides a connection interface for display, mouse, keyboard, touch screen, microphone, speaker, and other input / output devices. The network interface 640 provides a connection interface for various networking devices. The storage interface 650 provides a connection interface for external storage devices such as SD card and U disk.

[0127] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable non-transitory storage media including, but not limited to, disk storage, CD-ROMs, optical storage devices, and the like, containing computer usable program code.

[0128] So far, the image generation method, the image generation apparatus, and the non-volatile computer-readable storage medium according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein based on the above description.

[0129] The methods and systems of the present disclosure can be implemented in a number of ways. For example, the methods and systems of the present disclosure can be implemented via software, hardware, firmware, or any combination of software, hardware, and firmware. The above described order of steps for the methods is merely for illustration, and the steps of the methods of the present disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, the present disclosure can also be implemented as a program recorded in a recording medium, which includes machine readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the methods according to the present disclosure.

[0130] While certain specific embodiments of the present disclosure have been described in detail by way of example, it should be understood that the examples are for illustration only and should not be taken as limiting the scope of the present disclosure. Those skilled in the art will understand that modifications can be made to the above-described embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for generating an image, comprising: extracting a plurality of first contour feature points of a target in a first image, the first image being a frame image in a first dynamic video; adjusting the plurality of first contour feature points according to a motion of the target to be generated, to determine a plurality of second contour feature points; generating a second image by using an image generation model according to the plurality of second contour feature points, the adjusting the plurality of first contour feature points according to a motion of the target to be generated, to determine a plurality of second contour feature points comprises: determining a plurality of second contour feature point groups respectively according to a plurality of motions of the target to be generated, each second contour feature point group comprising a plurality of second contour feature points; the generating a second image by using an image generation model according to the plurality of second contour feature points comprises: generating a plurality of second images respectively as a plurality of key frame images by using the image generation model according to the plurality of second contour feature point groups; the method further comprises: generating a plurality of transition frame images between the plurality of key frame images by using a video interpolation algorithm, a number of the plurality of transition frame images being determined according to a displacement of contour feature points between the plurality of key frame images; generating a second dynamic video comprising the plurality of key frame images and the plurality of transition frame images.

2. The generation method of claim 1, wherein, the extracting a plurality of first contour feature points of a target in a first image comprises: extracting a contour line of the target in the first image and a plurality of key points on the target; determining the plurality of first contour feature points on the contour line according to the contour line and the plurality of key points.

3. The generation method of claim 2, wherein, the determining the plurality of first contour feature points on the contour line according to the contour line and the plurality of key points comprises: connecting the plurality of key points to determine a plurality of key connecting lines according to structure information of the target; determining the plurality of first contour feature points according to the plurality of key connecting lines and the contour line.

4. The generation method of claim 3, wherein, the determining the plurality of first contour feature points according to the plurality of key connecting lines and the contour line comprises: determining the plurality of first contour feature points according to intersection points of perpendicular lines of the plurality of key connecting lines and the contour line.

5. The generation method of claim 2, wherein, the extracting a contour line of the target in the first image and a plurality of key points on the target comprises: extracting the contour line of the target in the first image by using a semantic segmentation network model; extracting the plurality of key points on the target by using a target detection network model.

6. The generation method of claim 1, wherein, the generating a second image by using an image generation model according to the plurality of second contour feature points comprises: extracting feature information of a neighborhood of each first contour feature point in the first image by using a feature extraction model; generating the second image by using the image generation model according to the feature information and the plurality of second contour feature points.

7. The generation method of claim 6, wherein, the feature extraction model is a convolution kernel model.

8. The generation method of claim 1, wherein, the generating a plurality of transition frame images between the plurality of key frame images by using a video interpolation algorithm comprises: determining a first key frame image as a starting frame image of a current iteration and a second key frame image as an ending frame image of the current iteration in the plurality of key frame images; generating transition frame images of the current iteration between the starting frame image of the current iteration and the ending frame image of the current iteration using a video interpolation algorithm; determining the transition frame image of the current iteration as the ending frame image of the next iteration, generating transition frame images of the next iteration between the starting frame image of the current iteration and the ending frame image of the next iteration using the video interpolation algorithm, and repeating the step until an iteration condition is met; determining the transition frame image of the current iteration as the starting frame image of the next iteration, generating transition frame images of the next iteration between the starting frame image of the next iteration and the ending frame image of the current iteration using the video interpolation algorithm, and repeating the step until an iteration condition is met.

9. The generation method of any one of claims 1-8, further comprising: generating a second image sample from a plurality of contour feature point samples of a first image sample using the image generation model; calculating a generation loss function according to a difference between the second image sample and the first image sample; training the image generation model using the generation loss function.

10. The generation method of claim 9, further comprising: extracting feature information samples of a neighborhood of each contour feature point sample in the first image sample using a feature extraction model; wherein the training the image generation model using the generation loss function comprises: calculating a contrast loss function according to the feature information samples; training the image generation model, the feature extraction model, and an adversarial network model for distinguishing the first image sample and the second image sample using the contrast loss function and the generation loss function.

11. An image generation apparatus, comprising: an extraction unit configured to extract a plurality of first contour feature points of a target in a first image, the first image being a frame image in a first dynamic image; a determination unit configured to adjust the plurality of first contour feature points according to a motion of the target to be generated, and determine a plurality of second contour feature points; a generation unit configured to generate a second image from the plurality of second contour feature points using an image generation model, wherein the determination unit determines a plurality of second contour feature point groups according to a plurality of motions of the target to be generated, each second contour feature point group including a plurality of second contour feature points; the generation unit generates a plurality of second images as a plurality of key frame images from the plurality of second contour feature point groups using the image generation model, the generation unit generates a plurality of transition frame images between the plurality of key frame images using a video interpolation algorithm, a number of the plurality of transition frame images being determined according to a displacement of contour feature points between the plurality of key frame images, and generates a second dynamic image including the plurality of key frame images and the plurality of transition frame images.

12. The generation apparatus of claim 11, wherein: the generation unit generates a second image sample from a plurality of contour feature point samples of a first image sample using the image generation model; the generation apparatus further comprises: The training unit is configured to calculate a generation loss function according to a difference between the second image sample and the first image sample, and train the image generation model by using the generation loss function.

13. The generation apparatus of claim 12, wherein, The extraction unit is configured to extract, by using a feature extraction model, a feature information sample of a neighborhood of each contour feature point sample in the first image sample; The training unit is configured to calculate a contrast loss function according to the feature information sample, and train the image generation model, the feature extraction model, and an adversarial network model by using the contrast loss function and the generation loss function, the adversarial network model being configured to distinguish the first image sample from the second image sample.

14. An image generation apparatus, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the image generation method of any one of claims 1-10 based on instructions stored in the memory.

15. A non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the image generation method of any one of claims 1-10.

Images