Visual data generation method, device, electronic device, computer-readable storage medium, and computer program product

Abstract

The present application provides a method, device, electronic device, computer-readable storage medium and computer program product for generating visual data; the method includes: extracting features from character data to obtain character features of a first object in the character data; extracting features from layout data to obtain reference features of a second object in the layout data; performing multiple diffusion processes based on character features, reference features and image noise to obtain generated content features after each diffusion process; obtaining target visual data based on the generated content features after the last diffusion process; the target visual data includes the character features of the first object, the reference features of the second object and the semantic features of the layout data. Through the present application, the character consistency in the generated video or generated image can be enhanced, and the image generation quality can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, electronic device, computer-readable storage medium and computer program product for generating visual data. Background Art

[0002] In the related art, when a picture containing a person (called a layout diagram) is given and the person in the layout diagram is replaced with a specified character diagram, in order to ensure that the person in the generated image is consistent with the character diagram, a commonly used method is to use the position and posture of the person in the layout diagram and the feature information of the specified character diagram to jointly control the generation of the final image.

[0003] Since the related technology needs to comprehensively consider multiple conditions to control the output, especially when it is necessary to highlight the role consistency of the characters, it will be difficult to highlight the control effect of the semantics of the layout diagram, thus affecting the accuracy of the semantics of the final generated image, thereby reducing the character consistency during image generation and reducing the quality of the generated image. Summary of the invention

[0004] The embodiments of the present application provide a visual data generation method, device, electronic device, computer-readable storage medium and computer program product, which can enhance the consistency of characters in generated videos or generated images and improve the quality of image generation.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] An embodiment of the present application provides a method for generating visual data, the method comprising: extracting features from character data to obtain character features of a first object in the character data; extracting features from layout data to obtain reference features of a second object in the layout data; performing multiple diffusion processes based on the character features, the reference features and image noise to obtain generated content features after each diffusion process; wherein the image noise during the first diffusion process is the image noise obtained from the layout data; the image noise during the Nth diffusion process is the image noise obtained after adding noise to the generated content features after the N-1th diffusion process; N is an integer greater than 1; obtaining target visual data based on the generated content features after the last diffusion process; the target visual data includes the character features of the first object, the reference features of the second object and the semantic features of the layout data.

[0007] An embodiment of the present application provides a visual data generating device, including: a feature extraction module, used to extract features from character data to obtain character features of a first object in the character data; to extract features from layout data to obtain reference features of a second object in the layout data; a diffusion processing module, used to perform multiple diffusion processing based on the character features, the reference features and image noise to obtain generated content features after each diffusion processing; wherein the image noise during the first diffusion processing is the image noise obtained from the layout data; the image noise during the Nth diffusion processing is the image noise obtained after adding noise processing to the generated content features after the N-1th diffusion processing; N is an integer greater than 1; a data generation module, used to obtain target visual data based on the generated content features after the last diffusion processing; the target visual data includes the character features of the first object, the reference features of the second object and the semantic features of the layout data.

[0008] An embodiment of the present application provides an electronic device, which includes: a memory for storing computer-executable instructions or computer programs; and a processor for executing the computer-executable instructions or computer programs stored in the memory to implement the visual data generation method provided in the embodiment of the present application.

[0009] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the visual data generation method provided in the embodiment of the present application when executed by a processor.

[0010] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the visual data generation method provided in the embodiment of the present application is implemented.

[0011] The embodiments of the present application have the following beneficial effects:

[0012] First, feature extraction is performed on the character data to obtain the character features of the first object in the character data, and feature extraction is performed on the layout data to obtain the reference features of the second object in the layout data. Then, a first diffusion process is performed based on the character features, the reference features and the image noise obtained from the layout data to obtain the generated content features after the first diffusion process. The generated content features after the first diffusion process are subjected to noise processing to obtain image noise. The diffusion process is repeated multiple times based on the character features, the reference features and the image noise to obtain the generated content features after each diffusion process. In this way, by adding noise to the generated content features and then performing diffusion processing, the influence of high-frequency noise on the quality of the generated image can be reduced. After obtaining the generated content features after each diffusion process, the target visual data is obtained based on the generated content features after the last diffusion process; the target visual data includes the character features of the first object, the reference features of the second object and the semantic features of the layout data. In this way, by performing multiple diffusion processes and generating the final generated video or generated image, the character consistency in the generated video or generated image can be enhanced and the image generation quality can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is an optional flow chart of the visual data generation method provided in the embodiment of the present application;

[0014] Figure 2 is a schematic diagram of an implementation flow of obtaining the role characteristics of a first object provided in an embodiment of the present application;

[0015] Figure 3 is a schematic diagram of an implementation flow of obtaining a reference feature of a second object provided in an embodiment of the present application;

[0016] Figure 4 It is a schematic diagram of the implementation process of obtaining the generated content features provided in the embodiment of the present application;

[0017] Figure 5 is a schematic diagram of actual application results of the visual data generation method provided in an embodiment of the present application;

[0018] Figure 6 It is a schematic diagram of the principle of the visual data generation method provided in the embodiment of the present application;

[0019] Figure 7 is a structural block diagram of a visual data generating device provided in an embodiment of the present application;

[0020] Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0022] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0023] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0024] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0025] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0026] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0027] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0028] 1) Layout data: refers to the original image or video, which contains the characters to be replaced as well as the characters’ poses, positions, and background information. The layout data provides the semantic framework and spatial layout information of the scene for the image synthesis task. This information is used to guide how to accurately adjust the characters in the character data to the layout data during the image synthesis process to maintain the overall consistency and naturalness of the scene.

[0029] 2) Character data: refers to the specified image or video used to replace the character in the layout data in image synthesis or image editing tasks. Character data usually contains one or more characters with specific identities and characteristics (such as facial features, clothing, hairstyle, etc.). In the synthesis process, the character consistency of the character needs to be maintained so that the replaced character matches the character in the character data in terms of character characteristics, thereby achieving seamless visual integration.

[0030] In order to better understand the visual data generating method provided in the embodiment of the present application, the visual data generating method in the related art is first described below.

[0031] When solving the problem of character consistency in image generation, related technologies often need to comprehensively consider multiple factors to fine-tune the output results. In particular, when the character identity consistency condition needs to be highlighted, the control of image layout and action will be weakened accordingly. Therefore, when the character consistency condition is strong, the accuracy of layout control is usually difficult to guarantee.

[0032] Based on the problems existing in the related art, the embodiment of the present application provides a method for generating visual data. First, feature extraction is performed on the character data to obtain the character features of the first object in the character data, and feature extraction is performed on the layout data to obtain the reference features of the second object in the layout data; then, multiple diffusion processes are performed based on the character features, reference features and image noise to obtain the generated content features after each diffusion process; wherein, the image noise during the first diffusion process is the image noise obtained from the layout data; the image noise during the Nth diffusion process is the image noise obtained after adding noise to the generated content features after the N-1th diffusion process; N is an integer greater than 1; finally, based on the generated content features after the last diffusion process, the target visual data is obtained; the target visual data includes the character features of the first object, the reference features of the second object and the semantic features of the layout data. In this way, by performing multiple diffusion processes and generating the final generated video or generated image, the character consistency in the generated video or generated image can be enhanced and the image generation quality can be improved.

[0033] The visual data generation method provided by the embodiment of the present application is described in detail below in conjunction with the accompanying drawings. The method can be applied to electronic devices. The following is an exemplary description using the electronic device as a server. Figure 1 , Figure 1 is an optional flow chart of the visual data generation method provided in the embodiment of the present application, which will be combined with Figure 1 Steps S101 to S104 are shown for explanation.

[0034] Step S101, extracting features from the character data to obtain character features of a first object in the character data.

[0035] Here, the character data is a specified image or video used to replace an object in the layout data. The character data may include different types and styles. For example, the character data is a character image. The types of the character image may include: photographic images, digital art, painting images, illustrations, pixel art, comics and animation, etc.; the styles of the character image may include: realistic style, cartoon style, abstract style, surreal style, impressionist style and pixel style, etc. The character data is a character video. The types of the character video may include: real-shot video, three-dimensional animation video, two-dimensional animation video, dynamic illustration video, comic or animation video, etc.; the styles of the character video may include: realistic style video, cartoon style video, abstract style video, impressionist style video, etc. The character data usually contains one or more objects, and the first object in the character data is one or more objects in the specified character data. The first object may be a person, an animal, a plant or other object. The character feature is used to characterize the attribute features of the appearance of the first object. For example, the first object is a person, and the character feature is used to characterize the facial features, clothing features, skin color features and other features related to the appearance of the person or related information that can reflect the consistency of the object. Alternatively, the first object is an animal, such as a cat, and the character features are used to characterize the cat's facial features, body features, hair features, and other features related to the cat's appearance.

[0036] Feature extraction of images is a basic task in the field of computer vision, and its purpose is to extract useful key information from images. Methods for feature extraction of character data may include: gray-level co-occurrence matrix (extracting texture features by analyzing the spatial relationship between pixels in an image); edge detection operator (used to detect edges in an image); convolutional neural network (automatically learning the feature representation of an image by training a convolutional neural network); feature fusion technology (combining multiple feature extraction methods to obtain a more comprehensive and accurate image feature representation), etc. Feature extraction of character data can identify and extract key information or attributes that can represent the first object from the character data. Character data can be an image or video pre-entered by humans, or an image or video downloaded from the Internet.

[0037] Step S102: extracting features from the layout data to obtain reference features of the second object in the layout data.

[0038] In some embodiments, the layout data is an original image or an original video, and the layout data includes the object to be replaced and the posture, position, background information or some important props of the object. The layout data usually includes one or more objects. The second object may be a person, an animal, a plant or other object. The reference feature is used to characterize the attribute characteristics of the second object including information such as posture, action, clothing or important props. For example, the second object may be a person, and the reference feature is used to characterize the posture and position characteristics of the person. For example, a person may put his feet together, place his left hand on his knee, hold a mobile phone in his right hand, and lean on a chair. The reference feature can be used to characterize the above posture. Alternatively, the first object may be an animal, such as a cat curled up into a ball with its head resting on its front paws, and the reference feature can be used to characterize the above posture. The method for extracting features from the layout data can refer to step S101, which will not be repeated here.

[0039] Step S103, performing multiple diffusion processes based on the character features, reference features and image noise to obtain generated content features after each diffusion process.

[0040] Here, the image noise during the first diffusion process is the image noise obtained from the layout data; the image noise during the Nth diffusion process is the image noise obtained by adding noise to the generated content features after the N-1th diffusion process; N is an integer greater than 1.

[0041] In some embodiments, before performing the first diffusion process, image noise is first obtained from the layout data, and then diffusion process is performed based on the character features, reference features and image noise to obtain the generated content features after the first diffusion process. Next, before performing the second diffusion process, the generated content features after the first diffusion process are first subjected to noise processing to obtain the corresponding image noise, and then the second diffusion process is performed based on the character features, reference features and image noise to obtain the generated content features after the second diffusion process. Similarly, before the Nth diffusion process, the generated content features after the N-1th diffusion process are first subjected to noise processing to obtain the corresponding image noise, and then the Nth diffusion process is performed based on the character features, reference features and image noise to obtain the generated content features after the Nth diffusion process.

[0042] Step S104, obtaining target visual data based on the generated content features after the last diffusion process.

[0043] Here, the target visual data includes the role features of the first object, the reference features of the second object, and the semantic features of the layout data.

[0044] In some embodiments, when the layout data is an image, a generated image including the character features of the first object, the reference features of the second object, and the semantic features of the layout data can be generated based on the generated content features after the last diffusion process. When the layout data is a video, a generated video including the character features of the first object, the reference features of the second object, and the semantic features of the layout image can be generated based on the generated content features after the last diffusion process. For example, when the layout data is an image, there is a woman with fair skin, glasses, and a blue floral dress in the character data, and the woman is holding a dark cat with dark green eyes. The background in the layout data is in a park, on a park bench, a man in a black top and black shorts, with his feet together, his left hand on his knee, and his right hand holding a black and white spotted dog, the spotted dog leaning on the side of the man's right leg. Based on the generated content features after the last diffusion process, the generated image is: the background is still in the park, on the park bench, a woman with fair skin, glasses, and a blue floral dress, with her feet together, her left hand on her knees, and her right hand holding a dark-skinned cat with dark green pupils, and the cat is leaning on the right leg of the woman. When the layout data is a video, there is a woman with fair skin, glasses, and a blue floral dress in the character data, and the woman is standing on the balcony holding a dark-skinned cat with dark green pupils and a dark body. The background and objects in the layout data remain unchanged, but in the video where the layout data is located, the man is holding a dog with his right hand, and his left hand reaches out to the dog and strokes the dog's head. Then the generated video will be: the background is still in the park, on the park bench, a woman with fair skin, glasses, and a blue floral dress, with her feet together, her left hand on her knees, and her right hand holding a dark-skinned cat with dark green pupils and a dark body, and the cat is leaning on the right leg of the woman, and her left hand reaches out to the cat and strokes the cat's head.

[0045] The visual data generation method provided by the embodiment of the present application firstly extracts the character data to obtain the character features of the first object in the character data, and extracts the layout data to obtain the reference features of the second object in the layout data, then performs the first diffusion processing based on the character features, the reference features and the image noise obtained from the layout data to obtain the generated content features after the first diffusion processing, performs noise processing on the generated content features after the first diffusion processing to obtain the image noise, and continues to repeat the diffusion processing based on the character features, the reference features and the image noise to obtain the generated content features after each diffusion processing, so that by adding noise to the generated content features and then performing the diffusion processing, the influence of high-frequency noise on the generated image quality can be reduced. After obtaining the generated content features after each diffusion processing, the target visual data is obtained based on the generated content features after the last diffusion processing; the target visual data includes the character features of the first object, the reference features of the second object and the semantic features of the layout data, so that by performing multiple diffusion processing and generating the final generated video or generated image, the character consistency in the generated video or generated image can be enhanced and the image generation quality can be improved.

[0046] In some embodiments, see Figure 2 , Figure 2 is a schematic diagram of an implementation flow of obtaining the role characteristics of a first object provided in an embodiment of the present application, Figure 1 The illustrated step S101 can be implemented by following the steps S1011 to S1015, which are described in detail below.

[0047] Step S1011, performing face detection on the character data to obtain a facial image of the first object in the character data.

[0048] In some embodiments, face detection can be performed on the character data through a pre-trained face detection model to obtain a facial image of the first object in the character data. When the character data is an image, face detection can be performed directly on the image; when the character data is a video, video frames in the video can be extracted first, and then face detection can be performed on each video frame. The face detection model may include: Haar cascade classifier (a set of weak classifiers obtained by training a large number of positive and negative samples, which are combined to form a strong classifier that can quickly detect faces); Local Binary Patterns (LBP) (a simple and effective texture feature extraction method that can also be used for face detection); Histogram of Oriented Gradients (HOG) (a method for detecting shape features of objects in an image, commonly used for face detection); Single Shot MultiBox Detector (SSD) (a detection framework based on deep learning that can simultaneously detect multiple objects in an image, including faces), etc. A face detection model can be selected according to actual conditions, and the selected model can be used to perform face detection on the character data, so that the position and bounding box information of the face of the first object in the character data can be obtained, thereby obtaining the facial image of the first object. It should be noted that the character data can contain one object or multiple objects. When there is only one object in the character data, the object is the first object; when there are multiple objects in the character data, the character data can be preprocessed before face detection (for example, other objects other than the first object are cropped or covered) so that only the first object is included in the character data, or, after obtaining multiple facial images corresponding to multiple objects in the character data, the facial image of the first object is selected for subsequent processing.

[0049] As an example, for example, there is only one object in the character data, and Haar cascade classifier is used for face detection. First, the Haar cascade classifier slides a small window over the entire character data, checking whether the features within each window match the facial features. When the window moves to an area containing a face, the Haar cascade classifier recognizes that this area may be part of a face. Then, the window continues to move and expand until it covers the entire face. During the sliding process, the Haar cascade classifier further refines the possible face area until the position and size of the face are finally determined, thereby determining the facial image of the face.

[0050] Step S1012: extract features from the facial image to obtain facial image features of the first object.

[0051] In some embodiments, a deep learning model may be used to extract features from a facial image, and the deep learning model can learn deep features of the first object's face, including texture, shape, and identity information, etc. The deep learning model converts the facial image into a feature vector (i.e., facial image features), which contains key information about the face.

[0052] As an example, through face detection, the facial region of an object in character data can be located. Then, the facial image is input into a pre-trained deep learning model (e.g., a convolutional neural network), and the deep learning model can extract facial image features.

[0053] Step S1013: Perform human body detection on the character data to obtain a body image of the first object in the character data.

[0054] In some embodiments, human body detection can be performed on the character data through a pre-trained human body detection model to obtain a body image of the first object in the character data. The human body detection model may include: You Only Look Once (YOLO) (capable of accurately identifying and locating the human body in the image); Faster Regional Convolutional Neural Network (Faster R-CNN) (a two-stage target detection model), etc. The implementation method of step S1013 can refer to step S1011 and will not be repeated here.

[0055] Step S1014: extract features from the body image to obtain clothing features of the first object.

[0056] In some embodiments, the implementation of step S1014 may refer to step S1012 and will not be repeated here.

[0057] Step S1015: determining the facial image features and clothing features as the character features of the first object.

[0058] In some embodiments, the obtained facial image features and clothing features are used together as the character features of the first object.

[0059] Through step S1011 to step S1015, face detection and body detection are performed on the character data to extract facial image features and clothing features respectively, thereby determining the character features of the first object, providing rich basic data for subsequent processing.

[0060] In some embodiments, see Figure 3 , Figure 3 is a schematic diagram of an implementation flow of obtaining a reference feature of a second object provided in an embodiment of the present application, Figure 1 The illustrated step S102 can be implemented by following the steps S1021 to S1022, which are described in detail below.

[0061] Step S1021: perform key point detection on the layout data to obtain a plurality of key points of the second object in the layout data.

[0062] In some embodiments, the layout data is subjected to key point detection using a detection model, thereby obtaining the positions of multiple key points of the second object in the layout data. When the layout data is an image, key point detection can be performed directly on the image. When the layout data is a video, the video frames in the video can be extracted first, and then key point detection can be performed on each video frame. The purpose of key point detection is to identify specific parts of the second object's body, such as the head, shoulders, elbows, wrists, hips, knees, and ankles. The key points may include: the center point of the head, the center point of the neck, the right shoulder point, the right elbow point, the right wrist point, the left shoulder point, the left elbow point, the left wrist point, the center point of the pelvis, the right hip joint point, the right knee point, the right ankle point, the left hip joint point, the left knee point, and the left ankle point. Detection models may include: scale-invariant feature transformation (extracting key points in the image and generating feature descriptors for each key point with scale invariance); graph convolutional networks (GCNs) (treating key points in the image as nodes in the graph, using graph convolutional networks to process the relationship between nodes for key point detection and pose estimation); Faster R-CNN; SSD; YOLO, etc.

[0063] As an example, for example, the second object in the layout data is in a standing posture. The detection model is used to detect key points of the layout data. After the detection model processes the image, it extracts features related to the person and searches for key points at different scales. First, the detection model detects the position of the person's head and marks the center point of the head. The detection model then identifies the key points of the person's neck, shoulders, chest, waist and other body parts. The detection model then further identifies the key points of the person's limbs, such as elbows, wrists, knees and ankles. Finally, the detection model assigns a precise position coordinate to each detected key point, and the coordinate represents the specific position of the key point in the image (for example, the coordinates of the center point of the head are (x1, y1); the coordinates of the center point of the neck are (x2, y2), etc.).

[0064] Step S1022: determining reference features of the second object according to the spatial position relationship between the multiple key points.

[0065] In some embodiments, the detection model can determine the spatial position relationship between multiple key points based on the position coordinates of the key points, and then determine the posture of the second object based on the spatial position relationship, and extract the reference features of the second object. Standing posture: If the key point detection result shows that the x-coordinates of the center point of the head, the center point of the neck, the center point of the shoulder, the center point of the hip and the center point of the hip are the same, it can be determined that the center point of the head is directly above the center point of the neck, and the center point of the shoulder is directly below the center point of the neck, and the center point of the hip is directly below the center point of the shoulder. Thus, it can be determined that the second object is in a standing posture. Bending posture: If the key point detection result shows that the y-coordinate of the center point of the head is less than the y-coordinate of the center point of the neck, and the y-coordinate of the center point of the neck is less than the y-coordinate of the center point of the shoulder, and the difference between the y-coordinate of the center point of the shoulder and the y-coordinate of the center point of the hip is within the set threshold range, it can be determined that the second object is in a bending posture. Hand-raising posture: If the key point detection result shows that the y-coordinate of the center point of a wrist is greater than the y-coordinate of the center point of the shoulder on the same side, and the difference between the x-coordinate of the center point of the wrist and the x-coordinate of the center point of the shoulder is within the set threshold range, then it can be determined that the second object is in a hand-raising posture. Running posture: If the key point detection result shows that the y coordinate of one ankle center point is lower than the y coordinate of the other ankle center point, and the difference between the x coordinates of the two ankle center points is within the set threshold range, this indicates that one foot is on the ground and the other foot is in the air, and it can be determined that the second object is in a running posture.

[0066] Through step S1021 to step S1022, key point detection is performed on the layout data to obtain the key point positions of specific parts of the second object's body, thereby determining the reference features of the second object, providing accurate data support for subsequent processing.

[0067] In some embodiments, see Figure 4 , Figure 4 is a schematic diagram of an implementation flow of obtaining generated content features provided in an embodiment of the present application, Figure 1 The illustrated step S103 can be implemented by following the steps S1031 to S1032, which are described in detail below.

[0068] Step S1031, obtaining a preset number of diffusion processing times.

[0069] In some embodiments, the preset number of diffusion processing times may be pre-set and may be set based on actual conditions or empirical values, and this embodiment of the present application does not limit this.

[0070] Step S1032 , performing multiple diffusion processes based on the character features, reference features and image noise according to the number of diffusion processes, and obtaining generated content features after each diffusion process.

[0071] In some embodiments, the above step S1032 of "performing multiple diffusion processes based on character features, reference features and image noise to obtain generated content features after each diffusion process" can be implemented in the following way: first, during the first diffusion process, the layout data is subjected to reverse denoising diffusion processing to obtain the first reference noise during the first diffusion process; then, the first diffusion process is performed based on the character features, the reference features and the first reference noise to obtain the generated content features after the first diffusion process; thereafter, during the Nth diffusion process, Gaussian noise is added to the generated content features after the N-1th diffusion process to obtain the noisy content features; the noisy content features are continued to be subjected to reverse denoising diffusion processing to obtain the second reference noise during the Nth diffusion process; finally, the Nth diffusion process is performed based on the character features, the reference features and the second reference noise to obtain the generated content features after the Nth diffusion process.

[0072] As an example, multiple diffusion processes can be performed according to the preset number of diffusion processes obtained. For example, the number of diffusion processes is three times. First, during the first diffusion process, the layout data is subjected to inverse denoising diffusion processing, and the inverse denoising diffusion processing can be achieved by inversion of the denoising diffusion implicit model (Denoising Diffusion Implicit Models, DDIM), that is, the layout data is noise-encoded by the DDIM inversion model, so as to obtain the first reference noise during the first diffusion process. The first reference noise can reflect the macro information of the layout data, for example, the layout of the image in the layout data, the action of the character, the image style and the video camera mode, etc.; the first reference noise can be used to characterize the semantic features of the layout data. DDIM is a generative model based on a diffusion process, which can not only be used to generate new images, but also to restore the original image from the noisy image. DDIM Inversion gradually adds noise to the original image (i.e., layout data) through a forward diffusion process until the original image becomes completely random noise (i.e., the first reference noise).

[0073] Then, a first diffusion process is performed based on the character feature, the reference feature and the first reference noise to obtain the generated content feature after the first diffusion process. The diffusion process can be implemented by gradually removing noise through the reverse diffusion process of DDIM, or can be implemented through other diffusion models (such as a generative diffusion model, a conditional diffusion model, etc.), which is not limited in the embodiments of the present application.

[0074] Afterwards, during the second diffusion process, Gaussian noise is first added to the generated content features after the first diffusion process to obtain the noisy content features, and the noisy content features are further subjected to inverse denoising diffusion processing to obtain the second reference noise during the second diffusion process. Finally, the second diffusion process is performed based on the character features, the reference features, and the second reference noise to obtain the generated content features after the second diffusion process.

[0075] Repeat the above process. In the third diffusion process, first add Gaussian noise to the generated content features after the second diffusion process to obtain the noisy content features, and then continue to perform reverse denoising diffusion process on the noisy content features to obtain the second reference noise for the third diffusion process. Finally, perform the third diffusion process based on the character features, reference features and the second reference noise to obtain the generated content features after the third diffusion process.

[0076] Through the above method, the character features and reference features are simultaneously integrated in the process of successive denoising of the image noise obtained by the inverse denoising diffusion processing, which can effectively integrate the semantic information of appearance information, posture information and layout data, improve the quality of generated content features, and thus improve the character consistency and the accuracy of image semantics.

[0077] Further, in some embodiments, the above-mentioned "adding Gaussian noise to the generated content features after the N-1th diffusion processing to obtain the noisy content features" can be achieved in the following way: adding a preset multiple of random Gaussian noise to the generated content features after the N-1th diffusion processing to obtain the noisy content features; the preset multiple is greater than 0 and less than a preset multiple threshold.

[0078] As an example, the preset multiple threshold value may be pre-set and may be set according to actual conditions or empirical values, and the embodiments of the present application do not limit this. For example, the preset multiple threshold value is set to 0.05, the value range of the random Gaussian noise is 0-1, and the value range of the corresponding preset multiple random Gaussian noise is 0-0.05. 0.05 times the random Gaussian noise may be added to the generated content features after the N-1th diffusion process to obtain the noisy content features. For example, suppose there is a content feature vector, which is a one-dimensional array, representing the content features after the N-1th diffusion process. First, a one-dimensional random Gaussian noise vector of the same size as the content feature vector needs to be generated. The generated noise vector is then multiplied by 0.05 times, and then added to the content feature vector to obtain the noisy content features.

[0079] By adding noise to the generated content features after each diffusion process in the above manner, the influence of noise points generated in the inverse denoising diffusion process can be reduced, thereby improving the quality of the generated image.

[0080] In some embodiments, the above step S1032 of "performing multiple diffusion processes based on character features, reference features and image noise to obtain generated content features after each diffusion process" can also be achieved in the following way: during each diffusion process, the facial image features, clothing features, reference features and image noise are input into the diffusion model; through the diffusion model, the facial image features, clothing features, reference features and image noise are diffused to obtain the generated content features after the diffusion process.

[0081] As an example, in each diffusion process, the facial image features, clothing features, reference features and image noise are first input into the diffusion model together, and then the diffusion model is used to perform diffusion processing on the facial image features, clothing features, reference features and image noise to obtain the generated content features after diffusion processing. For example, the first object in the character data is an anime girl wearing a striped shirt and black pants, and the second object in the layout data has her arms crossed in front of her chest, with the background in front of the school gate. The extracted facial image features, clothing features and reference features are input into the diffusion model, and the diffusion model begins to process the input data. The facial image features can be diffused first to retain the key features of the face while gradually removing noise. Then the diffusion model processes the clothing features to ensure that the color and texture of the clothing are retained during the denoising process while removing noise. The diffusion model processes the reference features to ensure that the relative positions and angles of the various parts of the body remain consistent so that the generated image can show the correct body posture. The diffusion model gradually removes noise throughout the diffusion process while optimizing details such as the face, clothing and posture. After a series of diffusion processing steps, the diffusion model outputs the generated content features after the diffusion processing, so that the generated content features retain the facial image features and clothing features of the first object, the reference features of the second object and the semantic information of the layout data.

[0082] In the above manner, the diffusion model is used to eliminate noise, while the facial image features and clothing features of the first object, the reference features of the second object and the semantic information of the layout data are retained, thereby improving the accuracy of the character details in the generated image and the integrity of the scene semantics.

[0083] By performing multiple diffusion processes from step S1031 to step S1032, the generated content features are gradually optimized to improve the consistency of roles and the accuracy of image semantics in the generated content features.

[0084] In some embodiments, when the obtained target visual data is an image, the above step S104 can also be implemented in the following manner: image feature decoding is performed on the generated content features after the last diffusion process to obtain a generated image, and the generated image is determined as the target visual data.

[0085] As an example, image feature decoding is the process of converting an abstract representation of an image back into a visually perceptible image format. For example, a related model is used to extract the facial image features and clothing features of an anime girl wearing a striped shirt and black pants in the character data, as well as the reference features of arms crossed on the chest in the layout data. The extracted facial image features, clothing features, and reference features can be encoded by the model into a format, such as a set of numbers, to represent the visual content of the image. After a preset number of diffusion processes, the generated content features after the last diffusion process are obtained. The model receives the generated content features after the last diffusion process and decodes them using a decoder component (which can be one or more layers of a neural network specifically designed to convert features back to image pixels). The decoder generates corresponding pixel values ​​based on the features and reconstructs various parts of the image, such as facial details of the character, texture of the clothing, and elements of the background. For the striped shirt worn by the anime girl, the decoder generates the stripe pattern and color of the shirt based on the features. For the girl's black pants, the decoder generates corresponding black pixels and ensures that the shape and position of the pants match the features. For the posture of crossed arms, the decoder reconstructs the exact position and angle of the arms. Finally, the decoder outputs a complete generated image. Finally, the generated image is determined as the target visual data.

[0086] In some embodiments, when the target visual data obtained is a video, the above step S104 can be implemented in the following manner: performing image feature decoding on the generated content features after the last diffusion process to obtain a generated image in a video frame sequence; and generating a video based on the video frame sequence.

[0087] As an example, decoding the image features of the generated content features after the last diffusion process to obtain a generated image in the video frame sequence is the same as the implementation method of "decoding the image features of the generated content features after the last diffusion process to obtain the generated image", which will not be repeated here. After obtaining each generated image in the video frame sequence, the video frame sequence can be synthesized into a video using specific video frame sequence to video software or tools. For example, the Fast Forward Moving Picture Expert Group (FFMPEG) can be used to quickly synthesize the video frame sequence into a video. Finally, the generated video is determined as the target visual data.

[0088] Through the above method, an image can be processed, or a frame of an image in a video can be processed to obtain a corresponding generated image or generated video, thereby expanding the scope of application of the visual data generation method.

[0089] Through the embodiment of the present application, feature extraction is performed on the character data to obtain the character features of the first object in the character data, and feature extraction is performed on the layout data to obtain the reference features of the second object in the layout data. Then, a first diffusion process is performed based on the character features, the reference features and the image noise obtained from the layout data to obtain the generated content features after the first diffusion process. The generated content features after the first diffusion process are subjected to noise processing to obtain image noise. The diffusion process is repeated multiple times based on the character features, the reference features and the image noise to obtain the generated content features after each diffusion process. In this way, by adding noise to the generated content features and then performing diffusion processing, the influence of high-frequency noise on the quality of the generated image can be reduced. After obtaining the generated content features after each diffusion process, the target visual data is obtained based on the generated content features after the last diffusion process; the target visual data includes the character features of the first object, the reference features of the second object and the semantic features of the layout data. In this way, by performing multiple diffusion processes and generating the final generated video or generated image, the character consistency in the generated video or generated image can be enhanced and the image generation quality can be improved.

[0090] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0091] Exemplarily, the embodiments of the present application can be applied in the following fields: Film and TV series production: In film production, the method provided by the embodiments of the present application can be used to replace the actors in the scene so as to reuse the same performance in different backgrounds or scenes. Virtual reality and augmented reality: In virtual reality, the user's virtual image can be replaced in real time in the virtual environment to provide a more immersive experience. In augmented reality, the user can superimpose the image of his own photo or video on the scene of the real world in real time. Game development: The game can allow players to embed their own image or favorite character into the game world. Personalized marketing: In advertising and marketing, the image of the target audience can be replaced in the advertising image according to their preferences to increase the attractiveness and participation of the advertisement. Education and training: In educational software, the image of students can be replaced in historical scenes or scientific experiments to increase the fun and interactivity of learning. Remote work and meetings: In remote work or online meetings, the image of an individual can be replaced in a professional background to enhance the professionalism and formality of the meeting. Security monitoring: In the field of security monitoring, the characters in the surveillance video can be replaced in a simulated scene to protect personal privacy.

[0092] The embodiment of the present application can solve the problem of low semantic accuracy of the generated image on the basis of achieving role replacement when generating the final generated image through the layout image (i.e., layout data). For example, given a layout image containing a person (i.e., the second object), the sampling initial noise (i.e., the first reference noise) corresponding to the layout image is obtained through DDIM Inversion, and then sampling is started from the sampling initial noise, and at the same time, the personalized text-to-image generation technology (Pulid) and the adapter (Image Prompt Adapter, IPAdapter) designed by the text-to-image diffusion model are used to pass the face information (equivalent to facial image features) and clothing information (equivalent to clothing features) of the character to be replaced (i.e., the first object) in the character image (i.e., character data). The generated image generated in this way can simultaneously have both the semantic information of the layout image and the role information of the character image, thereby achieving role consistency and accurate layout control.

[0093] When generating an image, by using the result of DDIM Inversion of the layout image as the initial noise for sampling, the macro information of the layout image can be referenced in the subsequent image generation process, such as the layout information and the actions of the characters in the layout image, thereby improving the semantic accuracy in the subsequent replacement process. Considering that the appearance of the characters in the layout image may be quite different from that of the character image, the identity information of the character image can be continuously enhanced while ensuring that the overall semantic information of the image is preserved through the strategy of gradual and multiple DDIM Inversion and then diffusion. However, it has been found through testing that directly performing multiple DDIM Inversions on the output image will result in over-sharpening, see Figure 5 , Figure 5 : is a schematic diagram of the actual application results of the visual data generation method provided in the embodiment of the present application. Figure 5 (a) is the character image; Figure 5 (b) is the layout image; Figure 5 (c) is the key point corresponding to the layout image; Figure 5 (d) is the generated image obtained by performing a diffusion process; Figure 5 (e) is the generated image obtained by multiple diffusion processes. It can be seen that Figure 5 The noise in the image will become more obvious with the number of DDIM Inversion rounds, which will eventually affect the quality of the image. Considering that noise is a high-frequency signal, it can be alleviated by adding 0.05 Gaussian noise to the inverted noise (directly applying Gaussian blur to the image is ineffective), thereby achieving multiple rounds of DDIM Inversion and enhancing the consistency of the characters.

[0094] See also Figure 6 , Figure 6 is a schematic diagram of the principle of the visual data generation method provided in the embodiment of the present application, Figure 6 The example shows the case where the generated visual data is an image. First, the human posture is extracted from the layout image 601 (i.e., layout data) to obtain posture features (i.e., reference features), and the clothing information and facial information of the character image 602 (i.e., character data) are extracted to obtain character feature information 603 (i.e., character features) such as clothing features and facial features. Then, the layout image 601 is input into the denoising diffusion implicit model inversion module 604, and the layout image 601 is subjected to DDIM Inversion (i.e., inverse denoising diffusion processing) to obtain the image noise corresponding to the layout image (i.e., the first reference noise). Then, the extracted posture features, clothing features, facial features, and image noise corresponding to the layout image are used together as inputs of the diffusion model 605 for diffusion generation (i.e., diffusion processing), and an intermediate result 606 (i.e., generated content features) is obtained. After that, the intermediate result 606 is added with 0.05 Gaussian noise 607 to obtain the noise image feature of the generated content feature (i.e., the content feature after adding noise), and then the noise image feature is input into the denoising diffusion implicit model inversion module 604, and the DDIM Inversion is performed to obtain the image noise (i.e., the second reference noise) for the next round of diffusion generation, and then the posture features, clothing features, facial features, and image noise are repeated as the input of the diffusion model 605 for the next round of diffusion generation to obtain the intermediate result 606 of the next round, and the above process is repeated N times. When the last processing is performed, the generated effect image 608 (i.e., the target visual data) is obtained according to the intermediate result 606. The final generated effect image 608 is output (the output can also be a video), that is, the output content is visual data: including static images and dynamic videos.

[0095] In the embodiment of the present application, through multiple rounds of DDIM inversion, the semantic accuracy of the generated image is improved under the premise of controllable generation of character consistency.

[0096] Based on the visual data generation method described in the above embodiment, Figure 7 A structural block diagram of a visual data generating device 100 provided in an embodiment of the present application is shown. The visual data generating device may be a device in an electronic device (for example, a server). The visual data generating device may be implemented in software, which may be software in the form of programs and plug-ins, etc., including the following software modules: a feature extraction module 101, a diffusion processing module 102 and a data generation module 103. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented.

[0097] Among them, the feature extraction module 101 is used to extract features from the character data to obtain the character features of the first object in the character data; to extract features from the layout data to obtain the reference features of the second object in the layout data; the diffusion processing module 102 is used to perform multiple diffusion processing based on the character features, reference features and image noise to obtain the generated content features after each diffusion processing; wherein the image noise during the first diffusion processing is the image noise obtained from the layout data; the image noise during the Nth diffusion processing is the image noise obtained after adding noise processing to the generated content features after the N-1th diffusion processing; N is an integer greater than 1; the data generation module 103 is used to obtain the target visual data based on the generated content features after the last diffusion processing; the target visual data includes the character features of the first object, the reference features of the second object and the semantic features of the layout data.

[0098] In some embodiments, the diffusion processing module 102 is further used to obtain a preset number of diffusion processing times; according to the number of diffusion processing times, multiple diffusion processings are performed based on the character features, reference features and image noise to obtain the generated content features after each diffusion processing.

[0099] In some embodiments, the diffusion processing module 102 is also used to perform reverse denoising diffusion processing on the layout data during the first diffusion processing to obtain a first reference noise during the first diffusion processing; perform the first diffusion processing based on the character features, the reference features and the first reference noise to obtain the generated content features after the first diffusion processing; during the Nth diffusion processing, add Gaussian noise to the generated content features after the N-1th diffusion processing to obtain the noisy content features; perform reverse denoising diffusion processing on the noisy content features to obtain the second reference noise during the Nth diffusion processing; perform the Nth diffusion processing based on the character features, the reference features and the second reference noise to obtain the generated content features after the Nth diffusion processing.

[0100] In some embodiments, the diffusion processing module 102 is further used to add a preset multiple of random Gaussian noise to the generated content features after the N-1th diffusion processing to obtain the noisy content features; the preset multiple is greater than 0 and less than a preset multiple threshold.

[0101] In some embodiments, the data generation module 103 is further used to perform image feature decoding on the generated content features after the last diffusion processing to obtain a generated image, and determine the generated image as the target visual data; or, perform image feature decoding on the generated content features after the last diffusion processing to obtain a generated image in a video frame sequence; and generate a video based on the video frame sequence.

[0102] In some embodiments, the feature extraction module 101 is also used to perform face detection on the character data to obtain a facial image of a first object in the character data; perform feature extraction on the facial image to obtain facial image features of the first object; perform body detection on the character data to obtain a body image of the first object in the character data; perform feature extraction on the body image to obtain clothing features of the first object; and determine the facial image features and clothing features as character features of the first object.

[0103] In some embodiments, the diffusion processing module 102 is also used to input facial image features, clothing features, reference features and image noise into the diffusion model during each diffusion processing; the facial image features, clothing features, reference features and image noise are diffused through the diffusion model to obtain the generated content features after the diffusion processing.

[0104] In some embodiments, the feature extraction module 101 is further used to perform key point detection on the layout data to obtain multiple key points of the second object in the layout data; and determine the reference features of the second object according to the spatial position relationship between the multiple key points.

[0105] It should be noted that the description of the device of the embodiment of the present application is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it is not repeated. For technical details not disclosed in the embodiment of the device, please refer to the description of the method embodiment of the present application for understanding.

[0106] The present application also provides an electronic device, see Figure 8 , Figure 8 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 8 As shown, the electronic device 130 includes: at least one processor 131 ( Figure 8 Only one is shown in the figure), a memory 132, and computer executable instructions 133 stored in the memory 132 and executable on at least one processor 131, when the processor 131 executes the executable instructions 133, the steps in any of the above-mentioned visual data generation method embodiments are implemented.

[0107] The electronic device may include but is not limited to a processor 131 and a memory 132. Those skilled in the art may understand that: Figure 8 This is merely an example of the electronic device 130 and does not constitute a limitation on the electronic device 130 . The electronic device 130 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0108] The processor 131 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPG), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0109] In some embodiments, the memory 132 may be an internal storage unit of the electronic device 130, such as a hard disk or memory of the electronic device 130. In other embodiments, the memory 132 may also be an external storage device of the electronic device 130, such as a plug-in hard disk, a smart memory card (SMC, Smart Media Card), a secure digital (SD, Secure Digital) card, a flash card, etc. equipped on the electronic device 130. Further, the memory 132 may also include both an internal storage unit of the electronic device 130 and an external storage device. The memory 132 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as program codes of a computer program. The memory 132 may also be used to temporarily store data that has been output or is to be output.

[0110] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the visual data generation method described in the embodiment of the present application.

[0111] The present application embodiment provides a computer-readable storage medium in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will execute the visual data generation method provided by the present application embodiment, for example, Figure 1 The visual data generation method is shown.

[0112] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0113] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0114] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0115] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0116] In summary, through the embodiment of the present application, feature extraction is performed on the character data to obtain the character features of the first object in the character data, and feature extraction is performed on the layout data to obtain the reference features of the second object in the layout data. Then, a first diffusion processing is performed based on the character features, the reference features and the image noise obtained from the layout data to obtain the generated content features after the first diffusion processing. The generated content features after the first diffusion processing are subjected to noise processing to obtain image noise. The diffusion processing is repeated multiple times based on the character features, the reference features and the image noise to obtain the generated content features after each diffusion processing. In this way, by adding noise to the generated content features and then performing diffusion processing, the influence of high-frequency noise on the quality of the generated image can be reduced. After obtaining the generated content features after each diffusion process, the target visual data is obtained based on the generated content features after the last diffusion process; the target visual data includes the role features of the first object, the reference features of the second object and the semantic features of the layout data, so that by performing multiple diffusion processes and generating the final generated video or generated image, the role consistency in the generated video or generated image can be enhanced, and the image generation quality can be improved; face detection and body detection are performed on the role data to extract facial image features and clothing features respectively, so as to determine the role features of the first object, providing rich basic data for subsequent processing; key point detection is performed on the layout data to obtain the key point positions of specific parts of the body of the second object, so as to determine the reference features of the second object, providing accurate data support for subsequent processing; the role is simultaneously integrated in the process of successively denoising the image noise obtained by the inverse denoising diffusion process Features and reference features can effectively integrate appearance information, posture information and semantic information of layout data, improve the quality of generated content features, and thus enhance the character consistency and accuracy of image semantics; adding noise to the generated content features after each diffusion process can reduce the impact of noise generated in the inverse denoising diffusion process and improve the quality of the generated image; using the diffusion model to eliminate noise, while also retaining the facial image features and clothing features of the first object, the reference features of the second object and the semantic information of the layout data, improves the accuracy of the character details in the generated image and the integrity of the scene semantics; performing multiple diffusion processes, gradually optimizing the generated content features, and improving the character consistency in the generated content features and the accuracy of the image semantics; the image can be processed, or a frame of the video can be processed to obtain the corresponding generated image or generated video, thereby expanding the scope of application of the visual data generation method.

[0117] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for generating visual data, characterized in that: The method comprises: Extracting features from the character data to obtain character features of a first object in the character data; Extracting features from the layout data to obtain reference features of the second object in the layout data; Based on the character feature, the reference feature and the image noise, multiple diffusion processes are performed to obtain generated content features after each diffusion process; wherein the image noise during the first diffusion process is the image noise obtained from the layout data; the image noise during the Nth diffusion process is the image noise obtained by adding noise to the generated content features after the N-1th diffusion process; N is an integer greater than 1; Based on the generated content features after the last diffusion process, target visual data is obtained; wherein the target visual data includes the role features of the first object, the reference features of the second object, and the semantic features of the layout data.

2. The method according to claim 1, characterized in that The performing multiple diffusion processes based on the character features, the reference features and the image noise to obtain the generated content features after each diffusion process includes: During the first diffusion processing, the layout data is subjected to inverse denoising diffusion processing to obtain a first reference noise during the first diffusion processing; the first diffusion processing is performed based on the character feature, the reference feature and the first reference noise to obtain a generated content feature after the first diffusion processing; During the Nth diffusion processing, Gaussian noise is added to the generated content features after the N-1th diffusion processing to obtain the noisy content features; the inverse denoising diffusion processing is performed on the noisy content features to obtain the second reference noise during the Nth diffusion processing; the Nth diffusion processing is performed based on the character features, the reference features and the second reference noise to obtain the generated content features after the Nth diffusion processing.

3. The method according to claim 2, characterized in that The step of adding Gaussian noise to the generated content features after the N-1th diffusion process to obtain the noisy content features includes: A preset multiple of random Gaussian noise is added to the generated content feature after the N-1th diffusion process to obtain a noisy content feature; the preset multiple is greater than 0 and less than a preset multiple threshold.

4. The method according to claim 1, characterized in that: The performing multiple diffusion processes based on the character features, the reference features and the image noise to obtain the generated content features after each diffusion process includes: Get the preset number of diffusion processing times; According to the number of diffusion processes, multiple diffusion processes are performed based on the character features, the reference features and the image noise to obtain the generated content features after each diffusion process.

5. The method according to claim 1, characterized in that The target visual data is obtained based on the generated content features after the last diffusion process, including: Performing image feature decoding on the generated content features after the last diffusion process to obtain a generated image, and determining the generated image as the target visual data; or, Perform image feature decoding on the generated content features after the last diffusion process to obtain a generated image frame in a video frame sequence; and generate a video based on the video frame sequence.

6. The method according to any one of claims 1 to 5, characterized in that: The step of extracting features from the role data to obtain role features of the first object in the role data includes: Performing face detection on the character data to obtain a facial image of a first object in the character data; Extracting features from the facial image to obtain facial image features of the first object; Performing human body detection on the character data to obtain a body image of a first object in the character data; Extracting features from the body image to obtain clothing features of the first object; The facial image features and the clothing features are determined as character features of the first object.

7. The method according to claim 6, characterized in that The performing multiple diffusion processes based on the character features, the reference features and the image noise to obtain the generated content features after each diffusion process includes: In each diffusion process, the facial image features, the clothing features, the reference features and the image noise are input into a diffusion model; The facial image features, the clothing features, the reference features and the image noise are diffused by the diffusion model to obtain generated content features after the diffusion processing.

8. The method according to any one of claims 1 to 5, characterized in that: The extracting features from the layout data to obtain reference features of the second object in the layout data includes: Performing key point detection on the layout data to obtain a plurality of key points of the second object in the layout data; Determine reference features of the second object according to the spatial position relationship between the multiple key points.

9. A visual data generating device, characterized in that: The device comprises: A feature extraction module is used to extract features from the character data to obtain the character features of the first object in the character data; and to extract features from the layout data to obtain reference features of the second object in the layout data; A diffusion processing module, configured to perform multiple diffusion processing based on the character feature, the reference feature and the image noise, and obtain the generated content feature after each diffusion processing; wherein the image noise during the first diffusion processing is the image noise obtained from the layout data; the image noise during the Nth diffusion processing is the image noise obtained by adding noise to the generated content feature after the N-1th diffusion processing; N is an integer greater than 1; The data generation module is used to obtain target visual data based on the generated content features after the last diffusion process; the target visual data includes the role features of the first object, the reference features of the second object and the semantic features of the layout data.

10. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, used to implement the visual data generation method described in any one of claims 1 to 8 when executing the computer executable instructions or computer programs stored in the memory.

11. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the visual data generating method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the visual data generating method according to any one of claims 1 to 8 is implemented.

Images