Image generation method and device, storage medium and program product

Abstract

The embodiment of the invention provides an image generation method and device, a storage medium and a program product. In the embodiment of the invention, the visual node editor is utilized to generate the image generation script of the image generation process comprising the plurality of nodes which are sequentially connected in series based on each node in the tool area and the function code thereof, and the image generation operation is responded, so that each node in the image generation process can be executed based on the initial text description information to obtain the target image; and a fixed image generation process is generated based on the visual node editor, so that the stability of image generation can be improved. Furthermore, when each node in the image generation process is executed based on the initial text description information, different AI models are used for image generation and restoration, and the quality of the generated image can be improved. Besides, the target image contains the advertisement word image corresponding to each advertisement word, so that the user can quickly know the key information based on the target image, and the conversion rate of the landing page corresponding to the target image is improved.

Description

Technical Field

[0001] The present application relates to the field of computer processing technology, and in particular to an image generation method, device, storage medium and program product. Background Art

[0002] With the development of Internet technology, the use of image generation models to generate service personnel portraits has been widely used. However, the current process of using image generation models is not fixed, resulting in various problems such as unstable style (e.g., the style of the generated image may deviate from the style required by the user) and limb abnormalities (e.g., fingers have extra or fewer fingers). In business scenarios where large-scale AI portrait images are generated, they must be manually screened before they can be put into use, which is labor-intensive, labor-intensive, and inefficient. Summary of the invention

[0003] Multiple aspects of the present application provide an image generation method, device, storage medium and program product to improve the stability and accuracy of generated images and save manual screening time.

[0004] The embodiment of the present application provides an image generation method, which is applied to a visual node editor on an electronic device. The visual node editor displays a graphical user interface, which includes a tool area and an editing area. The tool area includes working nodes based on AI to generate images, and each working node corresponds to its own function code. The method includes: pre-generating a raw image script corresponding to an image generation process based on the working nodes in the tool area and their function codes; the image generation process includes: a label screening node, an image matching node, an AI face replacement node, an image optimization node, an advertising word image generation node, and an image generation node connected in series in sequence; in response to an image generation operation, obtaining initial text description information of a target user, using the initial text description information as input information of the image generation process, and running the raw image script to generate a target image; wherein the raw image script The operation process includes: running a label screening node to determine multiple labels adapted to the target user, and selecting target text description information adapted to the multiple labels from the initial text description information; running an image matching node to match the real-person image of the target user from the real-person image library according to the target text description information; running an AI face replacement node to perform a face-changing operation on the real-person image using an AI face replacement model to obtain a first intermediate image; running an image optimization node to perform style optimization processing on the first intermediate image using a style generation model to obtain a second intermediate image; running an advertising word image generation node to determine at least one advertising word based on the target text description information and generate advertising word images corresponding to each advertising word; running an image generation node to synthesize the second intermediate image and the advertising word image using an image generation model to obtain a target image.

[0005] An embodiment of the present application also provides an electronic device, including: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps in the above method.

[0006] An embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor implements the steps in the above method.

[0007] An embodiment of the present application also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the processor is enabled to implement the steps in the above method.

[0008] In an embodiment of the present application, a visual node editor is used to generate a drawing script containing an image generation process of multiple nodes connected in series based on each node in the tool area and its function code. In response to the image generation operation, each node in the image generation process can be executed based on the initial text description information to obtain the target image. A fixed image generation process is generated based on the visual node editor, which can improve the stability of image generation. Furthermore, when executing each node in the image generation process based on the initial text description information, different AI models are used for image generation and repair to improve the quality of the generated image. In addition, the target image contains an advertising word image corresponding to the advertising word, which allows users to quickly understand key information based on the target image and improve the conversion rate of the landing page corresponding to the target image. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0010] Figure 1 A flowchart of an image generation method provided by an exemplary embodiment of the present application;

[0011] Figure 2 A schematic diagram of a target image provided for an exemplary embodiment of the present application;

[0012] Figure 3 A schematic structural diagram of an electronic device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0014] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0015] Aiming at the problems that AI portrait pictures generated by using image generation models may have unstable style, abnormal limbs, and the technical problems that manual screening requires a large workload, consumes a lot of manpower, and has low efficiency, in the embodiment of the present application, a visual node editor is used to generate a raw image script containing multiple nodes connected in series based on each node and its function code in the tool area. In response to the image generation operation, each node in the image raw image process can be executed based on the initial text description information to obtain the target image. A fixed image generation process is generated based on the visual node editor, which can improve the stability of image generation. Furthermore, when executing each node in the image raw image process based on the initial text description information, different AI models are used for image generation and repair, which can improve the quality of the generated image. In addition, the target image contains an advertising word image corresponding to the advertising word, which allows users to quickly understand key information based on the target image and improve the conversion rate of the landing page corresponding to the target image.

[0016] A solution provided by an embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0017] Figure 1 The following is a flow chart of an image generation method provided by an exemplary embodiment of the present application. Figure 1 As shown, the method is applied to a visual node editor on an electronic device. The visual node editor displays a graphical user interface. The graphical user interface includes a tool area and an editing area. The tool area includes a working node based on AI-generated images. Each working node corresponds to its own function code. The method includes:

[0018] 101. Generate a raw image script corresponding to the image generation process based on the work nodes and their function codes in the tool area in advance; the image generation process includes: a label screening node, an image matching node, an AI face replacement node, an image optimization node, an advertising word image generation node and an image generation node connected in series in sequence;

[0019] 102. In response to the image generation operation, initial text description information of the target user is obtained, the initial text description information is used as input information of the image generation process, and a generating script is run to generate a target image; wherein the running process of the generating script includes:

[0020] S1. Run the tag screening node to determine multiple tags that are suitable for the target user, and select target text description information that is suitable for the multiple tags from the initial text description information;

[0021] S2, running the image matching node to match the real person image of the target user from the real person image library according to the target text description information;

[0022] S3, running the AI ​​face replacement node, and using the AI ​​face replacement model to perform a face replacement operation on the real person image to obtain a first intermediate image;

[0023] S4, running the image optimization node, and performing style optimization processing on the first intermediate image using the style generation model to obtain a second intermediate image;

[0024] S5, running the advertising word image generation node, determining at least one advertising word based on the target text description information, and generating an advertising word image corresponding to each advertising word;

[0025] S6. Run the image generation node and use the image generation model to synthesize the second intermediate image and the advertising word image to obtain a target image.

[0026] In the embodiment of the present application, the electronic device may be a mobile terminal or a server device. This embodiment does not limit the implementation form of the mobile terminal. The mobile terminal may be a smart handheld device, such as a smart phone, a tablet computer, a laptop computer or a desktop computer, etc.; or, the mobile terminal may also be a smart wearable device, such as a smart watch, a smart bracelet, etc.; or, the mobile terminal may also be various smart home appliances with display screens, such as a smart TV, a smart large screen or a smart robot, etc.

[0027] In an embodiment of the present application, a raw image script corresponding to the image generation process is generated in advance based on the work nodes in the tool area and their function codes, including: responding to the drag operation on the graphical user interface, dragging multiple work nodes required for the image generation process from the tool area to the editing area; responding to the editing operation of the association relationship between multiple work nodes in the editing area, determining the association relationship between multiple work nodes; generating a raw image script according to the function code of each work node and the association relationship between multiple work nodes. Among them, the editing operation can be a connection operation. Taking the editing of the association relationship between work node A and work node B as an example, the connection operation can be specifically implemented by clicking work node A, and then clicking work node B, to realize the connection between work node A and work node B. The connection can have an arrow, and the arrow points to the work node in the later execution order.

[0028] In an embodiment of the present application, an AI face replacement node is run, and an AI face replacement model is used to perform a face-changing operation on a real-person image to obtain a first intermediate image, including: running the AI ​​face replacement node, using the AI ​​face replacement model to extract first facial features of a first image template and second facial features of the real-person image, and generating a first intermediate image based on the first facial features and the second facial features; wherein the first image template is an image template adapted to the target user.

[0029] In an embodiment of the present application, the AI ​​face replacement model includes a first facial feature extraction network layer, a facial feature fusion layer, and an image fusion network layer; when the AI ​​face replacement node is run, the AI ​​face replacement model is used to extract the first facial feature of the first image template and the second facial feature of the real person image, and a first intermediate image is generated based on the first facial feature and the second facial feature, including: inputting the first image template and the real person image into the first facial feature extraction network layer, extracting the first facial feature of the first image template and the second facial feature of the real person image; inputting the first facial feature and the second facial feature into the facial feature fusion network layer, and when there is a difference between the first facial feature and the second facial feature, fusing the first facial feature and the second facial feature to obtain a third facial feature; inputting the third facial feature and the first image template into the image fusion network layer, identifying the facial feature positions in the first image template, and according to the facial feature position information, correcting the facial feature information in the first image template to the third facial feature information to obtain the first intermediate image.

[0030] In an optional embodiment, the first image template and the real person image are input into the first facial feature extraction network layer to extract the first facial features of the first image template and the second facial features of the real person image, including: first extracting primary features of the image, such as the edges and textures of the facial features; gradually combining the primary features into more complex feature representations, which contain initial facial feature information; extracting features according to the feature hierarchy to be extracted, and obtaining the facial feature information of the first image template and the real person image, respectively.

[0031] In another optional embodiment, the semantic segmentation model can also be used to perform semantic segmentation on the face image, and the user's gender attribute can be identified based on the initial facial feature information contained in the local image after semantic segmentation. The semantic segmentation model can be used to segment a more accurate local image containing the initial facial feature information, and the initial facial feature information can be extracted based on the accurate local image, which can reduce the amount of data processing on the one hand, and improve the accuracy of the extracted initial facial feature information on the other hand.

[0032] Among them, semantic segmentation refers to a technology in the field of computer vision and image processing, the purpose of which is to assign each pixel in the image to a specific category or object, so as to realize the recognition and segmentation of all objects in the image, such as the recognition and segmentation of facial features such as eyebrows, eyes, nose, mouth and ears. The semantic segmentation model is obtained by pre-training a plurality of facial image samples containing facial feature information. This embodiment does not limit the type of semantic segmentation model. For example, the semantic segmentation model can be a segmentation model based on deep learning of a convolutional neural network (CNN), a U-Net model, a Deep Lab (Deep Lab Semantic Image Segmentation, DeepLab) series model or a mobile network (Efficient Convolutional Neural Networks for Mobile Vision Applications V2, MobileNetV2) model, etc. Among them, the DeepLab series model includes but is not limited to DeepLabv1, DeepLabv2, DeepLabv3 and DeepLabv3+ models. The semantic segmentation model can also be a model based on the Transformer visual network structure (Vi s ion Transformer, ViT) class, such as the SETR (Semant ic Segmentat ion Transformer) model, the Segmenter model or the SegFormer model, and so on.

[0033] Optionally, the first facial feature and the second facial feature are input into a facial feature fusion network layer, and when there is a difference between the first facial feature and the second facial feature, the first facial feature and the second facial feature are fused to obtain a third facial feature, including: performing a similarity judgment on the first facial feature and the second facial feature, and determining that there is a difference between the first facial feature and the second facial feature when the judgment result is less than a similarity threshold; when there is a difference, obtaining a first weight of the first facial feature and a second weight of the second facial feature, and fusing the first facial feature and the second facial feature based on the first weight and the second weight to obtain the third facial feature.

[0034] In an embodiment of the present application, an image optimization node is run, and a style generation model is used to perform style optimization processing on a first intermediate image to obtain a second intermediate image, including: running the image optimization node, using the style generation model to extract portrait feature information of a second image template, and fusing a third facial feature into the first intermediate image to obtain a second intermediate image.

[0035] In an optional embodiment, the style generation model includes a portrait feature extraction network layer and a style adaptation network layer; the style image generation node is run to run the style generation model, and the portrait feature information of the second image template is extracted by the style generation model, and the third facial feature is integrated into the first intermediate image to obtain the second intermediate image, including: inputting the first intermediate image and the second image template into the portrait feature extraction network layer, extracting the first portrait feature information of the second image template, and the first portrait feature includes facial feature area features and other area features; inputting the first portrait feature information and the first intermediate image into the style adaptation network layer, identifying the style of the first portrait feature information and the first intermediate image, and when the style of the first portrait feature information and the style of the first intermediate image do not match, adjusting the style of the first intermediate image according to the portrait feature information to obtain the second intermediate image.

[0036] In an optional embodiment, the image generation model includes a problem area recognition network layer and an image synthesis network layer; the image generation node is run, and the second intermediate image and the advertising word image are synthesized using the image generation model to obtain a target image, including: inputting the second intermediate image into the problem area recognition network layer, identifying at least one problem area of ​​the second intermediate image, and using the at least one problem area as the placement position of each advertising word, wherein the problem area is other than the face area; inputting the second intermediate image, at least one advertising word image, and the position information of at least one problem area into the image synthesis network layer, and adding at least one advertising word image to at least one problem area to obtain the target image, Figure 2 is an example of a target image.

[0037] Further optionally, the image generation process also includes: a first judgment node and a text image node; the first judgment node is located between the label filtering node and the image matching node, and when the judgment result of the first judgment node is yes, it goes to the image matching node, and when the judgment result is no, it goes to the text image node; the running process of the image script also includes: before running the image matching node, running the first judgment node, and judging whether there is already a real-life image of the target user in the real-life image library according to the target text description information; when the judgment result is yes, running the image matching node and its subsequent nodes, and when the judgment result is no, running the text image node, inputting the target text description information into the text image model, determining the second portrait feature of the target user according to the target text description information, generating a real-life image of the target user based on the second portrait feature, and storing the real-life image in the real-life image library.

[0038] Among them, the text graph model is used to output an image that conforms to the description of the target text information after a certain number of iterations based on the input target text information. This embodiment does not limit the type of text graph model, for example, it can be an SD model. In addition, when selecting an SD model, a model that has been generalized after pre-training, adapted to the user's ethnicity, and generates high-definition images can be selected, such as the Stab le Diffus ion XL (SDXL for short) model. The SDXL model is a new version of the SD model series, which has made significant improvements and optimizations in image generation quality.

[0039] Among them, the target text information refers to at least one prompt word corresponding to the user's gender and the user's actual needs. At least one prompt word includes at least a positive prompt word and a reverse prompt word. The positive prompt word is used to guide the model to generate a text description or keyword of a specific type or style and high-quality image. The reverse prompt word refers to a text description or keyword used to guide the model to avoid generating certain specific features or elements. The reverse prompt word is opposite to the positive prompt word, and the two work together in the generation process of the model to achieve more accurate control of facial features. In addition, this embodiment does not limit the method for obtaining the target text information. For example, the target text information is stored in association with each ID photo template, that is, each ID photo template is associated with the positive prompt word and the reverse prompt word related to the ID photo template.

[0040] Optionally, the target text description information is input into the text-image model, the second portrait features of the target user are determined according to the target text description information, and a real-life image of the target user is generated based on the second portrait features, including: determining the relative position information of each second portrait feature in the image according to the target text description information; determining the resolution of the real-life image, creating a latent space of the real-life image according to the resolution, and adapting the vector dimension of the latent space to the feature space dimension of the real-life image; using an encoder, mapping each second portrait feature and the relative position information to the latent space to obtain latent space features of the portrait features in the latent space; according to at least one prompt word, using a sampler, collecting each feature adapted to at least one prompt word from the latent space features of the portrait features; using a decoder, decoding the portrait feature vector to obtain a real-life image.

[0041] In some embodiments, in order to enhance the functionality and flexibility of the Wensheng graph model, a plug-in model may be integrated into the Wensheng graph model. This embodiment does not specifically limit the type of plug-in model. For example, the plug-in model may be an Instant ID model. The Instant ID model includes a facial feature constraint model and a facial feature extraction model. The facial feature constraint model may be a Controlnet model, and the facial feature extraction model may be an Ipadapter model. The facial feature constraint model and the facial feature extraction model coexist, that is, the image processing is performed separately through the decoupled cross-attention mechanism, and they will not affect each other.

[0042] Further optionally, the image generation process also includes: a digital human video generation node located after the image generation node, and the running process of the image generation script also includes: running the digital human video generation node, using the video generation model, based on the target image and initial text description information, to generate a digital human introduction video for the target user.

[0043] Optionally, a video generation model is used to generate a digital human introduction video for a target user based on a target image and initial text description information, including: determining a corresponding target digital human image based on a target AI image; obtaining a spoken copy and video scene description information based on the text description information corresponding to each node; performing audio conversion processing on the spoken copy to obtain a target audio; generating a target video based on the video scene description information; and generating a digital human introduction video based on the target audio, target video, and target digital human image. The spoken copy may include text information for explanation in a video, and the video scene description information may be used to explain the visual content in the video.

[0044] Further optionally, the target video may include multiple storyboard videos, each storyboard video is a part of the target video, and each part of the video corresponds to local scene description information. Then, based on the video scene description information, the target video is generated, including: determining the script structure according to the scene description information of each part of the video, the script structure includes the plot structure within the scene and the correlation between the scene description information of each part of the video; using the artificial intelligence big model to generate the script content based on the script structure, the script content includes the spoken text and the storyboard video description information; determining the digital human image according to the storyboard video description information; based on the digital human image, converting the spoken text into spoken audio; synthesizing the storyboard description information, the spoken audio and the digital human image to obtain each storyboard video.

[0045] In an optional embodiment, the artificial intelligence big model can be used to generate the script content based on the original materials corresponding to the first file page and the script structure, including: using the first artificial big model based on the video scene description information and the script structure to generate the oral copy of each storyboard; using the second artificial intelligence big model based on the oral copy of each storyboard to generate the storyboard description information corresponding to each storyboard. Among them, the oral copy can include the narration or dialogue text in the current storyboard, and the storyboard description information can be used to illustrate the visual content of each storyboard, for example, it can include scene layout, character action, attribute information, etc. The first artificial intelligence big model can understand the overall framework of the video and the order of each storyboard based on the script structure, and understand the basis of the script content based on the video scene description information of the target object, so that the script content of each storyboard can be intelligently generated. The storyboard description information can make the picture of the generated storyboard video accurately convey the expected information and emotions, and can also maintain coordination and consistency with the oral copy. Among them, the first artificial intelligence model and the second artificial intelligence model are AI intelligent models. The first artificial intelligence large model can also integrate the video scene description information into the script based on the provided video scene description information and the script structure through understanding the video scene description information, and generate a spoken copy that conforms to the script structure and storyboard. The second artificial intelligence large model can generate storyboard description information for describing the visual content corresponding to the spoken copy through understanding the spoken copy, so that the visual effects presented in the generated storyboard video match the spoken content. Therefore, through the collaborative work of the first video generation model and the second video generation model, high-quality script content can be efficiently generated, and the copy in the script content is closely integrated with the visual effects.

[0046] In an optional embodiment, the first artificial intelligence model is used to generate a storyboard oral copy based on the video scene description information and the script structure, including: obtaining a script template and script requirements corresponding to the script structure; generating a first prompt word according to the first prompt template based on the video scene description information, the script template and the script requirements; inputting the first prompt word into the first artificial intelligence model, so as to use the first artificial intelligence model to generate a storyboard oral copy based on the video scene description information, the script template and the script requirements according to the script requirements. Among them, the script template may refer to the script content pre-generated under the script structure. The first prompt template may be a predefined framework or structure for guiding and standardizing the content input into the first artificial intelligence model. In an embodiment of the present application, the first prompt template may include elements such as script structure, target audience, copy length, format requirements, language and tone, and specific requirements. The first prompt word can be generated by filling the video scene description information, the script template and the script requirements into the corresponding elements of the first prompt template, and by inputting the first prompt word into the first artificial intelligence model, the first artificial intelligence model can generate a storyboard oral copy based on the video scene description information, the script template and the script requirements according to the script requirements.

[0047] It should be noted that the first artificial intelligence model can also parse the pictures and video content contained in each file page to obtain the content information implied in the pictures or videos, and the video scene description information can include the content information implied in the pictures or videos.

[0048] It should also be noted that the reasoning process of each model in the above embodiment can be performed on the server side or on the terminal device. When the reasoning process is performed on the terminal device, the terminal device can be connected to the server side for communication, and the terminal device can send a model conversion request message to the server side, the model conversion request message includes the system type of the mobile terminal, available resource information and the model function required by the mobile terminal, so as to request the server side to convert each model into a text copy generation model and an audio-visual copy generation model based on the system type, available resource information and model function; receive the model conversion response message returned by the server side, the model conversion response message is used to notify the mobile terminal to obtain the model files of the text copy generation model and the audio-visual copy generation model from the remote persistent storage space; obtain the model files of the text copy generation model and the audio-visual copy generation model from the remote persistent storage space and store them in the local persistent storage space, and when using the text copy generation model and the audio-visual copy generation model, load the model files from the local persistent storage space to run the text copy generation model and the audio-visual copy generation model.

[0049] Among them, the text copy generation model and the audio-visual copy generation model are obtained by converting the model formats of the respective models, and each model is developed on the server side through a target model development framework suitable for the server side.

[0050] The communication connection between the server and the terminal device can be achieved by wireless communication connection or wired connection. For the wireless communication connection, optionally, the communication connection can be achieved through a mobile network. Accordingly, the network standard of the mobile network can be 2G (GSM), 2.5G (GPRS), 3G (WCDMA, TD-SCDMA, CDMA2000, UTMS), 4G (LTE), 4G+ (LTE+), 5G, WiMax or any of the new network standards that will appear in the future. In addition, when the server and the terminal device are located in the same local area network, wireless communication connection can also be achieved through Bluetooth, WiFi, infrared, zigbee or NFC. For the wired communication connection, optionally, the communication connection can be achieved through Ethernet, serial communication, universal serial bus (USB), high-definition multimedia interface (HDMI) and the like. Among them, Ethernet is usually used for local area network (LAN) connection, which supports multiple transmission rates such as 10Mbps, 100Mbps, 1Gbps, 10Gbps, etc. Serial communication includes standards such as RS-232 and RS-485; it is used to connect computers and external devices such as keyboards, mice, printers, etc. USB includes USB 1.1, USB2.0, USB 3.0, USB 3.1, USB 4 and other versions, each with different transmission rates.

[0051] The technical solutions provided in the above-mentioned embodiments of the present application utilize a visual node editor to generate a drawing script for an image generation process including multiple nodes connected in series based on each node and its function code in the tool area. In response to the image generation operation, each node in the image generation process can be executed based on the initial text description information to obtain the target image. A fixed image generation process is generated based on the visual node editor, which can improve the stability of image generation. Furthermore, when executing each node in the image generation process based on the initial text description information, different AI models are used for image generation and repair to improve the quality of the generated image. In addition, the target image contains an advertising word image corresponding to an advertising word, which allows users to quickly understand key information based on the target image and improve the conversion rate of the landing page corresponding to the target image.

[0052] Figure 3 The structure diagram of an electronic device provided by an exemplary embodiment of the present application is shown in FIG. Figure 3As shown, it includes: a memory 30a and a processor 30b; the memory 30a is used to store a computer program; the processor 30b is coupled to the memory 30a and is used to execute the computer program to implement the following steps:

[0053] Generate a raw image script corresponding to the image generation process in advance based on the work nodes and their function codes in the tool area; the image generation process includes: a label screening node, an image matching node, an AI face replacement node, an image optimization node, an advertising word image generation node and an image generation node connected in series in sequence; in response to the image generation operation, obtain the initial text description information of the target user, use the initial text description information as the input information of the image generation process, and run the raw image script to generate the target image.

[0054] The running process of the image generation script includes:

[0055] S1. Run the tag screening node to determine multiple tags that are suitable for the target user, and select target text description information that is suitable for the multiple tags from the initial text description information;

[0056] S2, running the image matching node to match the real person image of the target user from the real person image library according to the target text description information;

[0057] S3, running the AI ​​face replacement node, and using the AI ​​face replacement model to perform a face replacement operation on the real person image to obtain a first intermediate image;

[0058] S4, running the image optimization node, and performing style optimization processing on the first intermediate image using the style generation model to obtain a second intermediate image;

[0059] S5, running the advertising word image generation node, determining at least one advertising word based on the target text description information, and generating an advertising word image corresponding to each advertising word;

[0060] S6. Run the image generation node and use the image generation model to synthesize the second intermediate image and the advertising word image to obtain a target image.

[0061] In an embodiment of the present application, when the processor 30b generates a graphics generation script corresponding to the image generation process based on the work nodes and their function codes in the tool area in advance, it is specifically used to: respond to the drag operation on the graphical user interface, drag multiple work nodes required for the image generation process from the tool area to the editing area; respond to the editing operation on the association relationship between multiple work nodes in the editing area, determine the association relationship between multiple work nodes; generate a graphics generation script according to the function code of each work node and the association relationship between multiple work nodes.

[0062] In an embodiment of the present application, when the processor 30b runs the AI ​​face replacement node and uses the AI ​​face replacement model to perform a face-changing operation on the real-person image to obtain a first intermediate image, it is specifically used to: run the AI ​​face replacement node, use the AI ​​face replacement model to extract the first facial features of the first image template and the second facial features of the real-person image, and generate a first intermediate image based on the first facial features and the second facial features; wherein the first image template is an image template adapted to the target user.

[0063] In an embodiment of the present application, the AI ​​face replacement model includes a first facial feature extraction network layer, a facial feature fusion layer, and an image fusion network layer; when the processor 30b runs the AI ​​face replacement node, uses the AI ​​face replacement model to extract the first facial feature of the first image template and the second facial feature of the real person image, and generates a first intermediate image based on the first facial feature and the second facial feature, it is specifically used to: input the first image template and the real person image into the first facial feature extraction network layer, extract the first facial feature of the first image template and the second facial feature of the real person image; input the first facial feature and the second facial feature into the facial feature fusion network layer, and when there is a difference between the first facial feature and the second facial feature, fuse the first facial feature and the second facial feature to obtain a third facial feature; input the third facial feature and the first image template into the image fusion network layer, identify the facial features position in the first image template, and according to the facial features position information, correct the facial features information in the first image template to the third facial features information to obtain the first intermediate image.

[0064] In the embodiment of the present application, when the processor 30b runs the image optimization node and uses the style generation model to perform style optimization processing on the first intermediate image to obtain the second intermediate image, it is specifically used to: run the image optimization node, use the style generation model to extract the portrait feature information of the second image template, and fuse the third facial feature into the first intermediate image to obtain the second intermediate image.

[0065] In an optional embodiment, the style generation model includes a portrait feature extraction network layer and a style adaptation network layer; when the processor 30b runs the style image generation node to run the style generation model, uses the style generation model to extract the portrait feature information of the second image template, and fuses the third facial feature into the first intermediate image to obtain the second intermediate image, it is specifically used to: input the first intermediate image and the second image template into the portrait feature extraction network layer, extract the first portrait feature information of the second image template, and the first portrait feature includes facial feature area features and other area features; input the first portrait feature information and the first intermediate image into the style adaptation network layer, identify the style of the first portrait feature information and the first intermediate image, and when the style of the first portrait feature information and the style of the first intermediate image do not match, adjust the style of the first intermediate image according to the portrait feature information to obtain the second intermediate image.

[0066] In an optional embodiment, the image generation model includes a problem area recognition network layer and an image synthesis network layer; when the processor 30b runs the image generation node and uses the image generation model to synthesize the second intermediate image and the advertising word image to obtain the target image, it is specifically used to: input the second intermediate image into the problem area recognition network layer, identify at least one problem area of ​​the second intermediate image, and use the at least one problem area as the placement position of each advertising word, and the problem area is other areas outside the face area; input the second intermediate image, at least one advertising word image and the position information of at least one problem area into the image synthesis network layer, and add at least one advertising word image to at least one problem area to obtain the target image.

[0067] Further optionally, the image generation process also includes: a first judgment node and a text image node; the first judgment node is located between the label filtering node and the image matching node, and when the judgment result of the first judgment node is yes, it goes to the image matching node, and when the judgment result is no, it goes to the text image node; the running process of the processor 30b in running the image script also includes: before running the image matching node, running the first judgment node, judging whether there is a real person image of the target user in the real image library according to the target text description information; when the judgment result is yes, running the image matching node and its subsequent nodes, and when the judgment result is no, running the text image node, inputting the target text description information into the text image model, determining the second portrait feature of the target user according to the target text description information, generating a real person image of the target user based on the second portrait feature, and storing the real person image in the real person image library.

[0068] Further optionally, the image generation process also includes: a digital human video generation node located after the image generation node, and the processor 30b also includes the following in the process of running the image generation script: running the digital human video generation node, using the video generation model, and generating a digital human introduction video for the target user based on the target image and the initial text description information; when the processor 30b generates the digital human introduction video for the target user based on the target image and the initial text description information using the video generation model, it is specifically used to: determine the corresponding target digital human image based on the target AI image; obtain the spoken text and video scene description information based on the text description information corresponding to each node; perform audio conversion processing on the spoken text to obtain the target audio; generate the target video based on the video scene description information; generate the digital human introduction video based on the target audio, the target video and the target digital human image.

[0069] Furthermore, if Figure 3 As shown, the electronic device also includes: a communication component 30c, a display 30d, a power component 30e, an audio component 30f and other components. Figure 3 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 3 Components shown.

[0070] The detailed implementation and beneficial effects of the electronic device provided in the embodiments of the present application have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0071] An exemplary embodiment of the present application further provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to implement the steps in the above-mentioned method embodiments.

[0072] An exemplary embodiment of the present application further provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the processor is enabled to implement the steps in the above-mentioned method embodiments.

[0073] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0074] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (Ultra Wide Band, UWB) technology, Bluetooth (BlueTooth, BT) technology and other technologies.

[0075] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0076] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0077] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0078] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, compact disk read-only memory (Compact Disk Read-Only Memory, CD-ROM), optical storage, etc.) that contain computer-usable program code.

[0079] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0080] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0082] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0083] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory (flashRAM). The memory is an example of a computer-readable medium.

[0084] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0085] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0086] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. An image generation method, characterized in that: A visual node editor applied to an electronic device, wherein the visual node editor displays a graphical user interface, wherein the graphical user interface includes a tool area and an editing area, wherein the tool area includes a working node based on AI-generated images, and each working node corresponds to a respective function code, wherein the method includes: Generate a raw image script corresponding to the image generation process in advance based on the work nodes and their function codes in the tool area; the image generation process includes: a label screening node, an image matching node, an AI face replacement node, an image optimization node, an advertising word image generation node and an image generation node connected in series in sequence; In response to the image generation operation, initial text description information of the target user is obtained, the initial text description information is used as input information of the image generation process, and the image generation script is run to generate the target image; wherein the running process of the image generation script includes: Running the tag screening node to determine a plurality of tags adapted to a target user, and selecting target text description information adapted to the plurality of tags from the initial text description information; Running the image matching node to match the real person image of the target user from a real person image library according to the target text description information; Running the AI ​​face replacement node, and performing a face replacement operation on the real person image using the AI ​​face replacement model to obtain a first intermediate image; Running the image optimization node to perform style optimization processing on the first intermediate image using a style generation model to obtain a second intermediate image; Running the advertising word image generation node, determining at least one advertising word based on the target text description information, and generating an advertising word image corresponding to each advertising word; The image generation node is run, and the second intermediate image and the advertising word image are synthesized using an image generation model to obtain the target image.

2. The method according to claim 1, characterized in that Generate a drawing script corresponding to the image generation process in advance based on the work nodes and their function codes in the tool area, including: In response to a drag operation on the graphical user interface, drag a plurality of work nodes required for the image generation process from the tool area to the editing area; In response to an editing operation on the association relationship between the plurality of working nodes in the editing area, determining the association relationship between the plurality of working nodes; The graph generation script is generated according to the function code of each working node and the association relationship between multiple working nodes.

3. The method according to claim 1, characterized in that Running the AI ​​face replacement node and using the AI ​​face replacement model to perform a face replacement operation on the real person image to obtain a first intermediate image includes: Run the AI ​​face replacement node, use the AI ​​face replacement model to extract the first facial features of the first image template and the second facial features of the real person image, and generate a first intermediate image based on the first facial features and the second facial features; wherein the first image template is an image template adapted to the target user.

4. The method according to claim 3, characterized in that The AI ​​face replacement model includes a first facial feature extraction network layer, a facial feature fusion layer, and an image fusion network layer; when the AI ​​face replacement node is run, the first facial feature of the first image template and the second facial feature of the real person image are extracted by using the AI ​​face replacement model, and a first intermediate image is generated based on the first facial feature and the second facial feature, including: Inputting the first image template and the real person image into the first facial features extraction network layer to extract the first facial features of the first image template and the second facial features of the real person image; Inputting the first facial feature and the second facial feature into the facial feature fusion network layer, and fusing the first facial feature and the second facial feature to obtain a third facial feature when there is a difference between the first facial feature and the second facial feature; The third facial feature and the first image template are input into the image fusion network layer, the positions of the facial features in the first image template are identified, and according to the facial feature position information, the facial feature information in the first image template is corrected to correspond to the third facial feature information to obtain a first intermediate image.

5. The method according to claim 4, characterized in that Running the image optimization node to perform style optimization processing on the first intermediate image using the style generation model to obtain a second intermediate image includes: The image optimization node is run, the portrait feature information of the second image template is extracted using the style generation model, and the third face feature is integrated into the first intermediate image to obtain a second intermediate image.

6. The method according to claim 5, characterized in that The style generation model includes a portrait feature extraction network layer and a style adaptation network layer; running the style image generation node to run the style generation model, using the style generation model to extract portrait feature information of the second image template, and fusing the third face feature into the first intermediate image to obtain a second intermediate image, including: Inputting the first intermediate image and the second image template into a portrait feature extraction network layer, extracting first portrait feature information of the second image template. The first portrait feature includes features of facial features and other regions. The first portrait feature information and the first intermediate image are input into the style adaptation network layer, and the styles of the first portrait feature information and the first intermediate image are identified. When the style of the first portrait feature information and the style of the first intermediate image do not match, the style of the first intermediate image is adjusted according to the portrait feature information to obtain a second intermediate image.

7. The method according to any one of claims 1 to 6, characterized in that: The image generation model includes a problem area recognition network layer and an image synthesis network layer; running the image generation node and synthesizing the second intermediate image and the advertising word image using the image generation model to obtain the target image includes: Inputting the second intermediate image into the problem area recognition network layer, identifying at least one problem area of ​​the second intermediate image, and using the at least one problem area as a placement position for each of the advertising words, wherein the problem area is an area other than a face area; The second intermediate image, the at least one advertising word image and the position information of the at least one problem area are input into the image synthesis network layer, and the at least one advertising word image is added to the at least one problem area to obtain the target image.

8. The method according to any one of claims 1 to 6, characterized in that: The image generation process also includes: a first judgment node and a text graph node; The first judgment node is located between the label screening node and the image matching node. When the judgment result of the first judgment node is yes, it goes to the image matching node. When the judgment result is no, it goes to the text generation node. The running process of the image generation script also includes: Before running the image matching node, running the first judgment node to judge whether a real person image of the target user already exists in the real image library according to the target text description information; When the judgment result is yes, the image matching node and its subsequent nodes are run, and when the judgment result is no, the text graph node is run, the target text description information is input into the text graph model, and the second portrait feature of the target user is determined according to the target text description information. Based on the second portrait feature, a real person image of the target user is generated, and the real person image is stored in the real person image library.

9. The method according to any one of claims 1 to 6, characterized in that: The image generation process further includes: a digital human video generation node located after the image generation node, and the running process of the image generation script further includes: running the digital human video generation node, using a video generation model, and generating a digital human introduction video of the target user based on the target image and the initial text description information; Using a video generation model, based on the target image and the initial text description information, a digital human introduction video of the target user is generated, including: Based on the target AI image, determine the corresponding target digital human image; Based on the text description information corresponding to each node, the spoken copy and video scene description information are obtained; Performing audio conversion processing on the spoken text to obtain target audio; Based on the video scene description information, generating a target video; A digital human introduction video is generated based on the target audio, the target video and the target digital human image.

10. An electronic device, characterized in that: include: Memory and processor; The memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps in the method according to any one of claims 1 to 9.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to implement the steps in the method according to any one of claims 1 to 9.

12. A computer program product, characterized in that The computer program product comprises a computer program / instruction, which, when executed by a processor, enables the processor to implement the steps of any one of the methods of claims 1 to 9.

Images