Image generation method, electronic device, storage medium, and computer program product

Abstract

The application provides an image generation method, an electronic device, a storage medium and a computer program product. The method comprises: in response to a prompt word for a first image, performing semantic analysis on the prompt word to obtain first information; wherein the prompt word is used to describe one or more target roles that need to be drawn in the first image, and the first information represents the attributes or characteristics of the one or more target roles; in response to a setting operation for the one or more target roles, obtaining first spatial constraint parameters of the one or more target roles; and based on the first information and the first spatial constraint parameters of the one or more target roles, outputting a second image, wherein the second image includes images of the one or more target roles drawn in the first image. The above scheme improves the accuracy of text-to-image and the matching degree with user demand.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image generation method, electronic device, storage medium, and computer program product. Background Technology

[0002] Text-to-image generation refers to the process of taking a text description as input and having a computer generate one or more images related to that description. It aims to establish an interpretable mapping between image space and text semantic space, converting low-dimensional semantic information of text into high-dimensional image information. It is a challenging cross-modal task.

[0003] The ability to generate images from text is developing rapidly. For example, the Stable Diffusion model has shown good performance in many application scenarios such as text-to-image conversion. Among related technologies, the method of generating images from text uses input prompts to generate images. However, the semantic understanding of the prompts may be insufficient, resulting in images generated based on the prompts that differ significantly from the user's expectations. Summary of the Invention

[0004] This application provides an image generation method, electronic device, storage medium, and computer program product, which improves the accuracy of text-generated images and their matching degree with user needs.

[0005] The technical solution of this application embodiment is implemented as follows: This application provides an image generation method, the method comprising: In response to a user's prompt for the first image, the prompt is semantically parsed to obtain first information; wherein the prompt describes one or more target characters that need to be drawn in the first image, and the first information characterizes the attributes or features of the one or more target characters; In response to a setting operation for the one or more target roles, the first spatial constraint parameters of the one or more target roles are obtained; Based on the first information and the first spatial constraint parameters of the one or more target characters, a second image is output, which includes images of the one or more target characters drawn in the first image.

[0006] This application provides an image generation apparatus, including: The parsing unit is configured to perform semantic parsing on the prompt words given by the user for the first image in response to obtain first information; wherein the prompt words are used to describe one or more target characters that need to be drawn in the first image, and the first information characterizes the attributes or features of the one or more target characters; The acquisition unit is configured to acquire the first spatial constraint parameters of the one or more target roles in response to a setting operation for the one or more target roles; An image output unit is configured to output a second image based on the first information and the first spatial constraint parameters of the one or more target characters, wherein the second image includes an image of the one or more target characters drawn in the first image.

[0007] This application provides an electronic device, characterized in that it includes a processor and a memory for storing a computer program or computer-executable instructions capable of running on the processor, wherein when the processor is used to run the computer program or computer-executable instructions, it performs the steps of any of the above methods.

[0008] This application provides a storage medium storing a computer program or computer-executable instructions thereon, characterized in that the computer program or computer-executable instructions, when executed by a processor, implement the steps of any of the above methods.

[0009] This application provides a computer program product, including a computer program or computer executable instructions, characterized in that the computer program or computer executable instructions implement the steps of any of the above methods when executed by a processor.

[0010] The embodiments of this application have the following beneficial effects: In response to a user's prompt for a first image, semantic parsing is performed on the prompt to obtain first information. The prompt describes one or more target characters to be drawn in the first image, and the first information characterizes the attributes or features of the one or more target characters. In response to a setting operation for the one or more target characters, first spatial constraint parameters for the one or more target characters are obtained. Based on the first information and the first spatial constraint parameters of the one or more target characters, a second image is output, which includes images of the one or more target characters drawn in the first image. It can be seen that in this embodiment, semantic parsing can be performed on the prompt to obtain the attributes or features of each target character to be drawn; furthermore, setting operations for the target characters can be converted into corresponding first spatial constraint parameters. This makes the first image output based on the first information and the first spatial constraint parameters of each target character more consistent with the user's expectations, improving the accuracy of the text-based image and its matching degree with user needs. Attached Figure Description

[0011] Figure 1 This is a schematic flowchart of an image generation method provided in an embodiment of this application; Figure 2This is a schematic diagram of an image generation method provided in an embodiment of this application; Figure 3 This is an example diagram of the first image provided in the embodiments of this application; Figure 4 This is a first example image provided by an embodiment of this application, showing a user defining a drawing area in a first image; Figure 5 This is a second example image provided in the embodiments of this application, showing a user defining a drawing area in a first image; Figure 6 This is an example diagram of the interactive interface for user input prompts provided in the embodiments of this application; Figure 7 This is an example diagram provided in this application embodiment to guide users in setting the position and size of a target character on a first image; Figure 8 This is an example diagram illustrating the position and size of a user-defined target role as provided in an embodiment of this application; Figure 9 This is an example diagram illustrating the setup effect of the rabbit and mouse provided in the embodiments of this application; Figure 10 This is an example diagram of an image drawn based on prompt words, provided in an embodiment of this application. Figure 11 This is a schematic diagram of the structure of the image generation apparatus provided in the embodiments of this application; Figure 12 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0012] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0013] The ability to generate images from text is developing rapidly. For example, the Stable Diffusion model performs well in many application scenarios such as text-to-image conversion. However, most user-input prompts suffer from insufficient description and lack of descriptive prompts. Currently, the creative performance of models like Stable Diffusion cannot be fully utilized. Instead, images can only be generated using traditional image generation methods based on the input prompts. This may result in insufficient semantic understanding of the prompts, leading to a significant difference between the generated image and the user's expectations.

[0014] Based on this, in various embodiments of this application, in response to a user's prompt word for the first image, the prompt word is semantically parsed to obtain first information; wherein, the prompt word is used to describe one or more target characters to be drawn in the first image, and the first information characterizes the attributes or features of the one or more target characters; in response to a setting operation for the one or more target characters, first spatial constraint parameters of the one or more target characters are obtained; based on the first information and the first spatial constraint parameters of the one or more target characters, a second image is output, the second image including the image of the one or more target characters drawn in the first image. It can be seen that in the embodiments of this application, the prompt word can be semantically parsed to obtain the attributes or features of each target character to be drawn; and the setting operation for the target character can be converted into the corresponding first spatial constraint parameters. This makes the first image output based on the first information and the first spatial constraint parameters of each target character more consistent with the user's expected image, improving the accuracy of the text-based image and the matching degree with user needs.

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] This application provides an image generation method applied to an electronic device. The electronic device supports or has text-to-image generation functionality, or the electronic device can run a dynamic canvas system. The electronic device may include a terminal and / or a server, which may be a cloud server or a non-cloud server, or a server cluster or computing platform. The electronic device may have a display screen or be able to connect to a display device. Figure 1 As shown, the method includes: Step 101: In response to a prompt word from the user for the first image, perform semantic parsing on the prompt word to obtain first information; wherein, the prompt word is used to describe one or more target characters that need to be drawn in the first image, and the first information characterizes the attributes or features of the one or more target characters.

[0017] Here, users can load or upload a first image through an interactive window. This first image can be understood as a base image or foundational image that needs to be modified or used to draw other content. The first image can be an image saved locally on the terminal, an image captured by the terminal, or an image downloaded by the terminal; there are no restrictions here. The interactive window can be the interface of an application (App) or a webpage; that is, users can load or upload the first image through an App with text-to-image functionality, in which case the electronic device can be a terminal or a server; users can also load or upload the first image through the interface of a webpage with text-to-image functionality, in which case the electronic device is a server.

[0018] Electronic devices can receive the first image directly or indirectly. For example, when the electronic device is a terminal, it can directly receive the first image; when the electronic device is a server, it can receive the first image indirectly. The server receives the first image through a device that displays an interactive window. In other words, the device that displays the interactive window can send the first image loaded or uploaded by the user to the server.

[0019] Upon acquiring a first image, the electronic device prompts the user to input a prompt word. For example, an input box can be displayed in the first image, prompting the user to input a prompt word. After the user inputs a prompt word for the first image, the electronic device acquires the input prompt word, performs semantic parsing on the prompt word in response, and obtains first information. The first information may include one or more of the following: type and attribute information of one or more target characters, and spatial relationships between target characters. When there is one target character, the first information includes the type and / or attribute information of the target character; when there are multiple target characters (e.g., greater than or equal to two), the first information includes one or more of the type and attribute information of multiple target characters, and spatial relationships between target characters. Target characters may include one or more of the following: people, animals, plants, objects, landscapes (e.g., waterfalls, clouds, mountains, etc.) to be drawn, and may also include the background image to be drawn. The attribute information of the target characters may include, but is not limited to, color, and / or, appendages or associated objects of the target characters. The spatial relationships between target characters may also be described as positional relationships. For example, the cue phrase "A mouse wearing a red hat stands next to a yellow rabbit, the rabbit holding a shovel" indicates that the target characters include a mouse and a rabbit. The mouse's attributes include wearing a red hat, and the rabbit's attributes include being yellow and holding a shovel. The spatial relationship between the mouse and the rabbit is that the mouse stands next to the rabbit. Another example is the cue phrase "A white crane in an ancient Chinese landscape painting, with a waterfall in the distance, 8K detail" indicating that the target characters include a crane and a waterfall. The crane's attributes include being white and having 8K detail, and the waterfall's attributes include having 8K detail. The spatial relationship between the crane and the waterfall is that the waterfall is in the distance from the crane.

[0020] It should be noted that before the user enters a prompt for the first image, for example, when the user loads the first image through an interactive window, the user can also define a drawing area, which can also be described as an editing area; or the electronic device can prompt the user to select a drawing area, or analyze the first image, mark candidate drawing areas, and prompt the user to select a drawing area; after the user selects a drawing area, the electronic device prompts the user to enter a prompt. The user can manually slide to select the drawing area, for example, by drawing a closed curve on the first image, and the electronic device returns the selected drawing area in real time; or, when the user manually draws a closed curve, the electronic device performs edge detection on the roughly selected area, identifies the boundary of the roughly selected area, and returns the drawing area. Of course, the electronic device can also return a pixel-level mask of the roughly selected area.

[0021] To avoid reducing the accuracy of text-to-text graphs due to semantic fragmentation, in one embodiment, the first information includes one or more of the type and attribute information of the one or more target roles and the spatial relationships between the target roles; The semantic parsing of the prompt words to obtain first information includes one or more of the following: Natural language processing techniques are used to perform semantic analysis on the prompt words, identify one or more entities included in the prompt words, and obtain the type of the one or more target roles; The entity's modifiers are identified from the prompts to obtain the target character's attribute information; Based on the syntactic dependency relationships between entities in the prompt words, the spatial relationships between target characters are determined.

[0022] Here, electronic devices can utilize Natural Language Processing (NLP) technology to analyze and parse the prompt words, identifying the various entities included in the prompt words and obtaining the types of each target role, with one entity corresponding to one target role type. The electronic devices also use NLP technology to identify the modifying attributes of entities, obtaining attribute information for the target roles. Modifying attributes refer to additional information that describes, limits, or names the characteristics, states, and subordinate relationships of entities, enriching their semantic expression. Syntactic dependency relationships between entities are grammatical structures describing the dominance and modification relationships between entities in a sentence at the syntactic level. For example, if the prompt word is "A mouse wearing a red hat stands next to a yellow rabbit, the rabbit holding a shovel," the identified entities include the mouse and the rabbit. The mouse's modifying attribute is a red hat, and the rabbit's modifying attribute is a yellow body.

[0023] It should be noted that electronic devices can also call pre-trained natural language processing models to analyze prompt words, and identify information such as entities and their modifying attributes, as well as syntactic dependencies between entities, from the prompt words.

[0024] Step 102: In response to the setting operation for the one or more target roles, obtain the first spatial constraint parameters of the one or more target roles.

[0025] Here, the settings operation can be used to set the position and / or size of a target character. The electronic device prompts the user to set the position and / or size of the target character. For example, the electronic device can output a dialog box prompting the user to set the position and / or size of the target character. Specifically, the electronic device can output a visual drag-and-drop interface and prompt the user to set the position and / or size of the target character. The visual drag-and-drop interface can support the user to adjust the position and / or size of the target character through dragging, gesture scaling, rotation, and other operations. It should be noted that when there are multiple target characters, the electronic device can prompt the user to set the position and / or size of each target character separately. For example, the electronic device prompts the user to set the position and / or size of one target character, and after the user completes the setting for that target character, it prompts the user to set the position and / or size of another target character. The electronic device can also prompt the user to set the position and / or size of each target character at once. In this case, the user needs to mark the target objects corresponding to each position to avoid misjudgment.

[0026] In response to a user's setting operation for each target character, the electronic device determines the size proportion of each target character in the first image or drawing area; based on the size proportion of each target character in the first image or drawing area, it determines the first spatial constraint parameters of each target character. The first spatial constraint parameters of the target character are used to constrain the position of the target character, and may also constrain the size of the target character; the size may include length, width, and depth.

[0027] It should be noted that the electronic device can also output or display the size ratio of each target character in the first image or drawing area in real time, so that the user can determine whether to adjust the position and / or size of the target character.

[0028] To avoid spatial control ambiguity and make the text image more precise and controllable, it is possible to support users in setting the position and / or size of each target character separately. Based on this, in one embodiment, the step of obtaining the first spatial constraint parameters of the one or more target characters in response to the setting operation for the one or more target characters includes: In response to a setting operation for a first target character, second information of the first target character is obtained. The second information includes the position information and / or size information of the first target character in the first image. The first target character is any one of the one or more target characters. Based on the second information of the first target role, the normalized coordinates of the first target role are determined. The normalized coordinates include the center coordinates and bounding box coordinates of the target role. Based on the normalized coordinates of the first target role, the first spatial constraint parameters of the first target role are determined.

[0029] Here, the electronic device prompts the user to set the position and / or size of the first target character. The setting operations for the first target character can include dragging gestures to scale, rotating, etc. For example, the electronic device can output a dialog box prompting the user to set the position and / or size of the first target character; the user can adjust the position and / or size of the first target character in the first image or its drawing area by dragging, scaling, rotating, etc.; the electronic device acquires or records the position and / or size ratio of the first target character in the first image, obtaining the second information of the first target character. Of course, if the first image includes a drawing area, the electronic device can also acquire or record the position and / or size ratio of the first target character in the drawing area, obtaining the second information of the first target character. The second information includes the position information and / or size information of the first target character in the first image or drawing area, and the size information includes two-dimensional dimensions (size) or three-dimensional dimensions (size and depth).

[0030] Having obtained the second information of the first target character, the second information is normalized to obtain the normalized coordinates of the first target character, and the normalized coordinates of the first target character are recorded. Based on the normalized coordinates of the first target character, the first spatial constraint parameters of the first target character are determined or generated. For example, the normalized coordinates of the first target character are converted into the first spatial constraint parameters, which indicate the position of the center and boundary of the image of the first target character.

[0031] Step 103: Based on the first information and the first spatial constraint parameters of the one or more target characters, output a second image, which includes images of the one or more target characters drawn in the first image.

[0032] Here, the electronic device can invoke a pre-trained image generation model to generate a second image based on the first information and the first spatial constraint parameters of the one or more target characters, and output the second image. Outputting the second image can be done by sending it to a display device or page for display, or by the electronic device directly displaying the second image. The image generation model can include, but is not limited to, a stable diffusion model, such as ControlNet. The input to the image generation model is the first information and the first spatial constraint parameters of each target character, and the output of the image generation model can be an image of each target character or the second image.

[0033] The electronic device can also generate images of each target character based on the attributes or features of each target character represented by the first information, and the first spatial constraint parameters of each target character, and generate and output a second image based on the images of all target characters. The first spatial constraint parameters of the target character are used to determine the location where the image of the target character is drawn, and can also determine the size of the image of the target character.

[0034] For example, if the first information includes the type and / or attribute information of the target character, the electronic device generates an image of the target character for any given target character based on the type and / or attribute information of the target character and the first spatial constraint parameter of the target character; if the first information does not include the spatial relationship between the target characters, then a second image is generated based on the images of all the target characters; if the first information includes the spatial relationship between the target characters, then a second image is generated based on the spatial relationship between the target characters and the images of all the target characters.

[0035] The process of generating a second image based on the images of all target characters can include: fusing or compositing the images of all target characters to obtain a second image; or, in scenarios where the target characters do not include a background image but a background image needs to be represented, a background image can be generated based on the images of all target characters, and then fusing or compositing the images of all target characters and the background image to obtain a second image. Generating a background image based on the images of all target characters can be based on the texture and / or color of each target character's image to generate a background image that is clearly distinguishable from the target character's image, or it can be based on the images or pixel values ​​surrounding the target character's image to make the background image similar to or compatible with the images surrounding the target character, such as making the grass around the target character appear continuous in the background image.

[0036] Generating a second image based on the spatial relationships between target characters and the images of all target characters can include: performing image synthesis on the images of all target characters based on the spatial relationships between target characters to obtain a second image; or, performing image synthesis on the images of all target characters based on the spatial relationships between target characters to obtain a synthesized image, and generating a background image based on the images of each target character in the synthesized image to obtain a second image.

[0037] To improve image generation efficiency, images of each target character can be generated in parallel and then synthesized into a global image. Based on this, in one embodiment, the step of outputting a second image based on the first information and the first spatial constraint parameters of the one or more target characters includes: Based on the first information and the first spatial constraint parameters of the one or more target characters, a third image of each target character is generated; The second image is generated and output based on the third images of all target characters; or, the second image is generated and output based on the second spatial constraint parameters and the third images of all target characters, wherein the second spatial constraint parameters are determined based on the first spatial constraint parameters of multiple target characters and are used to constrain the spacing between target characters.

[0038] Here, the electronic device generates a third image of the target character based on the attributes or characteristics of the target character represented by the first information and the first spatial constraint parameters of the target character; for example, it generates a third image of the target character based on the type and / or attribute information of one of the target characters represented by the first information and the first spatial constraint parameters of the target character; and generates third images of all target characters in this way.

[0039] When the third images of all target characters are obtained, the electronic device can perform image fusion or synthesis on the third images of all target characters to obtain a second image; after performing image fusion or synthesis on the third images of all target characters, a background image can also be generated based on the fused or synthesized image to obtain a second image.

[0040] Having obtained third images of all target characters, the electronic device can determine second spatial constraint parameters based on first spatial constraint parameters of some or all target characters. For example, based on the first spatial constraint parameters of every two adjacent target characters, a second spatial constraint parameter is generated between these two target characters to constrain the distance between them. If the first information does not include the spatial relationship between target characters, then a second image is generated and output based on the determined second spatial constraint parameters and the third images of all target characters; if the first information includes the spatial relationship between target characters, then a second image is generated and output based on the determined second spatial constraint parameters, the spatial relationship between target characters, and the third images of all target characters.

[0041] The process of generating a second image based on the determined second spatial constraint parameters and the third images of all target characters includes: determining the spacing between target characters based on the determined second spatial constraint parameters; performing image fusion or synthesis based on the indirection between target characters and the third images of all target characters to obtain the second image; or performing image fusion or synthesis based on the indirection between target characters and the third images of all target characters to obtain a fourth image, and generating a background image based on the fourth image to obtain the second image.

[0042] Based on the determined second spatial constraint parameters, the spatial relationships between target characters, and the third images of all target characters, a second image is generated. This may include: determining the spacing between target characters based on the determined second spatial constraint parameters; performing image fusion or synthesis based on the spacing between target characters, the spatial relationships between target characters, and the third images of all target characters to obtain the second image; or, performing image fusion or synthesis based on the spacing between target characters, the spatial relationships between target characters, and the third images of all target characters to obtain a fifth image, and generating a background image based on the fifth image to obtain the second image.

[0043] To ensure consistency in style among the independently generated third images of various target characters—for example, to ensure consistency in lighting direction and / or texture and / or image style across multiple target character images—the same seed value can be used to generate the third images of each target character. Based on this, in one embodiment, generating the third image of each target character based on the first information and the first spatial constraint parameters of the one or more target characters includes: Determine the seed value, which indicates the initial noise starting point of the generated image; Using the seed value, a third image of the first target character is generated based on the attributes or features of the first target character represented by the first information and the first spatial constraint parameters of the first target character. The first target character is any one of the one or more target characters.

[0044] Here, a seed value can be randomly selected. The electronic device can use this seed value to generate a third image of the first target character based on the attributes or features of the first target character represented by the first information and the first spatial constraint parameters of the first target character. For example, using this seed value, a third image of the first target character can be generated based on the type and / or attribute information of the first target character and the first spatial constraint parameters of the first target character.

[0045] It should be noted that the electronic device can also generate a third image of the first target character based on the attributes or features of the first target character represented by the first information and the first spatial constraint parameters of the first target character; for other target characters, using the seed value of the third image of the first target character, a third image of the second target character can be generated based on the attributes or features of the second target character represented by the first information and the first spatial constraint parameters of the second target character, where the second target character is any target character other than the first target character. When the electronic device uses a pre-trained image generation model to generate the third image of a target character, the seed value of the image generation model can be fixed.

[0046] To improve image generation efficiency, multiple computing nodes can be invoked to generate images of multiple target characters. Based on this, in one embodiment, generating a third image for each target character based on the first information and the first spatial constraint parameters of the one or more target characters includes: By using multiple computing nodes that meet the first condition, a third image of each target role is generated based on the first information and the first spatial constraint parameters of the one or more target roles. One computing node is used to generate a third image of one target role.

[0047] Here, the electronic device acquires the computing resource status of multiple available computing nodes. This acquisition can be done in real-time or at time intervals. Based on the computing resource status of these nodes, multiple computing nodes that meet a first condition are selected. Using these nodes, a third image of each target role is generated based on first information and first spatial constraint parameters of one or more target roles. The number of computing nodes that meet the first condition can be greater than or equal to the number of target roles, allowing for node replacement in case of failure or image generation delay exceeding a first threshold. A single computing node generates a third image of a target role based on the attributes or features represented by the first information and the first spatial constraint parameters of that role. Multiple computing nodes generate third images of multiple target roles in parallel. The process of generating the third image is described above and will not be repeated here. Computing nodes include, but are not limited to, GPUs and / or edge devices. Of course, at least two computing nodes can also work together to generate a third image of the same target character. For example, one computing node generates a sketch of the target character, and another computing node processes the sketch of the target character to obtain a third image of the target character.

[0048] It should be noted that the computing resource status of a computing node includes, but is not limited to, one or more of the following: resource utilization or available resources, network latency, and costs (such as transmission costs and billing costs). Resource utilization includes, but is not limited to, the utilization rate of remaining video memory and / or the graphics processing unit (GPU). The first criterion can be used to select computing nodes with sufficient available resources and / or low latency and / or low transmission costs. The first criterion includes, but is not limited to, one or more of the following: remaining video memory is greater than or equal to the second threshold, GPU utilization is less than or equal to the third threshold, network latency is less than or equal to the fourth threshold, and billing costs are less than or equal to the fifth threshold. The first to sixth thresholds can be set according to actual needs and are not restricted here.

[0049] It should be noted that the electronic device can predict in real time the delay time of each computing node in generating the third image of the target character. If the delay time exceeds the first threshold (e.g., 500 milliseconds), the electronic device can call another backup computing node to generate the third image of the target character. If an interruption operation by user input is detected, the generation of the third image of the target character is stopped immediately, and the computing resources of the computing node are released.

[0050] To improve image generation efficiency, in one embodiment, generating a third image for each target character using multiple computing nodes that satisfy a first condition, based on the first information and the first spatial constraint parameters of the one or more target characters, includes: Based on the first information, multiple sub-tasks are determined; Based on the first information and the first spatial constraint parameters of the one or more target roles, the third information of each of the plurality of sub-tasks is determined. The third information of a sub-task includes the attribute or feature of a target role represented by the first information and the first spatial constraint parameters of the corresponding target role. The multiple computing nodes are invoked in parallel to execute the multiple sub-tasks based on the third information of the multiple sub-tasks, so as to obtain the third image of each target character; wherein, one computing node is used to execute the corresponding sub-task based on the third information of one sub-task to obtain the third image of the target character corresponding to the sub-task.

[0051] Here, the electronic device determines multiple sub-tasks based on the first information. The number of sub-tasks can be the same as the number of target roles, with one sub-task corresponding to one target role. Based on the attributes or features of one or more target roles represented by the first information, and the first spatial constraint parameters of one or more target roles, the third information of each sub-task is determined. The third information of a sub-task includes the attributes or features of a target role represented by the first information and the first spatial constraint parameters of the corresponding target role. Multiple computing nodes are invoked in parallel to execute multiple sub-tasks based on the third information of the multiple sub-tasks, thereby obtaining a third image of each target role. For example, the electronic device sends the third information of a sub-task to each of the multiple computing nodes, so that the computing nodes execute the corresponding sub-task based on the received third information, obtain the third image of the target role corresponding to the sub-task, and return the third image of the target role to the electronic device.

[0052] It should be noted that when computing power nodes include local edge devices (or local GPUs) and cloud computing power nodes (such as cloud GPUs), electronic devices can prioritize calling local edge devices to execute the first subtask to reduce latency. The first subtask can be understood as a low-load subtask, such as one with a load below the sixth threshold, or one that generates a sketch of the target character. Idle cloud computing power nodes can then be called to execute the second subtask, and data can be transmitted via a high-speed network using Remote Direct Memory Access (RDMA) to reduce data transmission latency. The second subtask can be understood as a high-load subtask, such as one with a load greater than or equal to the sixth threshold. The second subtask includes, but is not limited to, high-resolution image generation tasks and / or super-resolution tasks (image super-resolution reconstruction tasks), for example, processing the sketch of the target character into a high-resolution image to obtain a third image of the target character. For sudden traffic spikes (such as sudden image generation tasks or subtasks), electronic devices can dynamically adjust the number of cloud computing power nodes, such as triggering Kubernetes cluster scaling up or down, or scaling up or down GPU container groups (Pods). For example, if the prompt is "There are white cranes in the ancient-style landscape painting, and a waterfall in the distance, with 8K details", the electronic device can detect that the local edge node supports 1080P rendering. It can call the local edge node to generate sketches of the white cranes and the waterfall, and route the super-resolution task to the cloud A100 node. The super-resolution task is used to enlarge the sketches of the white cranes and the waterfall to 8K resolution and then add details. The electronic device monitors the sudden increase in cloud load in real time and automatically expands to two GPUs to assist in processing the background waterfall.

[0053] The following section provides a more detailed description of this application with reference to application examples.

[0054] Figure 2 A schematic diagram of the image generation method according to an embodiment of this application is provided, as follows: Figure 2 As shown, the image generation method includes the following steps: Step 0: Identify the first image loaded or uploaded by the user, and identify the drawing area in the first image to determine which area to draw in.

[0055] Here, users can define the painting area in the first image or identify the painting area through edge detection; the painting area is also called the drawing area, and the first image is also called the canvas.

[0056] For example, users can access the system through methods such as... Figure 3 A closed curve is drawn on the first image shown, and the electronic device returns the drawing area enclosed by the closed curve in real time, such as... Figure 4 As shown.

[0057] For example, electronic devices detect when a user is in a certain position. Figure 3 When a closed curve is manually drawn on the first image shown, a pre-trained image segmentation model is used to perform edge detection and refinement on the user-drawn coarse selection area. This automatically identifies object boundaries and generates a pixel-level mask for the coarse selection area, resulting in the drawing area. The output is as follows: Figure 5 The pixel-level mask for the coarsely selected region is shown. A large-scale image segmentation model could be Meta's Segment Anything Model (SAM).

[0058] Step 1: Obtain the prompt words entered by the user.

[0059] Here, after the user confirms the drawing area, the electronic device prompts the user to enter a prompt word and retrieves the entered prompt word. The prompt word describes one or more target characters to be drawn in the drawing area included in the first image. For example... Figure 6 As shown, the electronic device outputs a dialog box prompting the user to enter a prompt word. The user enters "A mouse wearing a red hat is standing next to a rabbit. The rabbit is all yellow and is holding a shovel in its left hand." The following uses this prompt word as an example to illustrate the process of generating an image from text.

[0060] Step 2: The electronic device responds to the prompt words input by the user, performs semantic analysis on the prompt words, and obtains the first information.

[0061] Here, the first information represents the attributes or characteristics of one or more target characters. In the scenario where the prompt is "a mouse wearing a red hat stands next to a rabbit, the rabbit is all yellow, and is holding a shovel in its left hand," the first information includes the type of the target character, attribute information, and spatial relationship between the target characters. Specifically, the target characters include a mouse and a rabbit; the mouse's attribute information includes that it wears a hat and that the hat is red; the rabbit's attribute information includes that it is yellow and that it is holding a shovel; and the spatial relationship between the mouse and the rabbit is that the mouse is standing next to the rabbit.

[0062] In one embodiment, semantic parsing of the prompt words is performed to obtain first information, including: Natural language processing techniques are used to perform semantic analysis on the prompt words, identify the various entities included in the prompt words, and obtain the type of each target role; Identify the modifiers of each entity from the prompts to obtain the attribute information of each target role; Based on the syntactic dependency relationships between entities in the prompt words, the spatial relationships between target roles are determined.

[0063] Here, natural language processing models can be used to analyze prompt words, identify the entities contained in the prompt words, such as mice and rabbits; and identify the modifiers of each target character to obtain the attribute information of each target character, such as the attribute information of a mouse being a red hat and the attribute information of a rabbit being a yellow body; it can also identify the syntactic dependency relationships between entities in the prompt words, and determine the spatial relationships between target characters based on the syntactic dependency relationships between entities, such as a mouse standing next to a rabbit.

[0064] Step 3: Generate structured instructions based on the first information.

[0065] Here, the initial information is converted into machine-executable structured instructions, such as JSON instructions.

[0066] Step 4: Interactive spatial planning.

[0067] Here, the electronic device guides the user to locate each target character on the first image and adjust its size, or prompts the user to select the position of the target character in the drawing area and adjust its size through adjustment functions. Specifically, it can guide the user to set the position of the target character on the first image and adjust its position and / or size through dragging, gesture scaling, rotation, and other operations. For any target character, in response to the setting operation, the electronic device acquires second information about the target character, determines its normalized coordinates based on the second information, and determines its first spatial constraint parameters based on the normalized coordinates. The second information includes the target character's position and / or size information in the drawing area, and the normalized coordinates include the target character's center coordinates and bounding box coordinates.

[0068] Specifically, the electronic device can output a visual drag-and-drop interface and prompt the user to set the position and / or size of the target character. The visual drag-and-drop interface can support the user to adjust the position and / or size of the target character through dragging, gesture scaling, rotation and other operations. The electronic device can output the position and size ratio of the target character in the drawing area in real time.

[0069] For example, the output of electronic devices such as Figure 7 The interface shown prompts the user to set the rabbit's position and size. After clicking "Settings" on the interface, the user begins to set the position and size of the target character. The user sets the rabbit's position and size as follows: Figure 8As shown. In response to the user's setting operation for the rabbit, the electronic device acquires the rabbit's second information, determines and records the rabbit's first spatial constraint parameters based on the rabbit's second information. The rabbit's second information includes the rabbit's position information and / or size information in the drawing area, and the rabbit's first spatial constraint parameters include the rabbit's center coordinates (e.g., [0.6, 0.5]) and bounding box coordinates (e.g., [0.4, 0.3, 0.8, 0.7]).

[0070] With the same settings, the electronic device prompts the user to complete the settings for another target character (e.g., a mouse) until the user completes the settings for all target characters. For example, in response to the user's setting operation for the mouse, the electronic device acquires the mouse's second information, determines and records the mouse's first spatial constraint parameters based on the mouse's second information, and the mouse's second information includes the mouse's position information and / or size information in the drawing area.

[0071] After the user completes the settings for all target characters, the electronic device can output a rendering of the target characters' positions and / or sizes on the first image, generated based on the second information of the target characters. Figure 9 The image shows the setup effect of a rabbit and a mouse. The electronic device can also calculate the first spatial constraint parameters for each target character, determine the spacing between adjacent target characters, and obtain the second spatial constraint parameters. For example, the second spatial constraint parameter is that the minimum spacing between the rabbit and the mouse is 10 pixels.

[0072] Step 5: Fix the seed value and generate the third image for each target character independently.

[0073] Here, electronic devices can dynamically plan computing power nodes; with a fixed seed value, multiple computing power nodes generate third images of each target role based on the type, attribute information, and first spatial constraint parameters of each target role, so as to ensure that the lighting direction, texture, and image style of the third images of each target role remain consistent.

[0074] Specifically, the electronic device can determine multiple sub-tasks based on first information; based on the first information and the first spatial constraint parameters of each target role, it can determine the third information of each sub-task, wherein the third information of a sub-task includes the type, attribute information and first spatial constraint parameters of a target role; based on the computing power resource status (such as load, network latency, cost, etc.) of some or all available computing power nodes, it can select multiple computing power nodes from the available computing power nodes, and call the selected multiple computing power nodes in parallel to execute multiple sub-tasks based on the third information of multiple sub-tasks to obtain the third image of each target role; one computing power node is used to execute the corresponding sub-task based on the third information of a sub-task to obtain the third image of the target role corresponding to the sub-task.

[0075] The specific implementation process is as follows: 1) Electronic devices can intelligently analyze prompts and, based on the semantic complexity and / or image style requirements of the prompts, select a large model that matches the semantic complexity and / or image style requirements. This allows computing nodes to call the selected large model to generate a third image of the target character. For example, large models supporting text-to-image representation include SDXL, ControlNet, and FLUX, with ControlNet ultimately being selected. The electronic device determines multiple sub-tasks based on primary information. Based on the primary information and the primary spatial constraint parameters of each target character, it determines the third information for each sub-task. The third information of a sub-task includes the type, attribute information, and primary spatial constraint parameters of a target character.

[0076] 2) Monitor the computing resource status of available computing nodes in real time, such as collecting GPU utilization, remaining video memory, network latency, billing costs, etc.; based on the computing resource status of available computing nodes, select multiple computing nodes that meet the first condition from the available computing nodes. The first condition is described in the relevant description above and will not be repeated here.

[0077] 3) Determine the dynamic allocation strategy for computing power nodes, and support the dynamic adjustment of computing power nodes or computing power resources according to the concurrent task volume to avoid computing power nodes being overloaded or idle; computing power nodes include local edge devices (or local GPUs) and cloud computing power nodes (such as cloud GPUs).

[0078] Specifically, for low-load scenarios (e.g., the load of a subtask is less than the sixth threshold, or the load of a local computing node is less than the seventh threshold): the local edge device of the electronic device is preferentially called to generate a sketch or third image of the target character corresponding to the first subtask based on the third information of the first subtask, in order to reduce latency, and the load of the first subtask is less than the sixth threshold.

[0079] For high-load scenarios (e.g., the load of a subtask is greater than or equal to the sixth threshold, or the load of a local computing node is greater than or equal to the seventh threshold): Idle cloud computing nodes (e.g., idle A100 nodes in the cloud) are invoked to execute the second subtask, and data is transmitted via a high-speed RDMA network to reduce data transmission latency; for example, the second subtask is to process the sketch of the target character into a high-resolution image to obtain the third image of the target character. Specifically, for sudden traffic spikes (such as sudden image generation tasks, subtasks, or concurrent subtasks), electronic devices can dynamically adjust the number of cloud computing nodes to avoid overload or idleness, such as triggering Kubernetes cluster scaling up or down, or scaling up or down GPU Pods.

[0080] For example, in a scenario where the prompt is "A mouse wearing a red hat stands next to a rabbit, the rabbit is all yellow, and holds a shovel in its left hand," since the subtasks for generating the mouse and rabbit are low-load scenarios, the electronic device can directly call the local edge node to generate the third images of the mouse and rabbit. In a scenario where the prompt is "A white crane appears in an ancient-style landscape painting, with a waterfall in the distance, 8K detail," the electronic device detects that the local edge node supports 1080P rendering. It can call the local edge node to generate sketches of the white crane and waterfall, and route the super-resolution task to the cloud A100 node. The super-resolution task is used to enlarge the sketches of the white crane and waterfall to 8K resolution and add details, resulting in the third images of the white crane and waterfall.

[0081] 4) The electronic device dynamically schedules multiple computing power nodes based on the third information of each subtask in the multiple subtasks, and sends the third information of the subtask to each computing power node in the multiple computing power nodes to distribute the subtasks.

[0082] For example, a subtask is distributed to the first computing node to generate a third image of a mouse; another subtask is distributed to the second computing node to generate a third image of a rabbit. Of course, if a background image needs to be generated, a subtask for generating the background image can also be distributed to the third computing node. If the electronic device detects that the delay of any computing node executing the corresponding subtask (generating a sketch of the corresponding target character) exceeds a first threshold (e.g., 500 milliseconds), the electronic device schedules the subtask to a backup computing node that meets the first condition, where the delay of the backup computing node executing the task is less than or equal to 500 milliseconds. If the electronic device detects a user input interruption, such as entering a new prompt, triggering a cancellation command, or interrupting image generation, the electronic device instructs the computing node to cancel the corresponding subtask to release the computing resources occupied by the subtask.

[0083] The computing nodes invoke a selected large model (e.g., ControlNet) to draw a sketch of the corresponding target character in the drawing area based on the third information from the subtask. The input to the large model includes the third information from the subtask, and may also include an image in the drawing area marked with the position and size of each target character, for example... Figure 9 The image shown; the output of the large model is a sketch or third image of the target character.

[0084] For example, by setting or fixing the seed value of ControlNet, the first computing node calls ControlNet based on third-party information about the mouse. Figure 9The right-hand circled area in the image shown is used to draw an image of a mouse, resulting in a third image of the mouse. This third image includes an image of a mouse wearing a red hat. The second computing node calls ControlNet based on the third information about the rabbit. Figure 9 The image shown generates a third image of the rabbit in the circled area on the left. This third image includes a yellow rabbit holding a shovel. In other words, different computing nodes can call ControlNet with the same seed value in parallel to draw the third image of the corresponding target character.

[0085] Alternatively, the first computing node can first call ControlNet to draw a third image of the mouse based on the mouse's third information, and the second computing node can refer to or use the seed value of ControlNet called by the first computing node to draw a third image of the rabbit based on the rabbit's third information.

[0086] It should be noted that, when generating the third image of the corresponding target character, each computing node can also use an image inpainting model (such as LaMa) to generate or fill the background image (such as grassland continuation) in the third image of the target character based on the pixel values ​​of the pixels surrounding the third image of the target character.

[0087] Step 6: Based on the third images of each target character, synthesize the global image to obtain the second image.

[0088] Here, the electronic device, upon obtaining the third images of each target character, synthesizes a global image based on the third images of all target characters, or generates and outputs the second image based on the second spatial constraint parameters and the third images of all target characters. The second spatial constraint parameters are determined based on the first spatial constraint parameters of multiple target characters and are used to constrain the spacing between the target characters.

[0089] In this process, if the first information includes the spatial relationship between the target characters, a global image is synthesized based on the spatial relationship between the target characters, the second spatial constraint parameters, and the third images of all target characters to obtain the second image. For example, if the third images of the mouse and the rabbit are obtained, the global image is synthesized based on the spatial relationship between the mouse and the rabbit (the mouse is standing next to the rabbit), the second spatial constraint parameters (such as the minimum distance between the mouse and the rabbit being 10 pixels), the third images of the mouse and the rabbit.

[0090] It should be noted that, in one embodiment, the computing node can also generate a third image of the target character based solely on the type and attribute information of the target character. Once the electronic device obtains the third images of all target characters, it can inject spatial constraints through GLIGEN and output a global image by combining the third images of all target characters. The spatial constraints include at least the first spatial constraint parameters for each target character, and may also include second spatial constraint parameters and / or the spatial relationships between target characters. GLIGEN-injected spatial constraints refer to the joint encoding of spatial location information (e.g., first spatial constraint parameters, second spatial constraint parameters, and spatial relationships between target characters) with corresponding text semantics (e.g., user-inputted text / prompt words, or first information obtained by semantic parsing of prompt words) in a pre-trained text-based image diffusion model (such as Stable Diffusion) through a "gated self-attention mechanism," and dynamically injecting this information into the U-Net cross-attention layer. This achieves precise control over the position and size of objects in the generated image without needing to fine-tune the backbone model.

[0091] After synthesizing the global image, the electronic device can directly output the global image (second image); alternatively, the electronic device can verify the global image based on the first information, such as verifying whether the global image includes each target object, whether the attributes of each target object are correct, and whether the spatial relationships between the target objects are accurate. If the global image passes verification, it is output to the user for confirmation of satisfaction. If the user is not satisfied, the process returns to step 5 to regenerate the image. If the user is satisfied, the final global image is output, and the process ends.

[0092] In this embodiment, first information can be obtained by semantically parsing the prompt words, and the first information can be converted into structured instructions. The visual human-computer interaction can be converted into spatial constraint parameters. The second image is generated in two stages by combining the structured instructions corresponding to the first information and the spatial constraint parameters. The first stage is to independently generate the third image of each target character. The second stage is to perform global coordination based on the third images of all target characters to synthesize a global image. This text-based image generation scheme breaks through the limitations of semantic and spatial control coupling in traditional end-to-end text-based images, solves the problems of spatial control ambiguity and semantic fragmentation in local redrawing, and achieves collaborative optimization of high-precision spatial layout and semantic consistency of multiple characters. In addition, based on the real-time collection of user settings for the position and / or size of target characters by the visual drag-and-drop interface, the normalized coordinates of each target character are calculated. Based on the normalized coordinates of each target character, the first spatial constraint parameters of each target character and the second spatial constraint parameters between target characters are determined. The first spatial constraint parameters and the second spatial constraint parameters are used to constrain the position of each target character in the drawing area and the relative position between target characters. This transforms human experience into precise spatial instructions that can be executed by the machine, and solves the problem of compositional imbalance caused by the ambiguity of the relative positions of objects in complex scenes. Compared with existing solutions, the embodiments of this application have the advantages shown in Table 1.

[0093] Table 1

[0094] To implement the image generation method of this application embodiment, this application embodiment also provides an image generation apparatus, which is installed on an electronic device, such as... Figure 11 As shown, the device includes: The parsing unit 1101, in response to a prompt word from a user for a first image, performs semantic parsing on the prompt word to obtain first information; wherein, the prompt word is used to describe one or more target characters that need to be drawn in the first image, and the first information characterizes the attributes or features of the one or more target characters; The acquisition unit 1102 is configured to acquire the first spatial constraint parameters of the one or more target roles in response to a setting operation for the one or more target roles; Image output unit 1103 is used to output a second image based on the first information and the first spatial constraint parameters of the one or more target characters, wherein the second image includes an image of the one or more target characters drawn in the first image.

[0095] In one embodiment, the image output unit 1103 is specifically used for: Based on the first information and the first spatial constraint parameters of the one or more target characters, a third image of each target character is generated; The second image is generated and output based on the third images of all target characters; or, the second image is generated and output based on the second spatial constraint parameters and the third images of all target characters, wherein the second spatial constraint parameters are determined based on the first spatial constraint parameters of multiple target characters and are used to constrain the spacing between target characters.

[0096] In one embodiment, the image output unit 1103 is specifically configured to determine a seed value, which indicates the initial noise starting point of the generated image; and to generate a third image of the first target character based on the attributes or features of the first target character characterized by the first information and the first spatial constraint parameters of the first target character, using the seed value, wherein the first target character is any one of the one or more target characters.

[0097] In one embodiment, the acquisition unit 1102 is specifically used for: In response to a setting operation for a first target character, second information of the first target character is obtained. The second information includes the position information and / or size information of the first target character in the first image. The first target character is any one of the one or more target characters. Based on the second information of the first target role, the normalized coordinates of the first target role are determined. The normalized coordinates include the center coordinates and bounding box coordinates of the target role. Based on the normalized coordinates of the first target role, the first spatial constraint parameters of the first target role are determined.

[0098] In one embodiment, the first information includes one or more of the type and attribute information of the one or more target roles and the spatial relationships between the target roles; The parsing unit 1101 is specifically used for one or more of the following: Natural language processing techniques are used to perform semantic analysis on the prompt words, identify one or more entities included in the prompt words, and obtain the type of the one or more target roles; The entity's modifiers are identified from the prompts to obtain the target character's attribute information; Based on the syntactic dependency relationships between entities in the prompt words, the spatial relationships between target characters are determined.

[0099] In one embodiment, the image output unit 1103 is specifically used to generate a third image of each target character based on the first information and the first spatial constraint parameters of the one or more target characters by multiple computing power nodes that meet the first condition, wherein one computing power node is used to generate a third image of one target character.

[0100] In one embodiment, the image output unit 1103 is specifically used for: Based on the first information, multiple sub-tasks are determined; Based on the first information and the first spatial constraint parameters of the one or more target roles, the third information of each of the plurality of sub-tasks is determined. The third information of a sub-task includes the attribute or feature of a target role represented by the first information and the first spatial constraint parameters of the corresponding target role. The multiple computing nodes are invoked in parallel to execute the multiple sub-tasks based on the third information of the multiple sub-tasks, so as to obtain the third image of each target character; wherein, one computing node is used to execute the corresponding sub-task based on the third information of one sub-task to obtain the third image of the target character corresponding to the sub-task.

[0101] In practical applications, the parsing unit 1101, the acquisition unit 1102, and the image output unit 1103 can be implemented by the processor in the image generation device in conjunction with the communication interface.

[0102] It should be noted that the image generation apparatus provided in the above embodiments is only illustrated by the division of the above-described program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the apparatus can be divided into different program modules to complete all or part of the processing described above. Furthermore, the image generation apparatus and image generation method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0103] Based on the hardware implementation of the above program modules, and in order to implement the image generation method of this application embodiment, this application embodiment also provides an electronic device, such as... Figure 12 As shown, the electronic device 1200 includes: The communication interface 1201 enables information exchange with other network nodes; The processor 1202 is connected to the communication interface 1201 to enable information interaction with other network nodes and, when running a computer program, executes the methods provided by one or more technical solutions on the first node side. The computer program is stored in the memory 1203.

[0104] Specifically, the processor 1202 is used for: In response to a user's prompt for the first image, the prompt is semantically parsed to obtain first information; wherein the prompt describes one or more target characters that need to be drawn in the first image, and the first information characterizes the attributes or features of the one or more target characters; In response to a setting operation for the one or more target roles, the first spatial constraint parameters of the one or more target roles are obtained; Based on the first information and the first spatial constraint parameters of the one or more target characters, a second image is output, which includes images of the one or more target characters drawn in the first image.

[0105] In one embodiment, the processor 1202 is specifically used for: Based on the first information and the first spatial constraint parameters of the one or more target characters, a third image of each target character is generated; The second image is generated and output based on the third images of all target characters; or, the second image is generated and output based on the second spatial constraint parameters and the third images of all target characters, wherein the second spatial constraint parameters are determined based on the first spatial constraint parameters of multiple target characters and are used to constrain the spacing between target characters.

[0106] In one embodiment, the processor 1202 is specifically configured to determine a seed value, the seed value being used to indicate the initial noise starting point of the generated image; and to use the seed value to generate a third image of the first target character based on the attributes or features of the first target character characterized by the first information and the first spatial constraint parameters of the first target character, wherein the first target character is any one of the one or more target characters.

[0107] In one embodiment, the processor 1202 is specifically used for: In response to a setting operation for a first target character, second information of the first target character is obtained. The second information includes the position information and / or size information of the first target character in the first image. The first target character is any one of the one or more target characters. Based on the second information of the first target role, the normalized coordinates of the first target role are determined. The normalized coordinates include the center coordinates and bounding box coordinates of the target role. Based on the normalized coordinates of the first target role, the first spatial constraint parameters of the first target role are determined.

[0108] In one embodiment, the first information includes one or more of the type and attribute information of the one or more target roles and the spatial relationships between the target roles; the processor 1202 is specifically used for one or more of the following: Natural language processing techniques are used to perform semantic analysis on the prompt words, identify one or more entities included in the prompt words, and obtain the type of the one or more target roles; The entity's modifiers are identified from the prompts to obtain the target character's attribute information; Based on the syntactic dependency relationships between entities in the prompt words, the spatial relationships between target characters are determined.

[0109] In one embodiment, the processor 1202 is specifically used to generate a third image of each target character based on the first information and the first spatial constraint parameters of the one or more target characters by multiple computing power nodes that meet the first condition, wherein one computing power node is used to generate a third image of one target character.

[0110] In one embodiment, the processor 1202 is specifically used for: Based on the first information, multiple sub-tasks are determined; Based on the first information and the first spatial constraint parameters of the one or more target roles, the third information of each of the plurality of sub-tasks is determined. The third information of a sub-task includes the attribute or feature of a target role represented by the first information and the first spatial constraint parameters of the corresponding target role. The multiple computing nodes are invoked in parallel to execute the multiple sub-tasks based on the third information of the multiple sub-tasks, so as to obtain the third image of each target character; wherein, one computing node is used to execute the corresponding sub-task based on the third information of one sub-task to obtain the third image of the target character corresponding to the sub-task.

[0111] It should be noted that the specific processing procedures of processor 1202 and communication interface 1201 can be understood by referring to the above method.

[0112] Of course, in practical applications, the various components in electronic device 1200 are coupled together through bus system 1204. It can be understood that bus system 1204 is used to realize the connection and communication between these components. In addition to a data bus, bus system 1204 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 12 The general designated all buses as Bus System 1204.

[0113] The memory 1203 in this embodiment is used to store various types of data to support the operation of the electronic device 1200. Examples of such data include any computer program used to operate on the electronic device 1200.

[0114] The methods disclosed in the embodiments of this application can be applied to, or implemented by, the processor 1202. The processor 1202 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware in the processor 1202 or by instructions in software form. The processor 1202 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 1202 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, specifically in memory 1203. The processor 1202 reads information from memory 1203 and, in conjunction with its hardware, completes the steps of the aforementioned method.

[0115] In an exemplary embodiment, the electronic device 1200 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0116] It is understood that the memories (memory 1203 and second memory 1003) in the embodiments of this application can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0117] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 1203 storing a computer program or computer-executable instructions. The computer program or computer-executable instructions can be executed by the processor 1202 of the electronic device 1200 to complete the steps described in the aforementioned first node-side method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM; or it can be various devices including one or any combination of the above-mentioned memories.

[0118] By way of example, this application also provides a computer program product, including a computer program or computer executable instructions, which are stored in a storage medium. The computer program or computer executable instructions can be read from the storage medium and executed by the processor 1202 of the electronic device 1200 to complete the steps described in the aforementioned first node-side method.

[0119] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0120] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0121] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0122] It should be noted that terms such as "first" and "second" are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. "Multiple" can refer to two or more items, and "multiple" can refer to two or more items. The term "and / or" in this document merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, the term "one or more" in this document refers to any combination of at least two of the multiple elements. For example, including one or more of A, B, and C can represent including any one or at least two or more elements selected from the set consisting of A, B, and C.

[0123] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0124] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An image generation method, characterized in that, The method includes: In response to a user's prompt for the first image, the prompt is semantically parsed to obtain first information; wherein the prompt describes one or more target characters that need to be drawn in the first image, and the first information characterizes the attributes or features of the one or more target characters; In response to a setting operation for the one or more target roles, the first spatial constraint parameters of the one or more target roles are obtained; Based on the first information and the first spatial constraint parameters of the one or more target characters, a second image is output, which includes images of the one or more target characters drawn in the first image.

2. The method according to claim 1, characterized in that, The step of outputting a second image based on the first information and the first spatial constraint parameters of the one or more target roles includes: Based on the first information and the first spatial constraint parameters of the one or more target characters, a third image of each target character is generated; The second image is generated and output based on the third images of all target characters; or, the second image is generated and output based on the second spatial constraint parameters and the third images of all target characters, wherein the second spatial constraint parameters are determined based on the first spatial constraint parameters of multiple target characters and are used to constrain the spacing between target characters.

3. The method according to claim 2, characterized in that, The step of generating a third image for each target character based on the first information and the first spatial constraint parameters of the one or more target characters includes: Determine the seed value, which indicates the initial noise starting point of the generated image; Using the seed value, a third image of the first target character is generated based on the attributes or features of the first target character represented by the first information and the first spatial constraint parameters of the first target character. The first target character is any one of the one or more target characters.

4. The method according to any one of claims 1 to 3, characterized in that, The step of obtaining the first spatial constraint parameters of the one or more target roles in response to the setting operation for the one or more target roles includes: In response to a setting operation for a first target character, second information of the first target character is obtained. The second information includes the position information and / or size information of the first target character in the first image. The first target character is any one of the one or more target characters. Based on the second information of the first target role, the normalized coordinates of the first target role are determined. The normalized coordinates include the center coordinates and bounding box coordinates of the target role. Based on the normalized coordinates of the first target role, the first spatial constraint parameters of the first target role are determined.

5. The method according to any one of claims 1 to 3, characterized in that, The first information includes one or more of the following: type and attribute information of the one or more target roles, and spatial relationships between the target roles; The semantic parsing of the prompt words to obtain first information includes one or more of the following: Natural language processing techniques are used to perform semantic analysis on the prompt words, identify one or more entities included in the prompt words, and obtain the type of the one or more target roles; The entity's modifiers are identified from the prompts to obtain the target character's attribute information; Based on the syntactic dependency relationships between entities in the prompt words, the spatial relationships between target characters are determined.

6. The method according to claim 2 or 3, characterized in that, The step of generating a third image for each target character based on the first information and the first spatial constraint parameters of the one or more target characters includes: By using multiple computing nodes that meet the first condition, a third image of each target role is generated based on the first information and the first spatial constraint parameters of the one or more target roles. One computing node is used to generate a third image of one target role.

7. The method according to claim 6, characterized in that, The step of generating a third image for each target role by using multiple computing nodes that meet the first condition, based on the first information and the first spatial constraint parameters of the one or more target roles, includes: Based on the first information, multiple sub-tasks are determined; Based on the first information and the first spatial constraint parameters of the one or more target roles, the third information of each of the plurality of sub-tasks is determined. The third information of a sub-task includes the attribute or feature of a target role represented by the first information and the first spatial constraint parameters of the corresponding target role. The multiple computing nodes are invoked in parallel to execute the multiple sub-tasks based on the third information of the multiple sub-tasks, so as to obtain the third image of each target character; wherein, one computing node is used to execute the corresponding sub-task based on the third information of one sub-task to obtain the third image of the target character corresponding to the sub-task.

8. An electronic device, characterized in that, The method includes a processor and a memory for storing a computer program or computer-executable instructions capable of running on the processor, wherein, when the processor is used to run the computer program or computer-executable instructions, it performs the steps of the method according to any one of claims 1 to 7.

9. A storage medium having a computer program or computer-executable instructions stored thereon, characterized in that, When the computer program or computer-executable instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program or computer-executable instructions, characterized in that, The computer program or computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

Images