A picture generation method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a picture generation method, device, computer equipment and storage medium, wherein the method comprises: obtaining a target text to be generated into a shot picture; generating description information of each entity object in the target text under multiple description dimensions according to the target text; performing splitting processing on the target text to obtain multiple text segments, and generating picture element information under multiple feature dimensions for each text segment; for each text segment, performing fusion processing on the picture element information of the text segment under the multiple feature dimensions and the description information of each entity object under the multiple description dimensions to obtain shot picture description information corresponding to the text segment; and generating multiple shot pictures corresponding to the target text based on the shot picture description information corresponding to the multiple text segments respectively.

Description

Technical Field

[0001] This disclosure relates to the field of natural language processing technology, and more specifically, to an image generation method, apparatus, electronic device, and storage system. Background Technology

[0002] With the rise and development of short videos, their types are gradually diversifying. Among them, short comic videos created using existing written works as creative material are an important branch. However, the general production process of comic videos relies on professional film and television editors and illustrators, especially in areas such as character design and storyboard design, which require personnel with strong professional skills.

[0003] The character design and storyboard design in the aforementioned video creation process rely heavily on human inspiration and the artist's skill, resulting in low production efficiency. Furthermore, while some tools exist that can automatically generate images from text to assist in designing and generating visuals, their application in video production suffers from inconsistent rendering of the same video elements across different frames due to the randomness of the image generation algorithm. For example, the same character might appear drastically different in different frames, leading to unsatisfactory video results. Summary of the Invention

[0004] This disclosure provides at least one image generation method, apparatus, electronic device, and storage medium.

[0005] In a first aspect, embodiments of this disclosure provide a method for generating an image, the method comprising:

[0006] The process involves: acquiring the target text for generating the shot; generating description information for each entity object in the target text across multiple description dimensions; splitting the target text into multiple text fragments, and generating shot element information across multiple feature dimensions for each text fragment; the feature dimensions include entity object dimensions, and the shot element information under the entity object dimension includes entity object identifiers; fusing the shot element information of each text fragment across the multiple feature dimensions with the description information of each entity object across the multiple description dimensions to obtain shot description information corresponding to the text fragment; and generating multiple shot images corresponding to the target text based on the shot description information corresponding to the multiple text fragments.

[0007] In one possible implementation, generating descriptive information for each entity object in the target text under at least one descriptive dimension, based on the target text, includes:

[0008] The target text and the description dimension information for each entity object type in at least one entity object type are input into a trained neural network model to generate description information for each entity object in the target text under multiple description dimensions. The neural network model is used to determine each entity object involved in the target text, and for each entity object, according to the description dimension information corresponding to the entity object type to which the entity object belongs, to generate description information for the entity object under at least one description dimension.

[0009] In one possible implementation, the process of splitting the target text to obtain multiple text fragments includes:

[0010] The target text, along with attribute limitation information for the split text fragments, is input into a trained neural network model to obtain the multiple text fragments. The neural network model is used to split the input target text into text fragments corresponding to different camera shots, wherein the different camera shots have at least one different camera element.

[0011] In one possible implementation, generating image element information across multiple feature dimensions for each text fragment includes:

[0012] The text fragments and the dimensional information of the multiple feature dimensions are input into the neural network model to obtain the image element information under the multiple feature dimensions corresponding to each text fragment; the neural network model is used to generate the image element information based on each text fragment and the dimensional information of the multiple feature dimensions.

[0013] In one possible implementation, the process of fusing the image element information of the text fragment under the multiple feature dimensions with the description information of each entity object under the multiple description dimensions includes:

[0014] Based on the entity object identifier indicated by the screen element information under the entity object dimension, the description information of the target entity object corresponding to the entity object identifier under multiple description dimensions is determined from the description information of each entity object under multiple description dimensions.

[0015] The description information of the identified target entity object under multiple description dimensions is merged with the image element information to obtain the lens image description information.

[0016] In one possible implementation, after generating multiple camera shots corresponding to the target text, the method further includes:

[0017] Based on the shot description information corresponding to each text segment, generate the audio data corresponding to that text segment;

[0018] The video is generated based on the camera footage and audio data corresponding to each of the text segments, and the video includes the camera footage and audio data corresponding to each of the multiple text segments.

[0019] In one possible implementation, generating the video based on the camera shot and audio data corresponding to each text segment includes:

[0020] The video is generated based on the camera shot corresponding to each text segment, the audio data, and the text content of the text segment, wherein the text content is used as subtitle information in the video.

[0021] In one possible implementation, generating the video based on the camera shot corresponding to each text segment, the audio data, and the text content of the text segment includes:

[0022] The number of video frames corresponding to each shot is determined based on the number of video frames of the video to be generated, the number of segments of the multiple text segments, and the number of characters corresponding to each text segment;

[0023] The video is generated based on the shot and its corresponding video frame count, the audio data, and the text content of the text segment.

[0024] In one possible implementation, acquiring the target text of the shot to be generated includes:

[0025] The system receives raw text input by a user, as well as rewriting instructions for the raw text; the rewriting instructions include at least one type of modification direction information.

[0026] The original text and the rewrite prompt information are input into a neural network model to obtain the rewritten target text.

[0027] In one possible implementation, after generating the shot image description information, the method further includes:

[0028] In response to receiving modification instructions from the user, updated model assistance information is generated based on the modification instructions.

[0029] The model auxiliary information is input into the neural network model to obtain the updated shot description information.

[0030] Secondly, embodiments of this disclosure also provide an image generation apparatus, comprising:

[0031] The acquisition module is used to acquire the target text of the shot to be generated;

[0032] The first generation module is used to generate description information of each entity object in the target text under multiple description dimensions based on the target text.

[0033] The second generation module is used to split the target text to obtain multiple text fragments, and for each text fragment, generate screen element information under multiple feature dimensions; the feature dimensions include the entity object dimension, and the screen element information under the entity object dimension includes the entity object identifier;

[0034] The fusion processing module is used to fuse the image element information of the text fragment under multiple feature dimensions with the description information of each entity object under multiple description dimensions for each text fragment, so as to obtain the shot image description information corresponding to the text fragment;

[0035] The third generation module is used to generate multiple shots corresponding to the target text based on the shot description information corresponding to the multiple text fragments.

[0036] Thirdly, embodiments of this disclosure also provide a computer device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the first aspect above, or any possible implementation of the first aspect, are performed.

[0037] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the first aspect or any possible implementation of the first aspect.

[0038] For a description of the effects of the above-mentioned image generation apparatus, electronic device, and computer-readable storage medium, please refer to the description of the above-mentioned image generation method; it will not be repeated here.

[0039] As described above, the target text for generating the desired shot can be obtained, and based on the target text, descriptive information for each entity object in the target text can be generated under at least one descriptive dimension. Next, the target text is split into multiple text fragments, and for each text fragment, shot element information under multiple feature dimensions is generated. These feature dimensions include the entity object dimension, and the shot element information under the entity object dimension includes the entity object identifier. Then, for each text fragment, the shot element information under multiple feature dimensions is fused with the descriptive information of each entity object under multiple descriptive dimensions to obtain the shot description information corresponding to each text fragment. Finally, based on the shot description information corresponding to each text fragment, multiple shots corresponding to the target text are generated.

[0040] In the image generation method of this disclosure, for each entity object associated with the target text, corresponding description information is generated for each entity object. Subsequently, based on the description information of the entity objects, images of the entity objects in each shot are generated. In this way, the same entity object located in different shots has consistent description information, thus ensuring the consistency of the same entity object in different shots. Then, the image element information of each text fragment of the target text is fused with the description information of the entity objects involved in the text fragment. The fused shot description information has both a feature description of the entire shot and a specific description of the entity objects displayed therein. Therefore, image generation tools can be used to automatically generate shot images that conform to the features indicated by the shot description information, improving video production efficiency while ensuring the consistency of the same entity object in different shots.

[0041] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0043] Figure 1 A flowchart of an image generation method provided by an embodiment of this disclosure is shown;

[0044] Figure 2 This illustration shows a flowchart of obtaining entity object feature description information in the image generation method provided in this embodiment of the present disclosure;

[0045] Figure 3 This illustration shows a flowchart of the image generation method provided in this embodiment, which involves splitting target text and obtaining shot description information.

[0046] Figure 4 This illustration shows a flowchart of the image generation method provided in this embodiment, which generates video after generating a shot image.

[0047] Figure 5a This illustration shows a schematic diagram of a generated shot image in the image generation method provided in this embodiment of the present disclosure;

[0048] Figure 5b This illustration shows a schematic diagram of another shot generated in the image generation method provided in this embodiment of the present disclosure;

[0049] Figure 5c This illustration shows a schematic diagram of another shot generated in the image generation method provided in this embodiment of the present disclosure;

[0050] Figure 5d This illustration shows a schematic diagram of another shot generated in the image generation method provided in this embodiment of the present disclosure;

[0051] Figure 6 A schematic diagram of an image generation apparatus provided in an embodiment of this disclosure is shown;

[0052] Figure 7 A schematic diagram of a computer device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0054] Research has found that with the development of the video industry, different video types have gradually emerged to meet user needs. Among them, short comic videos made from existing written works are an important branch. This type of video allows users to quickly understand the highlights of a novel or other entertainment content in their fragmented time, thereby attracting users to actively learn about the full picture of the work.

[0055] This video production method relies heavily on the design inspiration of professionals and the skill level of illustrators. However, professional illustrators have relatively mature painting styles, which limits the diversity of the shots in the produced videos and makes the production efficiency of multi-frame shots in the videos low.

[0056] In addition, there are some tools on the market that can automatically generate images based on text content. However, when applied to video production, due to the randomness of the algorithm that generates the image each time, the same video elements may appear inconsistently in different shots. For example, the same person may look too different before and after, resulting in the video production result not meeting expectations.

[0057] Based on the above research, this disclosure provides a method for image generation. As described above, target text for the shot to be generated can be obtained, and descriptive information for each entity object in the target text can be generated in at least one descriptive dimension. Thus, when generating the entity object image for each shot based on the entity object's descriptive information, the same entity object located in different shots ensures consistency across different shots of the entire video because its descriptive information is consistent.

[0058] Furthermore, by fusing the image element information of each text fragment of the target text with the description information of the entity objects involved in the text fragment, the resulting image description information has both a feature description of the entire image and a specific description of the entity objects displayed within it. This allows image generation tools to automatically generate images that conform to the features indicated by the image description information, ensuring the consistency of the same entity objects in different images while improving video production efficiency.

[0059] Furthermore, in different stages of generating shot description information based on target text, such as generating entity object (e.g., character) description information, text splitting, and generating scene elements (scenes), this embodiment adds model-aided information to the neural network model, in addition to the original material information. Moreover, after obtaining the target text for the shot to be generated, before entering the generation stages mentioned above, this embodiment provides a mechanism for automatic text rewriting. The user can indicate the direction of modification, and the neural network model can automatically provide the rewritten target text. This provides a better creative foundation for subsequent generation stages, more easily meets user needs, and reduces the complexity of subsequent modifications.

[0060] The shortcomings of the above solutions are the result of the inventor's practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below should be considered as the inventor's contribution to this disclosure.

[0061] The technical solutions of this disclosure will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely to illustrate selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0062] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0063] To facilitate understanding of this embodiment, a detailed description of the image generation method disclosed in this disclosure is provided first. The execution entity of the image generation method provided in this disclosure is generally a computer device with certain computing capabilities. This computer device may include, for example, a terminal device, a server, or other processing devices. The terminal device may be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the image generation method can be implemented by a processor calling computer-readable instructions stored in memory.

[0064] The image generation method provided in this disclosure embodiment will be described below using a terminal device as an example.

[0065] Example 1

[0066] See Figure 1 The diagram shows a flowchart of an image generation method provided in Embodiment 1 of this disclosure. The method includes steps S101 to S105, wherein:

[0067] S101: Obtain the target text of the shot to be generated.

[0068] It should be noted that the target text can include text information from any authorized source. For example, in this embodiment, a corresponding comic or video can be generated using an excerpt from an authorized novel. Alternatively, the target text can include published multimedia content, unpublished text content provided by the author, or legitimate text content provided by other material processing users. Furthermore, the target text can include key information such as scenes, characters, and plots to provide sufficient context. If the target text lacks sufficient key information or plot content, it can be rewritten, and the rewritten target text can be used to execute subsequent processes, as detailed in the following description. Additionally, the target text can be text information in any language.

[0069] As one implementation method, an administrator or other relevant personnel can insert identification tags into the obtained raw text to obtain the target text; for example, the target text after insertion could be "In the evening in the city park," <red> Xiaoming and his friends are preparing to hold a concert.< / red> ",in, <red> and< / red> Labels are inserted identifiers that identify key content parts in a segment, so that the neural network model can highlight the key points when generating relevant descriptions.

[0070] In this embodiment of the disclosure, after obtaining the target text of the shot to be generated, before entering the subsequent generation stages, a mechanism for automatic text rewriting can be provided to the user. The user can indicate the direction of modification, and the neural network model can automatically provide the rewritten target text, thereby providing a better creative foundation for the subsequent generation stages, making it easier to meet user needs, and reducing the complexity of subsequent modifications.

[0071] Here, the rewriting instruction information can include at least one modification direction information. By inputting the original text and the rewriting prompt information into a neural network model, the rewritten target text can be obtained. At least one modification direction can indicate the dimension to be improved, such as the plot dimension, character dimension, or scene dimension. For example, regarding the plot dimension, after inputting the original text, the user can input rewriting prompt information such as "enrich the plot." Similarly, regarding the character dimension, the user can input prompts such as "increase the number of characters, or create a connection between character A and character B." And regarding the scene dimension, the user can input prompts such as "enrich the scene description of a certain segment."

[0072] The target text may include multiple text paragraphs, where one or more consecutive text paragraphs may correspond to the same shot. Based on this, the target text can be divided into multiple text segments, as described in S103 below.

[0073] S102: Based on the target text, generate description information for each entity object in the target text under multiple description dimensions.

[0074] In this step, descriptive information for each entity object in the target text can be generated across multiple descriptive dimensions. Specifically, existing descriptive information for each entity object can be extracted from the target text. If the descriptive information for some entity objects is incomplete, such as lacking descriptive information for certain descriptive dimensions, the descriptive information for these descriptive dimensions can be automatically completed based on the contextual semantics.

[0075] The entities here can include any scene objects such as characters and props. Generally, the same entity will appear in multiple shots. Figure 2This is a flowchart illustrating the process by which a neural network model generates character feature descriptions based on target text. In practice, artificial intelligence technology can be used to automatically generate the descriptive information of the aforementioned entity objects. Specifically, the target text, along with descriptive dimension information for each entity object type (at least one), can be input into a trained neural network model to generate descriptive information for each entity object in the target text across multiple descriptive dimensions. The neural network model is used to determine each entity object involved in the target text and, for each entity object, generate descriptive information for that entity object across at least one descriptive dimension, according to the descriptive dimension information corresponding to the entity object type to which that entity object belongs.

[0076] In this embodiment, the multiple descriptive dimensions for each entity object can be pre-defined in the input neural network model's requirement information (model auxiliary information). Taking characters as an example, the input neural network model's model auxiliary information could specify, "Based on the target text, extract the main body of each character, and summarize each character's gender, age, hairstyle, hair color, eye color, face shape, and clothing according to the text; if the original text does not provide corresponding descriptions, supplement them automatically based on the contextual logic in the target text." Here, "gender, age, hairstyle, hair color, eye color, face shape, and clothing" are the pre-defined multiple descriptive dimensions. It should be noted that the multiple descriptive dimensions given here are only examples, and the actual application should be based on actual needs. In addition to the indicated descriptive dimensions, the neural network model can also automatically supplement and improve the descriptive dimensions of relevant characters according to the needs of the plot, such as supplementing the character's personality description according to the plot requirements.

[0077] The neural network model here is pre-trained and can automatically generate descriptive information that meets the user's input requirements and the contextual semantics of the target text. In other words, the neural network model takes the target text, including the aforementioned descriptive dimensions, and various model-aided information as input, and automatically generates descriptive information for each entity object under the relevant entity object type. Entity object types include, for example, character types, prop types, animal types, etc.

[0078] Here, the model-aided information input by the user into the neural network model may include not only the aforementioned descriptive dimension information, but also other information that the neural network model needs to pay attention to when generating the model, such as precautions for each descriptive dimension, special explanations of the characteristics of each entity object, etc. For example, the requirements information input into the neural network model may also specify further requirements such as "the description of clothing should include descriptions of tops, pants, and shoes respectively, and specify the corresponding types and colors".

[0079] That is, when the target text is input into the pre-trained neural network model, the instruction information that instructs the neural network model to generate entity object description information, as well as the above-mentioned description dimension information, other requirements, etc., can also be input into the neural network model. The entity objects in the novel excerpt can be extracted through the neural network model. Here, taking the entity object as a character as an example, the neural network model performs multi-dimensional feature descriptions for each character based on the target text and the various input requirements.

[0080] For example, regarding the above-mentioned "in the evening in the city park," <red> Xiaoming and his friends are preparing to hold a concert.< / red> The neural network model can provide descriptive information about entity objects across multiple descriptive dimensions, such as:

[0081] "Xiaoming: Male, 17 years old, short black hair, hair sticking up in all directions, black eyes, round face, blue jacket, black pants, black sneakers."

[0082] The above describes the feature description of "Xiaoming" generated by the neural network model based on the target text and in accordance with instructions. This ensures consistency for the same person in different shots, as their descriptive information remains consistent, thus avoiding the problem of excessive differences in the same person across different shots.

[0083] S103: The target text is split into multiple text fragments, and for each text fragment, multiple feature dimensions of screen element information are generated; the feature dimensions include the entity object dimension, and the screen element information under the entity object dimension includes the entity object identifier.

[0084] In this step, the neural network model can be instructed to split the target text into multiple text fragments, and for each text fragment, generate image element information under multiple feature dimensions. Specifically, the target text can be split into text fragments corresponding to different camera shots. In practice, the target text contains... <red> and< / red> The marked content refers to the most exciting parts of the novel. <red> Starting with a tag< / red> When segmenting content ending with a tag, the original text can be preserved without further splitting. Then, based on the segmentation results, the scene element information for each text segment is extracted across multiple feature dimensions. If scene element information for certain feature dimensions is missing, it can be supplemented based on the contextual semantics.

[0085] The visual elements here can include, but are not limited to, any visual information such as character actions, background, and perspective. In addition, there must be at least one different visual element between the different shots mentioned above, such as changes in the characters, background, and character actions.

[0086] In addition to the target text, model-aided information for text segmentation can also be input into the neural network model. This model-aided information can be attribute-limiting information for the segmented text fragments. The neural network model segments the target text into text fragments that can generate corresponding shot images in the future according to these attribute-limiting information, and ensures that there is at least one different shot element between different shot images.

[0087] For example, the attribute limitation information here may include information indicating the character size range of the text fragment, information indicating that each text fragment cannot contain only dialogue content, and information indicating the above. <red>The corresponding content retains the original instructions and information, etc.

[0088] Furthermore, when segmenting the target text, the model can be instructed to insert a corresponding sequence number before each text segment in the model assistance information input to the neural network model, making the segmentation results more intuitive for the user.

[0089] In this embodiment of the disclosure, the user can input target text into the trained neural network model. Furthermore, the user can also input multiple feature dimensions into the model auxiliary information input to the neural network model, so that the neural network model can provide corresponding scene element information based on the multiple feature dimensions input by the user. For example, the user can specify in the requirements information input to the neural network model that "based on each segment result, imagine each segment result as a scene and generate the corresponding scene element information, including the perspective, the characters appearing, the subject of the image, the characters' emotions and actions, and the background of the image." It should be noted that the requirements information given here are only examples, and in actual applications, the actual needs should prevail. Figure 3 The diagram illustrates the process by which a neural network model splits target text into multiple text segments and generates scene element information for different shots based on the corresponding shot images for each text segment. In practice, the segmented text segments, along with dimensional information from multiple feature dimensions, can be input into the neural network model to automatically generate scene element information corresponding to different text segments using artificial intelligence technology.

[0090] The neural network model here is pre-trained and capable of generating image element information that meets user input requirements and corresponds to different shot images for each text segment. Users can use each text segment as input data and input multiple feature dimensions of the shot image corresponding to each text segment, so that the neural network model can generate image element information for each text segment's corresponding shot image according to these feature dimensions. Image elements can include character actions, character emotions, perspective, etc.

[0091] In addition to the aforementioned feature dimension information, user input to the neural network model can also include other information that the neural network model needs to pay attention to during generation. For example, considerations for each feature dimension and different requirements for generating image element information for different shot images. For instance, users can also specify further requirements in the input information to the neural network model, such as "if no person appears in the image, then the person's emotions and actions do not need to be output." It should be noted that the requirements given here are only examples, and the actual needs should prevail in practical applications.

[0092] Taking the aforementioned target text as an example, when the target text is input into a pre-trained neural network model, the neural network model can be instructed to segment the target text. Alternatively, by inputting the aforementioned multiple feature dimensions and various requirements, the neural network model can be instructed to generate image element information corresponding to different shot frames for each text segment obtained after segmentation.

[0093] For example, regarding the above-mentioned "in the evening in the city park," <red> Xiaoming and his friends are preparing to hold a concert.< / red> After completing the segmentation process, the neural network model can provide image element information across multiple feature dimensions for the text fragment, which may include:

[0094] Angle of view: 55mm

[0095] Character: Xiaoming

[0096] Mood: (Happy 1.4)

[0097] Action: Talking with friends

[0098] Background: A city park

[0099] It should be noted that in actual implementation, the image element information of the generated shot corresponds to each text segment. That is, if the target text is split into n text segments, there are n sets of image element information for the corresponding shot.

[0100] S104: For each of the text segments, the image element information of the text segment under the multiple feature dimensions is fused with the description information of each entity object under the multiple description dimensions to obtain the shot image description information corresponding to the text segment.

[0101] In this step, the image element information of different shots corresponding to each text fragment under multiple feature dimensions can be fused with the description information of each entity object under multiple description dimensions to obtain the shot description information corresponding to each text fragment. That is, by combining the description information of each entity object under multiple description dimensions with the image element information corresponding to the text fragment in which the entity object appears, we can obtain the entity object description for the shot and the scene description other than the entity object, that is, obtain the complete description for the shot.

[0102] Furthermore, after generating the shot description information, if the user believes that the currently generated shot description information has room for optimization, the shot description can be modified. Specifically, the modification instruction information for the shot description information can be input into the neural network model, instructing the neural network model to generate new shot description information that meets the user's expectations.

[0103] The modification instructions here can include requirements for plot attributes, such as making the plot richer or the ending more comedic; requirements for characters, such as changing the relationships between multiple characters; and requirements for scene backgrounds, such as changing the era of the scene background or adding more scenery to the scene.

[0104] In practical implementation, artificial intelligence technology can be used to automatically fuse the image element information under multiple feature dimensions and the descriptive information under multiple descriptive dimensions, thereby obtaining the image description information corresponding to each text fragment. Specifically, based on the entity object identifier indicated by the image element information under the entity object dimension, the descriptive information of the target entity object corresponding to the entity object identifier under multiple descriptive dimensions can be determined from the descriptive information of each entity object under multiple descriptive dimensions; and the determined descriptive information of the target entity object under multiple descriptive dimensions is merged with the image element information to obtain the shot image description information.

[0105] Here, based on the entity object identifier indicated by the image element information, the target entity object that should appear in the corresponding shot can be determined. Then, the description information of the target entity object can be found from the description information of each previously generated entity object. The two are then merged and processed to obtain shot description information that includes both entity object description and scene description.

[0106] Similarly, the neural network model here is pre-trained and can automatically generate scene description information that meets the user's input requirements. Users can use the aforementioned scene element information and character description information as input, along with their specific requirements, so that the neural network model can fuse these information and output the fused scene description information. It's important to note that the scene description information here corresponds one-to-one with the multiple text fragments mentioned above. That is, if the target text is split into n fragments, there are n sets of scene element information corresponding to each fragment. After fusing the character description information with the scene element information, n sets of scene description information corresponding one-to-one with each text fragment are generated.

[0107] In this embodiment of the disclosure, when the aforementioned image element information and character description information are input into a pre-trained neural network model, instruction information instructing the neural network model to fuse the image element information and character description information, as well as requirements for the fused result, can also be input into the neural network model. The neural network model then generates image description information corresponding to each text segment. The requirements for the fused result may include, but are not limited to, further requirements regarding the content and output format of the image description information.

[0108] For example, the information that a user requires to input into a neural network model could be:

[0109] "Based on the description information of each entity object under multiple description dimensions and the image element information under multiple feature dimensions, organize the shot description information in the format of serial number, number of people, background of the image, subject of the image, complete content of the character feature description, character's emotion, and character's action"; a further requirement could be "keep the character feature content intact and do not modify it"; it should be noted that the requirements given here are only examples and should be based on actual needs in actual applications.

[0110] For example, regarding the above-mentioned "in the evening in the city park," <red> Xiaoming and his friends are preparing to hold a concert.< / red> The corresponding image description information output by the neural network model is as follows:

[0111] "Single, in a city park, chatting with a friend, male, 17 years old, short black hair, hair sticking up in all directions, black eyes, round face, blue jacket, black pants, black sneakers, happy, chatting with a friend."

[0112] It should be noted that the above only provides an example of the image description information generated by the neural network model by combining the image element information and character description information corresponding to the text fragments. However, in actual implementation, the generated image description information corresponds to each text fragment. That is, if the target text is split into n text fragments, there will be n sets of corresponding image description information.

[0113] S105: Based on the scene description information corresponding to the multiple text fragments, generate multiple scene images corresponding to the target text.

[0114] In this step, a shot corresponding to the target text can be generated based on the shot description information.

[0115] In practice, multiple sets of shot description information generated by the neural network model can be input into another neural network model or image generation tool. This neural network model or image generation tool can generate corresponding shot images based on the shot description information input by the user. The user can adjust the corresponding parameters in the neural network model to generate multiple shot images that meet the user's expectations.

[0116] like Figure 4 The diagram illustrates the process of generating a video after generating a shot image. In this embodiment, after generating multiple shot images corresponding to the target text, a video can be generated using these multiple shot images; the generated video needs to include shot image data, audio data, and subtitle information. In specific implementations, a neural network model can be used to automatically generate shot images based on the aforementioned shot description information, and audio data corresponding to each text segment can be generated based on the shot description information corresponding to that text segment. The text content of the text segment is then used as subtitle information; subsequently, the obtained audio data, shot images, and subtitle information are used to generate the corresponding video.

[0117] It should be noted that the aforementioned shot images correspond one-to-one with multiple text fragments. In the final generated video, to ensure video continuity, one shot corresponds to multiple shot images. In the final video, each shot image corresponds to each video frame. These multiple shot images can be understood as multiple video frames, meaning that each shot in the video corresponds to multiple video frames.

[0118] The number of video frames corresponding to a single shot can be determined based on the number of images in the shot (i.e., the total number of video frames), the number of multiple text segments, and the number of characters in each text segment. For text segments with a larger number of characters, more subtitles are generated, requiring more video frames to display them fully. Furthermore, in practical implementations, subtitles do not need to be displayed entirely within a single video frame. The subtitle information can be split based on the text content and the corresponding shot image, displaying different but continuous subtitles in different video frames.

[0119] Once the camera angles and their corresponding video frame rates are determined, the final video can be obtained based on the audio data and the text content of the text segments. The audio data is used for the video's narration, and the text content of the text segments serves as the video's subtitle information.

[0120] like Figure 5a and Figure 5b As shown, Figure 5a and Figure 5b For the same entity object in the same scene across different shots, because the descriptive information for the same entity object differs across shots, the only difference between the same entity object in the same scene across different shots is its movement; for example... Figure 5c and Figure 5d As shown, Figure 5c and Figure 5d For the same entity object in different scenes across different shots, since the characteristic description information of the same entity object is consistent, the differences in the final generated shots will only be based on descriptions such as action and attire. Figure 5a , Figure 5b , Figure 5c and Figure 5d As can be seen, since the feature description information of each entity object was obtained in the early stage, the features of the same entity object are consistent before and after. The only difference is the action and clothing of the same entity object in different shots. There is no problem of the same entity object having too large a difference in different shots, which would cause the video to not meet expectations.

[0121] The steps described above—generating video footage, audio data, and subtitle information, and then generating video based on these elements—can be accomplished using a neural network model. This neural network model is pre-trained and can automatically perform operations that meet user requirements based on user input data and commands.

[0122] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0123] Based on the same inventive concept, this disclosure also provides an image generation device corresponding to the image generation method. Since the principle of the device in this disclosure for solving the problem is similar to that of the image generation method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0124] Example 2

[0125] Reference Figure 6 The diagram shown is a schematic representation of an image generation device according to Embodiment 2 of this disclosure. The device includes: an acquisition module 601, a first generation module 602, a second generation module 603, a fusion processing module 604, and a third generation module 605; wherein,

[0126] The acquisition module 601 is used to acquire the target text of the shot image to be generated;

[0127] The first generation module 602 is used to generate description information of each entity object in the target text under multiple description dimensions based on the target text;

[0128] The second generation module 603 is used to split the target text to obtain multiple text fragments, and for each text fragment, generate screen element information under multiple feature dimensions; the feature dimensions include the entity object dimension, and the screen element information under the entity object dimension includes the entity object identifier.

[0129] The fusion module 604 is used to fuse the image element information of the text fragment under multiple feature dimensions with the description information of each entity object under multiple description dimensions for each text fragment, so as to obtain the shot image description information corresponding to the text fragment;

[0130] The third generation module 605 is used to generate multiple shots corresponding to the target text based on the shot description information corresponding to the multiple text fragments respectively.

[0131] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0132] Corresponding to Figure 1 In addition to the image generation method, this disclosure also provides a computer device 700, such as... Figure 7 The diagram shown is a structural schematic of a computer device 700 provided in this embodiment of the disclosure, including:

[0133] The computer device 700 comprises a processor 701, a memory 702, and a bus 703. The memory 702 stores execution instructions and includes main memory 721 and external memory 722. Main memory 721, also known as internal memory, temporarily stores computational data in the processor 701, as well as data exchanged with external memory such as a hard disk. The processor 701 exchanges data with external memory 722 through main memory 721. When the computer device 700 is running, the processor 701 communicates with the memory 702 via the bus 703, causing the processor 701 to execute the following instructions:

[0134] Obtain the target text of the shot to be generated;

[0135] Based on the target text, generate descriptive information for each entity object in the target text under multiple descriptive dimensions;

[0136] The target text is split into multiple text fragments, and for each text fragment, multiple feature dimensions of image element information are generated; the feature dimensions include the entity object dimension, and the image element information under the entity object dimension includes the entity object identifier;

[0137] For each text segment, the image element information of the text segment under the multiple feature dimensions is fused with the description information of each entity object under the multiple description dimensions to obtain the shot description information corresponding to the text segment.

[0138] Based on the scene description information corresponding to the multiple text fragments, multiple scene images corresponding to the target text are generated.

[0139] This disclosure also includes a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the image generation method described in the above-described method embodiments. The storage medium can be either volatile or non-volatile computer-readable storage.

[0140] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the image generation method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0141] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0142] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or modules may be electrical, mechanical, or other forms.

[0143] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0144] In addition, the functional modules in the various embodiments of this disclosure can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0145] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0146] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.< / red>

Claims

1. A method for generating an image, characterized in that, include: Obtain the target text of the shot to be generated; Based on the target text, generate descriptive information for each entity object in the target text under multiple descriptive dimensions; The target text is split into multiple text fragments, and for each text fragment, multiple feature dimensions of image element information are generated; the feature dimensions include the entity object dimension, and the image element information under the entity object dimension includes the entity object identifier; For each text segment, the image element information of the text segment under the multiple feature dimensions is fused with the description information of the target entity object to obtain the shot image description information corresponding to the text segment. The description information of the target entity object is determined from the description information of each entity object under the multiple description dimensions. The target entity object is the entity object corresponding to the entity object identifier. Based on the scene description information corresponding to the multiple text fragments, multiple scene images corresponding to the target text are generated.

2. The method according to claim 1, characterized in that, Based on the target text, generate descriptive information for each entity object in the target text under at least one descriptive dimension, including: The target text and the description dimension information for each entity object type in at least one entity object type are input into a trained neural network model to generate description information for each entity object in the target text under multiple description dimensions. The neural network model is used to determine each entity object involved in the target text, and for each entity object, according to the description dimension information corresponding to the entity object type to which the entity object belongs, to generate description information for the entity object under at least one description dimension.

3. The method according to claim 1, characterized in that, The target text is split into multiple text fragments, including: The target text, along with attribute limitation information for the split text fragments, is input into a trained neural network model to obtain the multiple text fragments. The neural network model is used to split the input target text into text fragments corresponding to different camera shots, wherein the different camera shots have at least one different camera element.

4. The method according to claim 1, characterized in that, For each of the text fragments, generate image element information under multiple feature dimensions, including: The text fragments and the dimensional information of the multiple feature dimensions are input into a neural network model to obtain the image element information under the multiple feature dimensions corresponding to each text fragment; the neural network model is used to generate the image element information based on each text fragment and the dimensional information of the multiple feature dimensions.

5. The method according to claim 1, characterized in that, The text fragments are fused with the image element information under the multiple feature dimensions and the description information of the target entity object, including: The description information of the identified target entity object under multiple description dimensions is merged with the image element information to obtain the lens image description information.

6. The method according to claim 1, characterized in that, After generating multiple camera shots corresponding to the target text, the process further includes: Based on the shot description information corresponding to each text segment, generate the audio data corresponding to that text segment; A video is generated based on the camera footage and audio data corresponding to each of the text segments, and the video includes the camera footage and audio data corresponding to each of the multiple text segments.

7. The method according to claim 6, characterized in that, Based on the camera shot and audio data corresponding to each text segment, the video is generated, including: The video is generated based on the camera shot corresponding to each text segment, the audio data, and the text content of the text segment, wherein the text content is used as subtitle information in the video.

8. The method according to claim 7, characterized in that, The video is generated based on the camera shot, audio data, and text content corresponding to each text segment, including: The number of video frames corresponding to each shot is determined based on the number of video frames of the video to be generated, the number of segments of the multiple text segments, and the number of characters corresponding to each text segment; The video is generated based on the shot and its corresponding video frame count, the audio data, and the text content of the text segment.

9. The method according to claim 1, characterized in that, Obtain the target text of the shot to be generated, including: The system receives raw text input by a user, as well as rewriting instructions for the raw text; the rewriting instructions include at least one type of modification direction information. The original text and the rewrite prompt information are input into a neural network model to obtain the rewritten target text.

10. The method according to claim 1, characterized in that, After generating the shot description information, it also includes: In response to receiving modification instructions from the user, updated model assistance information is generated based on the modification instructions. The model auxiliary information is input into the neural network model to obtain the updated shot description information.

11. An image generation apparatus, characterized in that, include: The acquisition module is used to acquire the target text of the shot to be generated; The first generation module is used to generate description information of each entity object in the target text under multiple description dimensions based on the target text. The second generation module is used to split the target text to obtain multiple text fragments, and for each text fragment, generate screen element information under multiple feature dimensions; the feature dimensions include the entity object dimension, and the screen element information under the entity object dimension includes the entity object identifier; The fusion processing module is used to fuse the image element information of the text fragment under the multiple feature dimensions with the description information of the target entity object for each text fragment to obtain the shot image description information corresponding to the text fragment. The description information of the target entity object is determined from the description information of each entity object under the multiple description dimensions. The target entity object is the entity object corresponding to the entity object identifier. The third generation module is used to generate multiple shots corresponding to the target text based on the shot description information corresponding to the multiple text fragments.

12. A computer device, characterized in that, include: The computer device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the image generation method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the image generation method as described in any one of claims 1 to 10.

Images