A method and related device for generating text and picture

Abstract

The embodiments of the present application disclose a method and related device for generating text illustrations, which can be applied to scenarios such as cloud technology, artificial intelligence, smart transportation, network media, assisted driving, and the Internet of Things. Virtual scene data and virtual character data for a text segment are obtained, and a target storyboard with a high similarity to the virtual scene data is obtained from multiple storyboards, so that the virtual scene described by the virtual scene data can use the camera movement method of the target storyboard, and its popularity can also be greater than a preset welcome threshold. A fixed role model corresponding to the virtual character described by the virtual character data is obtained, and the fixed role model is used to control the character features of the same virtual character in different virtual scenes to be relatively close. Thereby, the fixed role model is fused with the image generation model to obtain an illustration generation model, and based on the virtual character data and the target storyboard, a text illustration for the text segment is generated by the illustration generation model, thereby improving the generation effect of the text illustration.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for generating text and graphics and related devices. Background Art

[0002] In order to attract people's interest in reading, text content gradually appears in the form of pictures and texts. Taking novels as an example, novels can be presented through picture articles, novel illustrations, or novel comic short videos.

[0003] In related technologies, when the text content includes a lot of content, the text content is generally divided into multiple text segments and input into the image generation model respectively. The image generation model generates text illustrations corresponding to each text segment by understanding the semantics of each text segment.

[0004] However, the method of generating text illustrations for multiple text segments of the same text content respectively does not produce good text illustrations, and the user's reading experience is poor. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a text-to-picture generation method and related devices, which are used to improve the effect of generating text-to-picture and improve the user's reading experience.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] On the one hand, an embodiment of the present application provides a method for generating a text image, the method comprising:

[0008] Acquire virtual scene data and virtual character data for the text segment, wherein the virtual scene data is used to describe scene features of the virtual scene corresponding to the text segment, and the virtual character data is used to describe character features of the virtual character included in the virtual scene;

[0009] According to the virtual scene data, a target storyboard is obtained from a plurality of storyboards, the popularity of the virtual scene described by the storyboard is greater than a preset popularity threshold, and the similarity between the virtual scene described by the target storyboard and the virtual scene described by the virtual scene data is greater than a first similarity threshold;

[0010] According to the virtual character data, a fixed role model corresponding to the virtual character included in the virtual character data is obtained, where different virtual characters correspond to different fixed role models, and the fixed role model is used to control the difference of the same virtual character in different virtual scenes to be less than a preset difference threshold;

[0011] The character fixed model and the image generation model are integrated to obtain a picture generation model;

[0012] According to the virtual character data and the target storyboard, a text illustration for the text segment is generated by the illustration generation model.

[0013] On the other hand, an embodiment of the present application provides a text and picture generation device, the device comprising: an acquisition unit, a selection unit, a fusion unit and a generation unit;

[0014] The acquisition unit is used to acquire virtual scene data and virtual character data for the text segment, wherein the virtual scene data is used to describe scene features of the virtual scene corresponding to the text segment, and the virtual character data is used to describe character features of the virtual character included in the virtual scene;

[0015] The selection unit is used to obtain a target storyboard from multiple storyboards according to the virtual scene data, the popularity of the virtual scene described by the storyboard is greater than a preset popularity threshold, and the similarity between the virtual scene described by the target storyboard and the virtual scene described by the virtual scene data is greater than a first similarity threshold;

[0016] The selection unit is further used to obtain, based on the virtual character data, a fixed role model corresponding to the virtual character included in the virtual character data, wherein different virtual characters correspond to different fixed role models, and the fixed role model is used to control the difference of the same virtual character in different virtual scenes to be less than a preset difference threshold;

[0017] The fusion unit is used to fuse the character fixed model and the image generation model to obtain a picture generation model;

[0018] The generating unit is used to generate a text illustration for the text segment through the illustration generation model according to the virtual character data and the target storyboard.

[0019] On the other hand, an embodiment of the present application provides a computer device, the computer device comprising a processor and a memory:

[0020] The memory is used to store a computer program and transmit the computer program to the processor;

[0021] The processor is configured to execute the method described in the above aspects according to the instructions in the computer program.

[0022] On the other hand, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method described in the above aspects.

[0023] On the other hand, an embodiment of the present application provides a computer program product including a computer program, which, when executed on a computer device, enables the computer device to execute the method described in the above aspects.

[0024] It can be seen from the above technical solution that virtual scene data and virtual character data for a text segment are obtained, the virtual scene data is used to describe the scene features of the virtual scene corresponding to the text segment, and the virtual character data is used to describe the character features of the virtual character included in the virtual scene. A target storyboard with a high similarity to the virtual scene data is obtained from multiple storyboards, so that the virtual scene described by the virtual scene data can use the camera movement method of the target storyboard, and its popularity can also be greater than a preset welcome threshold. Different virtual characters correspond to different role fixed models, and the role fixed model corresponding to the virtual character described by the virtual character data is obtained. The role fixed model is used to control the difference of the same virtual character in different virtual scenes to be less than the preset difference threshold, that is, to control the character features of the same virtual character in different virtual scenes to be closer. Thereby, the role fixed model is fused with the image generation model to obtain a picture generation model, and a text picture for the text segment is generated by the picture generation model based on the virtual character data and the target storyboard.

[0025] Thus, in different text illustrations corresponding to different text segments, the character features of the same virtual character in different virtual scenes are consistent, which improves the accuracy of text illustrations. Moreover, through the target storyboards applicable to the virtual scenes, the text illustrations have the storyboard function, which increases the diversity of text illustrations. At the same time, the popularity of the target storyboard is greater than the preset welcome threshold, and the text illustrations obtained based on the target storyboard are more likely to be popular. This improves the generation effect of the text illustrations and improves the user's reading experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A schematic diagram of an application scenario of a method for generating text with pictures provided in an embodiment of the present application;

[0028] Figure 2 A schematic diagram of a flow chart of a method for generating text and image matching provided in an embodiment of the present application;

[0029] Figure 3A schematic diagram of determining virtual scene data and virtual character data of a text segment provided in an embodiment of the present application;

[0030] Figure 4 A schematic diagram of determining a target storyboard provided in an embodiment of the present application;

[0031] Figure 5 A schematic diagram of a role fixed model obtained by training provided in an embodiment of the present application;

[0032] Figure 6 A schematic diagram of a text-to-picture system provided in an embodiment of the present application;

[0033] Figure 7 A schematic diagram of a scenario of a method for generating text and pictures provided in an embodiment of the present application;

[0034] Figure 8 A schematic diagram of the structure of a text and image generation device provided in an embodiment of the present application;

[0035] Fig. 9 A schematic diagram of the structure of a server provided in an embodiment of the present application;

[0036] Fig.10 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The embodiments of the present application are described below in conjunction with the accompanying drawings.

[0038] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] In the related art, since the text content includes multiple text segments, different text segments correspond to different virtual scenes, each virtual scene includes one or more virtual characters, and the same virtual character may appear repeatedly in different virtual scenes, the method of using the image generation model to generate text images for multiple text segments of the same text content may cause the following two problems:

[0040] (1) Although the scene data are different, the generation logic of the image generation model remains unchanged, which may lead to the same composition of the text illustrations corresponding to different scene data, such as all distant views, all close views, etc. That is, the image generation model does not have a storyboard function, resulting in a single style of text illustrations and serious homogeneity.

[0041] (2) The description of the same virtual character in different virtual scenes may be different, so the same virtual character may generate different virtual character images in different virtual scenes. In other words, the same virtual character in the text content cannot be effectively identified, resulting in a low degree of matching between the text fragments and the text illustrations, which affects the reader's reading experience and fails to attract more readers.

[0042] That is to say, the method of using an image generation model to generate text illustrations for multiple text segments of the same text content has a poor effect of the generated text illustrations and a poor reading experience for users.

[0043] Based on this, the embodiment of the present application provides a method and a related device for generating text illustrations. For a text segment for which a text illustration is to be generated, the text segment corresponds to a virtual scene, and the virtual scene includes one or more virtual characters. A target storyboard suitable for the virtual scene is obtained from a plurality of storyboards with high popularity, and then a character fixed model suitable for the virtual character included in the virtual scene is obtained, and the character fixed model and the image generation model are merged to obtain an illustration generation model. According to the virtual character data and the target storyboard, a text illustration for the text segment is generated by the illustration generation model.

[0044] Therefore, in different text illustrations corresponding to different text segments, the character features of the same virtual character in different virtual scenes are relatively different, thereby improving the accuracy of text illustrations. Moreover, by using the target storyboards suitable for virtual scenes, the text illustrations have a storyboard function, which increases the diversity of text illustrations. At the same time, if the popularity of the target storyboard is greater than the preset welcome threshold, the text illustrations obtained based on the target storyboard are more likely to be popular. This improves the generation effect of the text illustrations and improves the user's reading experience.

[0045] The text and image generation method provided in this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, network media, assisted driving, and the Internet of Things. It should be noted that the following application scenarios are only examples, and the text and image generation method provided in this embodiment can also be applied to other scenarios, which are not limited here.

[0046] For example, taking the cloud technology scenario as an example, a text and image generation platform is built based on cloud technology. After the user uploads text content or specifies text content on the generation platform, the text and image generation platform splits it into multiple text segments, and then obtains the virtual scene data and virtual character data of the text segment for each text segment, obtains the target storyboard according to the virtual scene data, obtains the character fixed model according to the virtual character data, and fuses the character fixed model with the image generation model to obtain a picture generation model, thereby generating a text picture for the text segment based on the virtual character data and the target storyboard through the picture generation model, thereby generating text pictures for each text segment included in the text content, and obtaining multiple text pictures corresponding to the text content.

[0047] The text and picture generation method provided in this application can be applied to computer devices with text and picture generation capabilities, such as terminal devices and servers. Among them, the terminal device can specifically be a desktop computer, a laptop computer, a smart phone, a tablet computer, an Internet of Things device and a portable wearable device. The Internet of Things device can be a smart speaker, a smart TV, a smart air conditioner, a smart car device, etc. The smart car device can be a car navigation terminal and a car computer, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc., but it is not limited to this; the server can be an independent physical server, or it can be a server cluster or distributed system composed of multiple physical servers, or it can be a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. Terminal devices and servers can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0048] To facilitate understanding of the text-with-picture generation method provided in the embodiment of the present application, the following uses a server as an example in which the execution subject of the text-with-picture generation method is an example to exemplarily introduce the application scenario of the text-with-picture generation method.

[0049] See also Figure 1 , which is a schematic diagram of an application scenario of a text-with-picture generation method provided in an embodiment of the present application. Figure 1As shown, the application scenario includes a terminal device 110 and a server 120, and the terminal device 110 and the server 120 can communicate through a communication network. The communication network uses standard communication technology and / or protocols, usually the Internet, but can also be any network, including but not limited to Bluetooth, local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, dedicated network or any combination of virtual private networks. In some embodiments, customized or dedicated data communication technology can be used to replace or supplement the above data communication technology.

[0050] In actual applications, a client for generating text and pictures is installed in the terminal device 110 , and a user can upload a text segment for which a text and picture are to be generated through the terminal device 110 , and the terminal device 110 sends the text segment to the server 120 .

[0051] The server 120 is a server corresponding to the client. After acquiring the text segment, it acquires virtual scene data for describing the virtual scene corresponding to the text segment and virtual character data for describing the character characteristics of the virtual character included in the virtual scene. It is understandable that the virtual scene may include one or more virtual characters.

[0052] Based on the virtual scene data, the server 120 obtains a target storyboard whose similarity to the virtual scene data is greater than a first similarity threshold from multiple storyboards whose popularity is greater than a preset popularity threshold. The target storyboard is not only suitable for the virtual scene, but also has a relatively high popularity.

[0053] Different virtual characters correspond to different fixed character models. The server 120 obtains the fixed character model corresponding to the virtual character described by the virtual character data. The fixed character model can control the difference of the same virtual character in different virtual scenes to be less than a preset difference threshold, that is, to ensure that the difference of the same virtual character in different virtual scenes is small. After the fixed character model and the image generation model are merged, the obtained picture generation model also has the characteristics of the fixed character model and the image generation model. Then, according to the virtual character data and the target storyboard, the picture generation model generates a text picture for the text segment.

[0054] Thus, in different text illustrations corresponding to different text segments, the character features of the same virtual character in different virtual scenes are consistent, thereby improving the accuracy of text illustrations. Moreover, by using target storyboards suitable for virtual scenes, the text illustrations have a storyboard function, increasing the diversity of text illustrations. At the same time, if the popularity of the target storyboard is greater than the preset welcome threshold, the text illustration obtained based on the target storyboard is more likely to be popular. This improves the generation effect of the text illustrations and improves the user's reading experience.

[0055] The text-with-picture generation method provided in the embodiment of the present application can be executed by the server. However, in other embodiments of the present application, the terminal device can also have similar functions as the server, thereby executing the text-with-picture generation method provided in the embodiment of the present application, or the terminal device and the server can jointly execute the text-with-picture generation method provided in the embodiment of the present application, and this embodiment does not limit this.

[0056] The following is a detailed introduction to a text-to-picture generation method provided by the present application through a method embodiment.

[0057] See also Figure 2 , which is a flow chart of the text-with-picture generation method provided in the embodiment of the present application. For the convenience of description, the following embodiment is still introduced by taking the execution subject of the text-with-picture generation method as a server as an example. Figure 2 As shown, the text-to-picture generation method includes S201-S205.

[0058] S201: Acquire virtual scene data and virtual character data for a text segment.

[0059] A text fragment is a fragment used to describe a virtual scene, such as "A crowd gathered at the entrance of a supermarket, the sky was gloomy, and the rain was pouring. Water dripped on the buildings and ripples appeared in the accumulated water on the ground. People held umbrellas or covered their heads, some hid under the porch, some stood on the roadside, and some waited at the intersection. The trees on the roadside were wet by the rain, the water droplets on the leaves were crystal clear, the vehicles on the road splashed water, and the light from the street lamps was blurred in the rain."

[0060] It should be noted that the text fragment can be obtained based on the text content, and the text content is the content expressed by text, such as novels, long articles, etc. The text content describes multiple virtual scenes by text, and each text fragment corresponds to a virtual scene. As a possible implementation method, the text content can be obtained first, and then the text content can be split to obtain multiple text fragments, so as to generate a text image for each text fragment, and then obtain multiple text images corresponding to the text content.

[0061] The virtual scene data is used to describe the scene features of the virtual scene corresponding to the text segment. The scene features are used to describe and distinguish the attributes or characteristics of the virtual scene, such as weather, time, environment, buildings, etc. Continuing with the aforementioned text segment as an example, the virtual scene data can be "scene: supermarket entrance", "season: summer", "time: daytime", "weather: raining", etc.

[0062] Each virtual scene may include one or more virtual characters, which are characters described by text content, such as people, animals, etc. The character features of virtual characters are used to describe and distinguish the attributes or characteristics of virtual characters, such as hairstyle, facial features, body shape, etc. Virtual character data is used to describe the character features of virtual characters included in the virtual scene, that is, the virtual character data not only identifies the character features of the virtual characters, but also identifies the association between the virtual scene and the virtual characters.

[0063] As a possible implementation, the acquired virtual character data may be used to describe the character features of the main virtual character included in the virtual scene, where the main virtual character is a virtual character that appears in the text content more than a preset threshold number of times. Since users generally only pay attention to the main virtual character when viewing text and pictures, and do not pay much attention to virtual characters other than the main virtual character, when generating text and pictures, only the main virtual character may be paid attention to, thereby reducing the amount of text to be processed and increasing the speed and difficulty of generating text and pictures.

[0064] S202: Acquire a target storyboard from a plurality of storyboards according to the virtual scene data.

[0065] A storyboard is an image used to describe a storyboard. A storyboard or storyboard script, also known as a storyboard, refers to various image media such as movies, animations, TV series, advertisements, music videos, etc. Before actual shooting or drawing, the composition of the image is described in the form of storyboards. The continuous images are broken down into units of one shot, and the shot movement method, duration, dialogue, special effects, etc. are marked. In other words, a storyboard has a storyboard function, so it can describe the shot movement method of the corresponding virtual scene, such as long shot, close shot, etc.

[0066] In an embodiment of the present application, the popularity of the virtual scene described by the storyboard is greater than a preset popularity threshold. For example, multiple high-burning storyboard pictures are obtained from an open source website, and each high-burning storyboard picture has a high popularity. Thus, each high-burning storyboard picture is used as a storyboard to obtain multiple storyboards, or a storyboard library including multiple storyboards is constructed.

[0067] According to the scene features of the virtual scene described by the virtual scene data, a target storyboard is obtained from multiple storyboards, so that the similarity between the virtual scene described by the target storyboard and the virtual scene described by the virtual scene data is greater than a first similarity threshold, that is, the virtual scene described by the target storyboard and the virtual scene described by the virtual scene data have a high similarity. Therefore, not only can the virtual scene described by the virtual scene data be represented by the target storyboard, the camera movement method of the target storyboard can be applicable to the virtual scene described by the virtual scene data, but also the popularity of the target storyboard is relatively high, and the popularity of the text illustrations subsequently generated based on the target storyboard will also be relatively high.

[0068] The embodiment of the present application does not specifically limit the size of the first similarity threshold, and those skilled in the art may set it according to actual needs.

[0069] S203: According to the virtual character data, a fixed role model corresponding to the virtual character included in the virtual character data is obtained.

[0070] Different virtual character data is used to describe different virtual characters. Different virtual characters are suitable for different character fixing models. The character fixing model is used to control the difference of the same virtual character in different virtual scenes to be less than the preset difference threshold, so that the character fixing model corresponding to each virtual character can fix its corresponding virtual character, that is, the difference of the character characteristics of the corresponding virtual character in different virtual scenes is small, so as to effectively identify each virtual character, such as making one or more combinations of the facial features, hairstyle, figure, and clothing of the virtual character consistent. To avoid the possibility of differences in the description of the same virtual character in different virtual scenes, the matching degree between the text fragment and the text illustration is low, thereby improving the reading experience of the reader.

[0071] The embodiments of the present application do not specifically limit the preset difference threshold, and those skilled in the art may set it according to actual needs. As a possible implementation method, the preset difference threshold may not be 0, but a smaller value, such as 20%, so that the virtual characters in different virtual scenes are slightly different by adjusting the preset difference threshold, rather than being exactly the same, such as virtual character A wearing white clothes in virtual scene B, and virtual character A wearing blue clothes in virtual scene B, etc. This makes the text illustrations generated subsequently more diverse and improves the reader's reading experience.

[0072] S204: Fusing the character fixed model and the image generation model to obtain an image generation model.

[0073] The image generation model is used to generate images based on input data, but the image generation model is not global, so it generates corresponding text images for each text segment, and may not be able to ensure that the character features of the same virtual character in different virtual scenes are different.

[0074] Based on this, the embodiment of the present application integrates the role fixing model used to fix the virtual character with the image generation model to obtain the picture generation model, so that the picture generation model not only has the function of the image generation model, but also has the function of the role fixing model, thereby fixing the virtual character.

[0075] The present application embodiment does not specifically limit the image generation model, and those skilled in the art can set it according to actual needs. For example, a stable diffusion model can be used as an image generation model.

[0076] The diffusion model aims to eliminate the continuous application of Gaussian noise to the training images through a series of denoising autoencoders. The stable diffusion model is a variant of the diffusion model and is called the latent diffusion model (LDM). The stable diffusion model consists of three main components: variational autoencoder (VAE), convolutional network for biomedical image segmentation (U-Net), and text encoder. Unlike the traditional learning of denoising image data in pixel space, the training process of the stable diffusion model is carried out by training VAE to transform the image into a low-dimensional latent space. The process of adding and removing Gaussian noise is applied to this latent representation, and then the final denoised output is decoded back to the pixel space. In the forward diffusion process, Gaussian noise is iteratively applied to the compressed latent representation. Each denoising step is completed by a U-Net architecture containing a residual network (ResNet), and the latent representation is obtained by forward diffusion to backward denoising. Finally, the VAE decoder transforms the representation back to the pixel space to generate the output image. The way the latent representation is used makes the stable diffusion model have the advantage of reducing the computational requirements for training and generation.

[0077] S205: Generate a text illustration for the text segment through an illustration generation model according to the virtual character data and the target storyboard.

[0078] As can be seen from the foregoing, the virtual scene described by the target storyboard has a high degree of similarity with the virtual scene described by the virtual scene data. The virtual character data is used to describe the character features of the virtual character included in the virtual scene described by the virtual scene data. The virtual character data and the target storyboard are input into the picture generation model, so that the picture generation model generates text pictures for the text fragments. Moreover, in the process of generating text pictures, the picture generation model can also control the virtual character in the current virtual scene and in other virtual scenes to be less different, that is, to fix the virtual character. In addition, because the target storyboard has a camera movement mode, the text picture generated based on the target storyboard has a camera movement effect, which is equivalent to the picture generation model having a storyboard function, thereby improving the diversity of text pictures.

[0079] As a possible implementation method, although the virtual scene described by the target storyboard is highly similar to the virtual scene described by the virtual scene data, there are still differences between the two. Therefore, in order to improve the accuracy of text illustrations, text illustrations for text fragments can be generated by an illustration generation model based on virtual character data, virtual scene data and target storyboards. Therefore, the illustration generation model can not only learn the camera movement method based on the target storyboard, but also refine the scene features in the text illustration based on the virtual scene data, thereby improving the accuracy of text illustrations.

[0080] It can be seen from the above technical solution that virtual scene data and virtual character data for a text segment are obtained, the virtual scene data is used to describe the scene features of the virtual scene corresponding to the text segment, and the virtual character data is used to describe the character features of the virtual character included in the virtual scene. A target storyboard with a high similarity to the virtual scene data is obtained from multiple storyboards, so that the virtual scene described by the virtual scene data can use the camera movement method of the target storyboard, and its popularity can also be greater than a preset welcome threshold. Different virtual characters correspond to different role fixed models, and the role fixed model corresponding to the virtual character described by the virtual character data is obtained. The role fixed model is used to control the difference of the same virtual character in different virtual scenes to be less than the preset difference threshold, that is, to control the character features of the same virtual character in different virtual scenes to be closer. Thereby, the role fixed model is fused with the image generation model to obtain a picture generation model, and a text picture for the text segment is generated by the picture generation model based on the virtual character data and the target storyboard.

[0081] Thus, in different text illustrations corresponding to different text segments, the character features of the same virtual character in different virtual scenes are consistent, which improves the accuracy of text illustrations. Moreover, through the target storyboards applicable to the virtual scenes, the text illustrations have the storyboard function, which increases the diversity of text illustrations. At the same time, the popularity of the target storyboard is greater than the preset welcome threshold, and the text illustrations obtained based on the target storyboard are more likely to be popular. This improves the generation effect of the text illustrations and improves the user's reading experience.

[0082] As a possible implementation method, the embodiment of the present application does not specifically limit the method of determining the virtual scene data and the virtual character data, and two methods are used as examples for explanation below.

[0083] Method 1: Get the result through three questions and answers. See A1-A4 for details (not shown in the figure).

[0084] A1: Obtain target text content, first role prompt words, first scene prompt words and related prompt words.

[0085] The target text content is the text content waiting for the generation of text matching pictures. The target text content can be divided into multiple text segments, each of which can generate a corresponding text matching picture through S201-S205, thereby obtaining multiple text matching pictures corresponding to the target text content.

[0086] It is understandable that in the specific implementation of the present application, data related to users, such as user information obtained by users in the process of uploading target text content, etc., when the above embodiments of the present application are applied to specific products or technologies, it is necessary to obtain separate permission or consent from the user, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0087] The first role prompt word is a question sentence used to ask a question to the first text question-answering model, and based on the question sentence, the first text question-answering model is instructed to output the virtual role and role features of the virtual role included in the target text content.

[0088] The first text question-answering model has a question-answering function, that is, generating an answer to the question statement according to the input question statement. The embodiment of the present application does not specifically limit the first text question-answering model, and those skilled in the art can set it according to actual needs. For example, the first text question-answering model can be a Chat Generative Pre-trained Transformer (ChatGPT), which is specially designed for the task of conducting conversations and generating natural language texts. Through large-scale pre-training and fine-tuning processes, it can generate coherent and logical replies related to the input text. ChatGPT is trained on massive conversation data, so that it has the ability to understand and generate conversations. It can receive user input and generate corresponding replies to make the conversation more fluent and natural. The training process of ChatGPT involves a large amount of text data, including conversations on the Internet, social media, novels, etc. By learning these data, it can capture the grammar, semantics, contextual information and article structure information of the language.

[0089] Taking the first text question-answering model as a chat generation pre-trained converter and the target text content as a novel as an example, task prompts refer to keywords or phrases used in the input to guide ChatGPT to generate a specific type of response. These task prompts can help guide ChatGPT to generate answers or content related to a specific task. For example, the first role prompt is equivalent to a character prompt, which is used to extract characters in the full text of the novel. The first role prompt can be as follows:

[0090] Suppose you are a director who wants to make a movie. You have a novel, <target text content>. You need to extract the protagonists from the director's perspective and describe the characteristics of the protagonists. Return the result in the following json format:

[0091]

[0092] It should be noted that by adding multiple versions of languages ​​such as Chinese and English to question and answer sentences such as the first role prompt words, the first text question and answer model can better understand the question and answer sentences, so that it can be applied to more types of target text content and expand the application scenarios.

[0093] The first scene prompt word is a question sentence used to ask questions to the first text question and answer model. Based on the question sentence, the first text question and answer model is instructed to output virtual scene data corresponding to each text segment included in the target text content. For example, the target text content is divided into multiple text segments so that each text segment corresponds to a virtual scene.

[0094] As a possible implementation method, since the descriptions of various virtual scenes in the target text content may be discontinuous, such as there may be environments that are not mentioned in the novel paragraph, in order to allow the first text question answering model to make associations based on context information, so that the virtual scenes can remain consistent in continuous pictures, that is, the similarity of adjacent virtual scenes is greater than a preset value.

[0095] Continuing with the example that the first text question-answering model generates a pre-trained converter for chat and the target text content is a novel, the first scene prompt word is equivalent to the scene prompt word, which is used to extract information related to the virtual scene in each text segment of the target text content, that is, to extract virtual scene data. The first scene prompt word can be as follows:

[0096] Suppose you are a director, and here is a novel, <target text content>, you are required to summarize the scenes in the whole novel and sort them by numbers, and associate the numbered scenes with each paragraph in the novel. The output is required to be in JSON format, according to the following example:

[0097]

[0098] The associated prompt word is a question sentence used to ask questions to the first text question-answering model. Based on the question sentence, the first text question-answering model is instructed to output the virtual characters included in each virtual scene, that is, to establish an association relationship between the virtual scene and the virtual character.

[0099] Taking the first text question-answering model as a chat generation pre-trained converter and the target text content as a novel as an example, the first role prompt word is equivalent to the character prompt word, which is used to extract the characters in the full text of the novel. The first role prompt word can be as follows:

[0100] Suppose you are a director, and here is a novel, <target text content>. You are required to construct a storyboard based on the novel content and the logic of filming, and associate {character ID} and {scene ID} with the paragraphs corresponding to the storyboard according to the sequence number. You are required to output the result in json format. The storyboard needs to be related to the novel content, and the background of the same scene must be consistent. json format return example:

[0101]

[0102] A2: According to the first role prompt word and the target text content, the virtual role included in the target text content and the role characteristics of the virtual role are obtained through the first text question answering model.

[0103] From the above, it can be seen that the first text question and answer model can output the corresponding answer based on the input question and answer sentence. Therefore, the first role prompt word is a question and answer sentence for the target text content. The corresponding answer can be obtained through the first text question and answer model, that is, the virtual character included in the target text content and the role characteristics of the virtual character.

[0104] A3: According to the first scene prompt word and the target text content, a plurality of virtual scenes included in the target text content and virtual scene data corresponding to each virtual scene are obtained through the first text question-answering model.

[0105] Similarly, the first scene prompt word is a question-and-answer sentence for the target text content, and the first text question-and-answer model can obtain its corresponding answer, that is, the multiple virtual scenes included in the target text, and the virtual scene data corresponding to each virtual scene. In other words, the first text question-and-answer model not only obtains the answer corresponding to the first scene prompt word, but also divides the target text content into multiple text segments, thereby eliminating the need for users to divide the target text content into multiple text segments, thereby improving the user experience.

[0106] A4: According to the role characteristics of the virtual role, the virtual scene data, the associated prompt words and the target text content, the virtual role included in each virtual scene is obtained through the first text question-answering model.

[0107] The virtual character data includes the virtual characters included in each virtual scene and the character features of the virtual characters.

[0108] The associated prompt words are question-and-answer sentences for the target text content, and their corresponding answers can be obtained through the first text question-and-answer model, that is, an association relationship between the role characteristics of the virtual character and the virtual scene data is established, and the virtual characters included in each virtual scene are obtained, thereby obtaining the virtual character data.

[0109] Thus, by asking questions to the first text question-answering model three times, respectively: (1) the first text question-answering model is made to understand the full text of the target text content, so as to obtain the role characteristics of each virtual character globally, so that the role characteristics can describe the virtual character more accurately. (2) The first text question-answering model is made to understand the full text of the target text content, so as to divide the target text content into multiple text segments, so that each text segment corresponds to a virtual scene, and each virtual scene corresponds to a storyboard, so that each text segment can use a storyboard map later, thereby eliminating the need for the user to divide the target text content into multiple text segments, improving the degree of automation and reducing the workload of the user. (3) The first text question-answering model is made to understand the full text of the target text content, so as to establish an association relationship between the virtual character and the virtual scene, that is, which virtual character or characters are included in each virtual scene, so as to generate corresponding text illustrations for each text segment later.

[0110] Method 2: Get the result through one question and answer. See B1-B2 for details (not shown in the figure).

[0111] B1: Obtain the target text content, the second role prompt word and the second scene prompt word.

[0112] The second role prompt word is a question sentence used to ask questions to the second text question-answering model, and based on the question sentence, the second text question-answering model is instructed to output the virtual role data included in the target text content. It is understandable that the second role prompt word is different from the first role prompt word. The second text question-answering model can be the same as the first text question-answering model, or it can be different from the first text question-answering model, and this application does not make specific limitations on this.

[0113] The second scene prompt word is a question sentence used to ask the second text question answering model, and based on the question sentence, the second text question answering model is instructed to output virtual scene data corresponding to each text segment included in the target text content. It can be understood that the second role prompt word can be the same as the first role prompt word, or it can be different from the first role prompt word.

[0114] B2: According to the target text content, the second role prompt word and the second scene prompt word, the virtual scene data and the virtual role data for the target text content are obtained through the second text question-answering model.

[0115] The second text question-answering model can output corresponding answers based on the input question-answering sentences. Therefore, the second role prompt words and the second scene prompt words are question-answering sentences for the target text content. The corresponding answers, namely the virtual scene data and virtual role data for the target text content, can be obtained through the second text question-answering model.

[0116] Therefore, compared with method one, method two can obtain the virtual scene data and virtual character data corresponding to each virtual scene included in the target text content through only one question and answer. While improving the degree of automation, it reduces the number of interactions with the question and answer model. The question and answer method is simple and convenient.

[0117] As a possible implementation method, if a text question answering model such as the first text question answering model or the second text question answering model has a correction function, the answer output by the text question answering model can be corrected by the correction function of the text question answering model, thereby improving the accuracy of the answer.

[0118] Specifically, an introspection prompt word is obtained, which is a question sentence used to verify the answer output by the text question-answering model, such as verifying whether the virtual characters included in each virtual scene output in method 1, or the virtual scene data and virtual character data output in method 2 are accurate. Verification is performed through the text question-answering model based on the virtual scene data, virtual character data and introspection prompt words. The verification result can be pass or fail. If passed, the original virtual scene data can be directly used as the updated virtual scene data, and the original virtual character data can be used as the updated virtual character data. If failed, the updated virtual scene data and updated virtual character data can be obtained based on the newly output answer.

[0119] Taking the first text question-answering model as a chat generation pre-trained converter and the target text content as a novel as an example, ChatGPT, as a general large model, sometimes produces incorrect results in special tasks, such as the generated characters are inconsistent with the characters in the text fragment, the generated scenes are inconsistent with the scenes in the text fragment, etc. Considering the above phenomena, we can use the introspection ability of ChatGPT, that is, the correction function of the text question-answering model, and use the introspection prompt words to let ChatGPT introspect on the existing results, so as to further increase the accuracy of the results.

[0120] For example, the introspection prompt words can be "Now please check and evaluate your output results. Make sure that your answer is accurate and meets the needs of users. Please conduct an in-depth analysis of your answer to see if there is any room for improvement or optimization. Mainly confirm whether the novel disassembly results are consistent with the original text." By using introspection prompt words, ChatGPT will check whether the output answer is consistent with the original text, thereby improving the robustness of novel disassembly.

[0121] Therefore, by using the correction function of the text question-answering model, combined with the self-reflection prompt words, and by asking the text question-answering model again, the accuracy of the final answer is improved.

[0122] In order to facilitate further understanding of the technical solution provided in the embodiments of the present application, the following takes the first text question-answering model as a chat-generated pre-trained converter and the target text content as a novel as an example to provide an overall exemplary introduction to obtaining virtual scene data and virtual character data for each text fragment.

[0123] See also Figure 3 , which is a schematic diagram of determining virtual scene data and virtual character data of a text segment provided in an embodiment of the present application.

[0124] S301: Obtain the text to be input.

[0125] The text to be input may be a novel text.

[0126] S302: Perform text cleaning on the text to be input to obtain target text content.

[0127] The embodiments of the present application do not specifically limit the specific method of text cleaning, such as removing unknown symbols and garbled characters that may exist in the text to be input.

[0128] S303: Construct question and answer sentences.

[0129] The question and answer sentence can be the first role prompt word, the first scene prompt word and the associated prompt word in the aforementioned method one, and can also be the second role prompt word and the second scene prompt word in the aforementioned method two.

[0130] Taking the first method as an example, the first role prompt word α, the first scene prompt word β and the associated prompt word γ can be constructed.

[0131] S304: Input the question-answering sentence into the text question-answering model to obtain virtual scene data and virtual character data.

[0132] Continuing with method 1 as an example, the first role prompt word α and the target text content are input into ChatGPT to obtain the virtual role information α o , that is, the virtual characters and character features of the virtual characters included in the target text content. Input the first scene prompt word β and the target text content into ChatGPT to obtain the virtual scene data β o . The virtual character information α o , virtual scene data β o And the target text content is input into ChatGPT to obtain the paragraph information of the target text content γ o , that is, multiple virtual scenes included in the target text content, and virtual scene data corresponding to each virtual scene.

[0133] S305: Verify the virtual scene data, the virtual character data and the self-reflection prompt words through a text question-answering model to obtain updated virtual scene data and updated virtual character data.

[0134] Continuing with method 1 as an example, according to the virtual character information α o , the target text content and the introspection prompt words are verified through ChatGPT to obtain the updated virtual character information α o According to the virtual scene data β o , the target text content and the introspection prompt words are verified through ChatGPT to obtain the updated virtual scene data β o . According to the paragraph information γ o , the target text content and introspection prompt words are verified through ChatGPT to obtain the updated paragraph information γ o. Based on the updated virtual character information α o , updated virtual scene data β o and updated paragraph information γ o The updated virtual scene data and the updated virtual character data are obtained.

[0135] As a possible implementation method, the embodiment of the present application provides a specific implementation method of S202, that is, a specific implementation method of obtaining a target storyboard from multiple storyboards based on virtual scene data, see S2021-S2023 (not shown in the figure) for details.

[0136] S2021: Based on the virtual scene data, feature extraction is performed through a multimodal feature extractor to obtain scene features of the virtual scene data.

[0137] The multimodal feature extractor is used to perform feature extraction on input data of different modalities. The multimodal input data can be text, image, etc., which is not specifically limited in this application. Therefore, the virtual scene data is input into the multimodal feature extractor for feature extraction, and the scene features corresponding to the virtual scene data can be obtained.

[0138] Continue with the above Figure 3 For example, see Figure 4 , which is a schematic diagram of determining a target storyboard provided in an embodiment of the present application.

[0139] Can be based on updated paragraph information γ i Included scene identifier scene_id, from the virtual scene data β i The description of the virtual scene is obtained by indexing in , so that the multimodal feature extractor is used to extract features and obtain the scene features of the virtual scene data. Wherein, i is the i-th text segment among the multiple text segments included in the target text content.

[0140] S2022: Based on the multiple storyboards, feature extraction is performed using a multimodal feature extractor to obtain storyboard image features corresponding to the multiple storyboards.

[0141] Similarly, multiple storyboards are respectively input into the multimodal feature extractor for feature extraction to obtain storyboard image features corresponding to each storyboard, wherein the storyboard image features are used to describe the features of the storyboards.

[0142] The embodiment of the present application does not specifically limit the order of executing S2021 and S2022, and they can also be executed simultaneously. As a possible implementation method, if multiple storyboards do not change, S2022 can be executed once in advance to obtain the storyboard image features of each storyboard, and save them for use. For example, after executing S2021, S2023 is executed based on the saved multiple storyboard image features, and there is no need to execute S2022 every time, thereby improving the speed of generating text and pictures.

[0143] Continue to see Figure 4 , feature extraction can be performed on each of the multiple storyboards included in the storyboard library to obtain and save multiple storyboard image features.

[0144] S2023: Determine the similarity between the scene feature and the multiple storyboard image features, and use the storyboard image corresponding to the storyboard image feature whose similarity is greater than a first similarity threshold as the target storyboard image.

[0145] It is understandable that both images and texts are carriers of information, but they are expressed in different ways. Image features used to describe images usually include color, shape, texture, spatial layout, etc., while text features used to describe texts include vocabulary, grammar, semantics, context, etc. Scene features belong to text features, and storyboard image features belong to image features. The similarity between text features and image features refers to the degree of similarity between the two in terms of content, style, semantics, etc.

[0146] If, based on the similarity calculation, the similarity between a certain storyboard image feature and a scene feature is greater than a first similarity threshold, that is, the similarity between a certain storyboard image feature and a scene feature is relatively high, then the storyboard corresponding to the storyboard image feature can be used as the target storyboard, so that the target storyboard can be used to represent the virtual scene described by the virtual scene data, the camera movement method of the target storyboard can be applicable to the virtual scene described by the virtual scene data, and the popularity of the target storyboard is relatively high, and the popularity of the subsequent text illustrations generated based on the target storyboard will also be relatively high.

[0147] Continue to see Figure 4 , the similarity between the scene features and the multiple storyboard image features corresponding to the storyboard library is calculated respectively, the similarity between the scene features and each storyboard image feature is obtained, and the storyboards corresponding to the top ten storyboard image features are recalled, and sorted according to the similarity, and the top three storyboards are used as target storyboards.

[0148] Therefore, feature extraction is performed on the storyboard and virtual scene data respectively through a multimodal feature extractor to obtain scene features and storyboard image features. According to the similarity between the scene features and the storyboard image features, a suitable storyboard is determined from multiple storyboards as a target storyboard, so that the text illustration obtained based on the target storyboard with higher accuracy has higher accuracy.

[0149] As a possible implementation, multiple storyboard image features similar to the scene feature can be obtained through S2023, so as to obtain multiple storyboards as target storyboards. For example, the storyboards corresponding to the storyboard image features ranked top three in similarity are used as the target storyboards. The following takes the target storyboard including the first storyboard and the second storyboard as an example, and two usage scenarios are described.

[0150] Case 1: Get text and picture based on user selection. See C1-C4 for details (not shown in the figure).

[0151] C1: Generate a first text illustration for the text segment through an illustration generation model based on the virtual character data and the first storyboard.

[0152] The similarity between the storyboard image features of the first storyboard and the scene features of the virtual scene described by the virtual scene data is greater than a first similarity threshold. The first text matching image is a text matching image obtained based on the first storyboard.

[0153] As a possible implementation, the first text illustration for the text segment can be generated by an illustration generation model according to the virtual character data, the virtual scene data and the first storyboard. Thus, the accuracy of the first text illustration can be increased by adding virtual scene data.

[0154] C2: Generate a second text illustration for the text segment through an illustration generation model based on the virtual character data and the second storyboard.

[0155] The similarity between the storyboard image features of the second storyboard and the scene features of the virtual scene described by the virtual scene data is greater than a first similarity threshold, and the second storyboard and the first storyboard are different storyboards. The second text image is a text image obtained based on the second storyboard.

[0156] As a possible implementation, the second text illustration for the text segment can also be generated by an illustration generation model based on the virtual character data, the virtual scene data and the second storyboard. Thus, the accuracy of the second text illustration can be increased by adding virtual scene data.

[0157] C3: If a selection operation for the first text image is obtained, the first text image is used as the text image of the text segment.

[0158] C4: If a selection operation for the second text image is obtained, the second text image is used as the text image of the text segment.

[0159] The first text matching image and the second text matching image are displayed to the user, and the user can select a more appropriate text matching image from the first text matching image and the second text matching image and use it as the text matching image for the text segment, as specifically shown in C3 and C4.

[0160] Therefore, by selecting multiple more suitable storyboards (such as the first storyboard or the second storyboard) from multiple storyboards as target storyboards, multiple selectable text illustrations (such as the first text illustration or the second text illustration) are obtained, so that the multiple selectable text illustrations can be displayed to the user, making it convenient for the user to select a more suitable text illustration as the text illustration for the current text segment, enriching the text illustrations, increasing the user's choices, and improving the user's experience.

[0161] Case 2: Automatically select a text image with a higher similarity. See D1-D6 for details (not shown in the figure).

[0162] D1: Generate a first text illustration for the text segment through an illustration generation model according to the virtual character data and the first storyboard.

[0163] The similarity between the storyboard image features of the first storyboard and the scene features of the virtual scene described by the virtual scene data is greater than a first similarity threshold. The first text matching image is a text matching image obtained based on the first storyboard.

[0164] As a possible implementation, the first text illustration for the text segment can be generated by an illustration generation model according to the virtual character data, the virtual scene data and the first storyboard. Thus, the accuracy of the first text illustration can be increased by adding virtual scene data.

[0165] D2: Generate a second text illustration for the text segment through an illustration generation model based on the virtual character data and the second storyboard.

[0166] The similarity between the storyboard image features of the second storyboard and the scene features of the virtual scene described by the virtual scene data is greater than a first similarity threshold, and the second storyboard and the first storyboard are different storyboards. The second text image is a text image obtained based on the second storyboard.

[0167] D3: Based on the text fragment, feature extraction is performed through a multimodal feature extractor to obtain the text features of the text fragment.

[0168] D4: Based on the first text image, feature extraction is performed by a multimodal feature extractor to obtain a first image feature of the first text image.

[0169] D5: Based on the second text image, feature extraction is performed by a multimodal feature extractor to obtain second image features of the second text image.

[0170] D6: Determine the similarity between the first image feature and the text feature, and determine the similarity between the second image feature and the text feature, and use the text image corresponding to the image feature with higher similarity as the text image for the text segment.

[0171] Therefore, by selecting multiple relatively suitable storyboards (such as the first storyboard or the second storyboard) from multiple storyboards as target storyboards, multiple pending text illustrations (such as the first text illustration or the second text illustration) are obtained, so as to respectively calculate the similarities between the illustration image features (such as the first illustration image features or the second illustration image features) corresponding to the multiple pending text illustrations and the text features, and use the pending text illustrations with higher similarity as the text illustration of the current text segment, thereby improving the accuracy of the text illustrations.

[0172] The embodiment of the present application does not specifically limit the method for obtaining the fixed role model, and the following is an example of a method for obtaining the fixed role model by training a model. In this embodiment, the execution subject of the text and picture generation method is still taken as an example of a server.

[0173] See also Figure 5 , which is a schematic diagram of a role fixed model obtained by training provided in an embodiment of the present application.

[0174] S501: Acquire multiple virtual characters included in the target text content and character features corresponding to the multiple virtual characters.

[0175] The target text content includes multiple text segments, each text segment corresponds to a virtual scene, and each virtual scene may not include a virtual character, or may include one or more virtual characters, so that the target text content includes one or more virtual characters. The following is an example in which the target text content includes multiple virtual characters.

[0176] S502: For a target virtual character among the multiple virtual characters, determine multi-perspective character data for the target virtual character according to character characteristics of the target virtual character.

[0177] The target virtual character is one of the multiple virtual characters included in the target text content. For example, the target virtual character can be represented as p k , that is, the kth virtual character among multiple virtual characters is the target virtual character.

[0178] The multi-view character data is used to describe the character features of the target virtual character at different viewing angles, such as the character features corresponding to the front view, the character features corresponding to the left view, etc. As a possible implementation, the multi-view character data may be the character features corresponding to the front view, the left view, the right view, the top view, and the rear view, that is, the multi-view character data may be the character features corresponding to the virtual character at the five viewing angles.

[0179] As a possible implementation method, a diffusion model or other model can be used to generate corresponding character features of the target virtual character under five viewing angles based on the character data of the target virtual character, thereby obtaining multi-view character data for the target virtual character.

[0180] As a possible implementation method, based on the character features corresponding to the five views, the data volume can be increased through data augmentation methods such as cropping and horizontal flipping to obtain multi-view character data with a larger data volume, so as to improve the accuracy of subsequent model training.

[0181] S503: According to the multi-view character data, a model is generated through an initial picture matching method to obtain a predicted text picture matching method.

[0182] The initial picture generation model is obtained by fusing the initial character fixed model and the image generation model. Compared with the picture generation model, the initial picture generation model is a model that has not been trained yet, and its accuracy is less than or equal to the picture generation model. Similarly, compared with the character fixed model, the initial character fixed model is a model that has not been trained yet, and its accuracy is less than or equal to the character fixed model.

[0183] The predicted text image is the text image obtained by the initial image generation model.

[0184] S504: According to the difference between the predicted text illustration and the target virtual character, the model parameters of the image generation model in the initial illustration generation model are fixed, and the model parameters of the initial character fixing model in the initial illustration generation model are adjusted to obtain a character fixing model suitable for the target virtual character.

[0185] The predicted text illustration is generated based on the multi-perspective character data corresponding to the target virtual character, so the predicted text illustration includes the character image of the target virtual character. Based on the difference between the character image and the target virtual character, the model parameters of the image generation model in the initial illustration generation model are fixed, and the model parameters of the initial character fixed model in the initial illustration generation model are adjusted to obtain a character fixed model suitable for the target virtual character.

[0186] In the related art, the model parameters of the initial picture generation model are generally adjusted, that is, the model parameters of the image generation model included in the initial picture generation model are adjusted. Since the image generation model is generally a large model, the adjustment of the initial picture generation model takes a long time.

[0187] Based on this, the embodiment of the present application no longer fine-tunes all the model parameters included in the initial picture generation model, but generates the initial picture generation model based on the image generation model and the initial character fixed model, fixes the model parameters of the image generation model in the initial picture generation model, and adjusts the model parameters of the initial character fixed model in the initial picture generation model, that is, no longer adjusts all the model parameters of the large model such as the initial picture generation model, but fixes the model parameters of the image generation model unchanged, and only adjusts the parameters of the initial character fixed model to achieve fine-tuning of the large model. Therefore, by using very little training data, the accuracy of the large model can be improved, allowing the large model to "stand out" from the downstream tasks that originally did not perform so well and become an expert in this task. This saves training resources while also improving the training speed.

[0188] It is understandable that the text-with-picture generation method provided in the embodiment of the present application can mainly involve artificial intelligence technology, and the generation of text-with-picture for text fragments is automatically realized through artificial intelligence technology. Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machine has the functions of perception, reasoning and decision-making.

[0189] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0190] In the embodiments of the present application, the artificial intelligence technologies mainly involved include the above-mentioned natural language processing technology direction. Natural language processing (Natural Language processing, NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between people and computers using natural language. Natural language processing involves natural language, that is, the language used by people in daily life, which is closely related to linguistic research; it also involves important technologies for model training in the fields of computer science, mathematics and artificial intelligence. Among them, the pre-trained model is developed from the large language model (Large Language Model) in the field of NLP. After fine-tuning, the large language model can be widely used in downstream tasks. Natural language processing technology generally includes text processing, semantic understanding, machine translation, robot question and answer, knowledge graph and other technologies.

[0191] Among them, the pre-training model (PTM), also known as the cornerstone model or large model, refers to a deep neural network (DNN) with large parameters. It is trained on a large amount of unlabeled data, and the function approximation ability of the large-parameter DNN is used to enable the PTM to extract common features from the data. After fine tuning, parameter efficient fine-tuning (PEFT), prompt learning (prompt-tuning) and other technologies, it is suitable for downstream tasks. Therefore, the pre-training model can achieve ideal results in a few-shot or zero-shot scenario. According to the data modality processed, PTM can be divided into language models (such as ELMO, BERT, GPT, etc.), visual models (such as swin-transformer, ViT, V-MOE, etc.), speech models (such as VALL-E), multimodal models (such as ViBERT, CLIP, Flamingo, Gato, etc.), etc., among which the multimodal model refers to a model that establishes two or more data modality feature representations. Pre-trained models are important tools for outputting artificial intelligence generated content (AIGC), and can also be used as a general interface to connect multiple specific task models.

[0192] In the text and image generation method provided in the embodiment of the present application, the role fixed model, the image generation model, etc. can all be models obtained based on the pre-trained model. The embodiment of the present application does not specifically limit the role fixed model, and those skilled in the art can select it according to actual needs.

[0193] For example, you can choose the Low-Rank Adaptation of Large Language Models (LoRA) model for large language models. Among them, LoRA is a training technology in the diffusion model, which belongs to the parameter efficiency (low rank) fine-tuning method of large language models. It is mainly used to fine-tune large language models. LoRA can reduce the size of the fine-tuning model by 10 to 100 times by making minor modifications to the standard fine-tuning model (checkpoint). The training process of LoRA is to train the parameter increments of certain layers while fixing the parameters of the large model. These parameter increments can be turned into fewer trainable parameters through matrix decomposition, thereby greatly reducing the amount of training parameters required for fine-tuning. Specifically, LoRA reduces the number of parameters by decomposing a weight matrix into two matrices for storage. This can reduce the demand for computing resources when fine-tuning the model, so that ordinary users can also use LoRA on general graphics processing units (GPUs). Compared with other diffusion model training techniques (such as Dreambooth and textual inversion), LoRA strikes a balance between model size and effect.

[0194] S505: Determine the multiple virtual characters as target virtual characters respectively, and obtain the character fixed models corresponding to the multiple virtual characters respectively.

[0195] Therefore, different virtual characters correspond to different character fixing models, so that the character fixing model can more specifically fix the virtual characters of each text segment included in the target text content, increase the acceptance and refinement of the target text content, and attract more readers.

[0196] Furthermore, since different virtual characters correspond to different fixed character models, if the virtual scene includes multiple virtual characters, the fixed character models corresponding to the multiple virtual characters can be fused with the image generation model to obtain the image generation model.

[0197] Specifically, virtual scene data and multiple virtual character data for the text fragment are obtained, and different virtual character data are used to describe different virtual characters. Based on the multiple virtual character data, fixed role models corresponding to the virtual characters included in the multiple virtual character data are obtained, and the fixed role models corresponding to the multiple virtual characters and the image generation model are fused to obtain a picture generation model.

[0198] Therefore, when the virtual scene includes multiple virtual characters, the character fixed models corresponding to the multiple virtual characters are obtained, and the multiple character fixed models are fused with the image generation model to obtain the illustration generation model. In this way, when the illustration generation model generates text illustrations according to the virtual character data and the target storyboard, it is possible to fix each virtual character in the text illustration in a targeted manner, thereby improving the accuracy of the text illustration.

[0199] As a possible implementation, in addition to fixing the virtual character through the character fixing model, the embodiment of the present application also implements fine-tuning of the virtual character through the difference adjustment model.

[0200] Specifically, a difference adjustment model is obtained, and the text illustration for the text fragment is identified by the difference adjustment model to determine whether the character features of the multiple virtual characters included in the text illustration are similar, that is, to determine whether the multiple virtual characters included in the text illustration are similar in character image. If the similarity of the multiple character features is greater than the second similarity threshold, it means that at least two of the multiple virtual characters are similar and the difference between the two is low. In order to make the virtual characters different and facilitate users to distinguish them, the virtual characters corresponding to the character features with a similarity greater than the second similarity threshold are fine-tuned, and the difference between the virtual characters is improved while fixing the virtual characters, thereby obtaining an updated text illustration.

[0201] Therefore, through the difference fine-tuning model, the distinction between multiple virtual characters included in the text illustrations can be made higher, which is convenient for users to distinguish and improves the reading experience of readers. This ensures that the image differences of the same virtual character in multiple text illustrations with related relationships are small, while improving the distinction between each virtual character.

[0202] In order to facilitate further understanding of the technical solution provided in the embodiments of the present application, the text illustration generation method provided in the embodiments of the present application is taken as an example in which the execution subject is a server and the target text content is a novel, and the text illustration generation method is introduced as a whole in combination with the text illustration generation system.

[0203] See also Figure 6 , which is a schematic diagram of a text-to-picture system provided in an embodiment of the present application.

[0204] The text-illustration system includes a script disassembly module 601, a character model training module 602, a storyboard retrieval module 603 and a fusion module 604, and each module is described below.

[0205] The script disassembly module 601 is mainly used to perform in-depth analysis and disassembly of the novel. After the novel is input into the script disassembly module 601, key information such as characters (i.e., virtual characters), plots (used to extract scenes), and scenes (i.e., virtual scenes) in the novel can be extracted through the aforementioned method one or method two to obtain the virtual character data and virtual scene data included in each text segment.

[0206] The character model training module 602 is used to train and obtain a fixed character model, such as the character features of the virtual character obtained by the script disassembly module 601, and train the corresponding fixed character models of the virtual characters included in the novel through the aforementioned S501-S505. In order to ensure that the character image of the generated virtual character is consistent with the character image described in the novel through the fixed character model, and to ensure the consistency of the same virtual character in multiple virtual scenes, that is, the difference of the same virtual character in multiple virtual scenes is small, thereby improving the accuracy and authenticity of the text illustration.

[0207] The storyboard retrieval module 603 is used to obtain a target storyboard from multiple storyboards according to the virtual scene data, such as retrieving the most matching target storyboard from multiple storyboards according to the virtual scene data such as plots and scenes provided by the script disassembly module 601. This allows the text illustrations generated subsequently to be highly matched with the plots and scenes described in the novel, thereby improving the fidelity and attractiveness of the text illustrations.

[0208] The fusion module 604 obtains the virtual character data and virtual scene data included in each text segment from the script disassembly module 601, and obtains the target storyboard corresponding to each text segment from the storyboard retrieval module 603, and generates text illustrations corresponding to each text segment through the illustration generation model corresponding to each text segment, thereby obtaining the novel illustration corresponding to the novel based on the multiple text illustrations corresponding to the multiple text segments included in the novel. The illustration generation model is obtained by obtaining the role fixed model and image generation model corresponding to the virtual character included in each text segment from the character model training module 602.

[0209] See also Figure 7 , which is a scene diagram of a text-with-picture generation method provided in an embodiment of the present application. Figure 7The fusion module 604 is described in detail in . By splitting the novel through the script disassembly module 601, multiple text segments, character features of the virtual characters included in the novel, and action descriptions and expression descriptions of each virtual character (the action descriptions and expression descriptions can make the virtual characters more vivid) can be obtained first, and then virtual scene data and virtual character data corresponding to each text segment can be obtained, such as scene identification, character identification, updated results obtained through the correction function, etc., and even the camera movement method suitable for each text segment can be selected based on multiple preset camera movement methods.

[0210] For one of the text segments, the script disassembly module 601 can obtain the virtual character data and virtual scene data included in the text segment, obtain the role fixed models corresponding to the multiple virtual characters included in the current virtual scene from the character model training module 602, and fuse the multiple role fixed models with the image generation model to obtain the picture generation model. The target storyboard corresponding to the current virtual scene is obtained from the storyboard retrieval module 603. The target storyboard can be divided into a scene line draft and a scene segmentation map, so that the picture generation model can learn the drawing method of the target storyboard according to the scene line draft, and learn the camera movement method of the target storyboard according to the scene segmentation map. Thus, the picture generation model can obtain the text picture corresponding to the text segment according to the virtual scene data, the virtual character data and the target storyboard.

[0211] Thus, text illustrations corresponding to the respective text segments included in the novel are generated, thereby obtaining a novel illustration including a plurality of text illustrations.

[0212] From the above technical solutions, it can be seen that the text-to-picture system can realize efficient, accurate and automatic novel illustration generation without manual intervention, which can significantly save time costs and generate novel illustrations with excellent effects. Specifically, it is mainly reflected in the following aspects:

[0213] (1) Improve production efficiency: The text-to-picture system can significantly improve the production efficiency of novel illustrations corresponding to novels. It can even generate novel tweets in the form of picture articles or videos of novel illustrations based on novel illustrations, reducing human resource investment and production costs. Compared with the traditional manual production method, it can complete the generation of a large number of high-quality novel tweets in a short time, meeting the user's demand for fast production of novel tweets.

[0214] (2) Ensure content consistency: The text-to-picture system can accurately generate novel tweets related to the novel based on the content of the novel, ensuring that the generated novel tweets are relatively consistent with the original content. This allows readers to better understand the plot of the novel when reading the novel tweets, improving their reading experience.

[0215] (3) Improve the attractiveness of novel tweets: The novel tweets generated by the text-to-picture system are highly attractive and can attract more readers’ attention. Through automated novel tweet generation, novel tweets with rich expressiveness and unique style can be generated, bringing readers a more attractive reading experience.

[0216] (4) Adapting to diverse needs: The text-to-picture system has strong versatility and scalability, and can adapt to the production needs of novel tweets of different types and styles. Whether it is realistic or fantasy themes, it can provide users with satisfactory novel tweets.

[0217] With respect to the text-with-picture generation method described above, the present application also provides a corresponding text-with-picture generation device, so that the above-mentioned text-with-picture generation method can be applied and implemented in practice.

[0218] See also Figure 8 , which is a schematic diagram of the structure of a text and picture generation device provided in an embodiment of the present application. Figure 8 As shown, the text and picture generation device 800 includes: an acquisition unit 801, a selection unit 802, a fusion unit 803 and a generation unit 804;

[0219] The acquisition unit 801 is used to acquire virtual scene data and virtual character data for a text segment, wherein the virtual scene data is used to describe scene features of a virtual scene corresponding to the text segment, and the virtual character data is used to describe character features of a virtual character included in the virtual scene;

[0220] The selection unit 802 is used to obtain a target storyboard from multiple storyboards according to the virtual scene data, the popularity of the virtual scene described by the storyboard is greater than a preset popularity threshold, and the similarity between the virtual scene described by the target storyboard and the virtual scene described by the virtual scene data is greater than a first similarity threshold;

[0221] The selection unit 802 is further configured to obtain, based on the virtual character data, a fixed role model corresponding to the virtual character included in the virtual character data, wherein different virtual characters correspond to different fixed role models, and the fixed role model is configured to control the difference of the same virtual character in different virtual scenes to be less than a preset difference threshold;

[0222] The fusion unit 803 is used to fuse the character fixed model and the image generation model to obtain a picture generation model;

[0223] The generating unit 804 is used to generate a text illustration for the text segment through the illustration generation model according to the virtual character data and the target storyboard.

[0224] It can be seen from the above technical solution that virtual scene data and virtual character data for a text segment are obtained, the virtual scene data is used to describe the scene features of the virtual scene corresponding to the text segment, and the virtual character data is used to describe the character features of the virtual character included in the virtual scene. A target storyboard with a high similarity to the virtual scene data is obtained from multiple storyboards, so that the virtual scene described by the virtual scene data can use the camera movement method of the target storyboard, and its popularity can also be greater than a preset welcome threshold. Different virtual characters correspond to different role fixed models, and the role fixed model corresponding to the virtual character described by the virtual character data is obtained. The role fixed model is used to control the difference of the same virtual character in different virtual scenes to be less than the preset difference threshold, that is, to control the character features of the same virtual character in different virtual scenes to be closer. Thereby, the role fixed model is fused with the image generation model to obtain a picture generation model, and a text picture for the text segment is generated by the picture generation model based on the virtual character data and the target storyboard.

[0225] Thus, in different text illustrations corresponding to different text segments, the character features of the same virtual character in different virtual scenes are consistent, which improves the accuracy of text illustrations. Moreover, through the target storyboards applicable to the virtual scenes, the text illustrations have the storyboard function, which increases the diversity of text illustrations. At the same time, the popularity of the target storyboard is greater than the preset welcome threshold, and the text illustrations obtained based on the target storyboard are more likely to be popular. This improves the generation effect of the text illustrations and improves the user's reading experience.

[0226] As a possible implementation manner, the selection unit 802 is specifically configured to:

[0227] According to the virtual scene data, feature extraction is performed by a multimodal feature extractor to obtain scene features of the virtual scene data;

[0228] According to the plurality of storyboards, feature extraction is performed respectively by the multimodal feature extractor to obtain storyboard image features corresponding to the plurality of storyboards respectively;

[0229] Determine the similarity between the scene feature and the plurality of storyboard image features, and use the storyboard image corresponding to the storyboard image feature having a similarity greater than the first similarity threshold as the target storyboard image.

[0230] As a possible implementation manner, if the target storyboard includes a first storyboard and a second storyboard, the generating unit 804 is specifically configured to:

[0231] Generate a first text illustration for the text segment by using the illustration generation model according to the virtual character data and the first storyboard;

[0232] Generate a second text illustration for the text segment by using the illustration generation model according to the virtual character data and the second storyboard;

[0233] If a selection operation for the first text image is obtained, the first text image is used as the text image of the text segment;

[0234] If a selection operation for the second text image is obtained, the second text image is used as the text image of the text segment.

[0235] As a possible implementation manner, if the target storyboard includes a first storyboard and a second storyboard, the generating unit 804 is specifically configured to:

[0236] Generate a first text illustration for the text segment by using the illustration generation model according to the virtual character data and the first storyboard;

[0237] Generate a second text illustration for the text segment by using the illustration generation model according to the virtual character data and the second storyboard;

[0238] According to the text segment, feature extraction is performed by the multimodal feature extractor to obtain text features of the text segment;

[0239] According to the first text image, feature extraction is performed by the multimodal feature extractor to obtain a first image feature of the first text image;

[0240] According to the second text image, feature extraction is performed by the multimodal feature extractor to obtain a second image feature of the second text image;

[0241] Determine the similarity between the first image feature and the text feature, and determine the similarity between the second image feature and the text feature, and use the text image corresponding to the image feature with higher similarity as the text image for the text segment.

[0242] As a possible implementation, if the text segment is a part of the target text content, the text and picture generating device 800 further includes a role fixed model training unit, which is used to:

[0243] Acquire multiple virtual characters included in the target text content and character features corresponding to the multiple virtual characters respectively;

[0244] For a target virtual character among the multiple virtual characters, determining multi-perspective character data for the target virtual character according to the character characteristics of the target virtual character, wherein the multi-perspective character data is used to describe the character characteristics of the target virtual character at different perspectives;

[0245] According to the multi-view character data, a predicted text image is obtained by using an initial image generation model, wherein the initial image generation model is obtained by fusing the initial character fixed model and the image generation model;

[0246] According to the difference between the predicted text picture and the target virtual character, the model parameters of the image generation model in the initial picture generation model are fixed, and the model parameters of the initial character fixed model in the initial picture generation model are adjusted to obtain a character fixed model suitable for the target virtual character;

[0247] The multiple virtual characters are respectively determined as the target virtual characters, and the fixed character models corresponding to the multiple virtual characters are obtained.

[0248] As a possible implementation, if the virtual scene includes multiple virtual characters, the acquisition unit 801 is specifically used to acquire virtual scene data and multiple virtual character data for the text segment;

[0249] The selection unit 802 is specifically configured to obtain, according to the plurality of virtual character data, fixed role models corresponding to the virtual characters included in the plurality of virtual character data;

[0250] The fusion unit 803 is specifically used to fuse the character fixed models corresponding to the multiple virtual characters and the image generation model to obtain the image generation model.

[0251] As a possible implementation, the text-with-picture generating device 800 further includes an adjustment unit, which is used to:

[0252] Get the difference-adjusted model;

[0253] According to the text illustration for the text fragment, character features of multiple virtual characters included in the text illustration are identified through the difference adjustment model. If the similarity of the multiple character features is greater than a second similarity threshold, the virtual characters corresponding to the character features whose similarity is greater than the second similarity threshold are fine-tuned to obtain an updated text illustration.

[0254] As a possible implementation, the text-with-picture generating device 800 further includes a splitting unit, which is used to:

[0255] Acquire target text content, a first character prompt word, a first scene prompt word, and an associated prompt word, wherein the target text content includes a plurality of the text segments, the first scene prompt word is a question sentence for acquiring a virtual character and the character features of the virtual character, the first scene prompt word is a question sentence for acquiring the virtual scene data, and the associated prompt word is a question sentence for associating the virtual scene with the virtual character;

[0256] According to the first role prompt word and the target text content, obtaining the virtual role included in the target text content and the role characteristics of the virtual role through a first text question-answering model;

[0257] According to the first scene prompt word and the target text content, obtaining multiple virtual scenes included in the target text content and virtual scene data corresponding to each of the virtual scenes through the first text question-answering model;

[0258] According to the role characteristics of the virtual character, the virtual scene data, the associated prompt words and the target text content, the virtual characters included in each of the virtual scenes are obtained through the first text question-answering model, and the virtual character data includes the virtual characters included in each of the virtual scenes and the role characteristics of the virtual characters.

[0259] As a possible implementation, the text-with-picture generating device 800 further includes a splitting unit, which is used to:

[0260] Acquire target text content, a second role prompt word and a second scene prompt word, wherein the second role prompt word is a question sentence for acquiring the virtual role data, and the second scene prompt word is a question sentence for acquiring the virtual scene data;

[0261] According to the target text content, the second role prompt word and the second scene prompt word, the virtual scene data and the virtual role data for the target text content are obtained through a second text question-answering model.

[0262] As a possible implementation, if the text question answering model also has a correction function, the text and picture generating device 800 further includes a correction unit for:

[0263] Obtaining a self-reflection prompt word, wherein the self-reflection prompt word is a question sentence used to verify the answer output by the text question answering model;

[0264] According to the virtual scene data, the virtual character data and the introspection prompt words, verification is performed through the text question-answering model to obtain updated virtual scene data and updated virtual character data.

[0265] As a possible implementation manner, the generating unit 804 is specifically configured to generate a text illustration for the text segment through the illustration generation model according to the virtual character data, the virtual scene data and the target storyboard.

[0266] The present application also provides a computer device, which may be a server or a terminal device. The following will introduce the computer device provided by the present application from the perspective of hardware entity. Fig. 9 The following is a schematic diagram of the server structure. Fig.10 Shown is a schematic diagram of the structure of the terminal equipment.

[0267] See also Fig. 9 , which is a schematic diagram of a server structure provided in an embodiment of the present application. The server 1400 may have relatively large differences due to different configurations or performances, and may include one or more processors 1422, such as central processing units (CPU), memory 1432, one or more application programs 1442 or storage media 1430 (such as one or more massive storage devices) for data 1444. Among them, the memory 1432 and the storage medium 1430 may be temporary storage or permanent storage. The program stored in the storage medium 1430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the processor 1422 may be configured to communicate with the storage medium 1430 to execute a series of instruction operations in the storage medium 1430 on the server 1400.

[0268] The server 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1458, and / or one or more operating systems 1441, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0269] The steps performed by the server in the above embodiment can be based on the Fig. 9 The server structure shown.

[0270] The CPU 1422 is used to perform the following steps:

[0271] Acquire virtual scene data and virtual character data for the text segment, wherein the virtual scene data is used to describe scene features of the virtual scene corresponding to the text segment, and the virtual character data is used to describe character features of the virtual character included in the virtual scene;

[0272] According to the virtual scene data, a target storyboard is obtained from a plurality of storyboards, the popularity of the virtual scene described by the storyboard is greater than a preset popularity threshold, and the similarity between the virtual scene described by the target storyboard and the virtual scene described by the virtual scene data is greater than a first similarity threshold;

[0273] According to the virtual character data, a fixed role model corresponding to the virtual character included in the virtual character data is obtained, where different virtual characters correspond to different fixed role models, and the fixed role model is used to control the difference of the same virtual character in different virtual scenes to be less than a preset difference threshold;

[0274] The character fixed model and the image generation model are integrated to obtain a picture generation model;

[0275] According to the virtual character data and the target storyboard, a text illustration for the text segment is generated by the illustration generation model.

[0276] Optionally, CPU 1422 may also execute method steps of any specific implementation of the text and image generation method in the embodiments of the present application.

[0277] See also Fig.10 , which is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application. Taking the terminal device as a smart phone as an example, Fig.10 The block diagram of the partial structure of the smart phone is shown, and the smart phone includes: a radio frequency (RF) circuit 1510, a memory 1520, an input unit 1530, a display unit 1540, a sensor 1550, an audio circuit 1560, a wireless fidelity (WiFi) module 1570, a processor 1580, and a power supply 1590. Those skilled in the art can understand that Fig.10 The structure of the smartphone shown in the figure does not constitute a limitation of the smartphone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0278] Combine the following Fig.10 A detailed introduction to the various components of a smartphone:

[0279] The RF circuit 1510 may be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information from the base station, it is sent to the processor 1580 for processing; in addition, the designed uplink data is sent to the base station.

[0280] The memory 1520 may be used to store software programs and modules. The processor 1580 implements various functional applications and data processing of the smartphone by running the software programs and modules stored in the memory 1520 .

[0281] The input unit 1530 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the smartphone. Specifically, the input unit 1530 may include a touch panel 1531 and other input devices 1532. The touch panel 1531, also known as a touch screen, can collect user touch operations on or near it and drive the corresponding connection device according to a pre-set program. In addition to the touch panel 1531, the input unit 1530 may also include other input devices 1532. Specifically, other input devices 1532 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0282] The display unit 1540 may be used to display information input by the user or information provided to the user and various menus of the smartphone. The display unit 1540 may include a display panel 1541, and the display panel 1541 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0283] The smartphone may also include at least one sensor 1550, such as a light sensor, a motion sensor, and other sensors. As for other sensors that may be configured in the smartphone, such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., they will not be described in detail here.

[0284] The audio circuit 1560, the speaker 1561, and the microphone 1562 can provide an audio interface between the user and the smartphone. The audio circuit 1560 can transmit the received audio data to the speaker 1561 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1562 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1560 and converted into audio data, and then the audio data is output to the processor 1580 for processing, and then sent to another smartphone through the RF circuit 1510, or the audio data is output to the memory 1520 for further processing.

[0285] The processor 1580 is the control center of the smartphone, and uses various interfaces and lines to connect various parts of the entire smartphone, and executes various functions of the smartphone and processes data by running or executing software programs and / or modules stored in the memory 1520, and calling data stored in the memory 1520. Optionally, the processor 1580 may include one or more processing units.

[0286] The smart phone also includes a power supply 1590 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1580 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption management.

[0287] Although not shown, the smartphone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0288] In the embodiment of the present application, the memory 1520 included in the smart phone can store a computer program and transmit the computer program to the processor.

[0289] The processor 1580 included in the smart phone can execute the text and picture generation method provided in the above embodiment according to the instructions in the computer program.

[0290] The embodiment of the present application also provides a computer-readable storage medium for storing a computer program, which is used to execute the text and image generation method provided in the above embodiment.

[0291] The embodiment of the present application also provides a computer program product including a computer program, which, when executed on a computer device, enables the computer device to execute the method steps of any specific implementation of the text and image generation method in the embodiment of the present application.

[0292] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the above program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the above storage medium can be at least one of the following media: read-only memory (English: Read-Only Memory, abbreviated: ROM), RAM, magnetic disk or optical disk, etc. Various media that can store computer programs.

[0293] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0294] It should be noted that each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, in which the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0295] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a technician familiar with the technical field within the technical scope disclosed in the present application should be included in the protection scope of the present application. Based on the implementation methods provided in the above aspects, the present application can also be further combined to provide more implementation methods. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for generating text and picture, characterized in that: The method comprises: Acquire virtual scene data and virtual character data for the text segment, wherein the virtual scene data is used to describe scene features of the virtual scene corresponding to the text segment, and the virtual character data is used to describe character features of the virtual character included in the virtual scene; According to the virtual scene data, a target storyboard is obtained from a plurality of storyboards, the popularity of the virtual scene described by the storyboard is greater than a preset popularity threshold, and the similarity between the virtual scene described by the target storyboard and the virtual scene described by the virtual scene data is greater than a first similarity threshold; According to the virtual character data, a fixed role model corresponding to the virtual character included in the virtual character data is obtained, where different virtual characters correspond to different fixed role models, and the fixed role model is used to control the difference of the same virtual character in different virtual scenes to be less than a preset difference threshold; The character fixed model and the image generation model are integrated to obtain a picture generation model; According to the virtual character data and the target storyboard, a text illustration for the text segment is generated by the illustration generation model.

2. The method according to claim 1, characterized in that The step of obtaining a target storyboard from a plurality of storyboards according to the virtual scene data comprises: According to the virtual scene data, feature extraction is performed by a multimodal feature extractor to obtain scene features of the virtual scene data; According to the plurality of storyboards, feature extraction is performed respectively by the multimodal feature extractor to obtain storyboard image features corresponding to the plurality of storyboards respectively; Determine the similarity between the scene feature and the plurality of storyboard image features, and use the storyboard image corresponding to the storyboard image feature having a similarity greater than the first similarity threshold as the target storyboard image.

3. The method according to claim 2, characterized in that If the target storyboard includes a first storyboard and a second storyboard, then generating a text illustration for the text segment by using the illustration generation model according to the virtual character data and the target storyboard includes: Generate a first text illustration for the text segment by using the illustration generation model according to the virtual character data and the first storyboard; Generate a second text illustration for the text segment by using the illustration generation model according to the virtual character data and the second storyboard; If a selection operation for the first text image is obtained, the first text image is used as the text image of the text segment; If a selection operation for the second text image is obtained, the second text image is used as the text image of the text segment.

4. The method according to claim 2, characterized in that: If the target storyboard includes a first storyboard and a second storyboard, then generating a text illustration for the text segment by using the illustration generation model according to the virtual character data and the target storyboard includes: Generate a first text illustration for the text segment by using the illustration generation model according to the virtual character data and the first storyboard; Generate a second text illustration for the text segment by using the illustration generation model according to the virtual character data and the second storyboard; According to the text segment, feature extraction is performed by the multimodal feature extractor to obtain text features of the text segment; According to the first text image, feature extraction is performed by the multimodal feature extractor to obtain a first image feature of the first text image; According to the second text image, feature extraction is performed by the multimodal feature extractor to obtain a second image feature of the second text image; Determine the similarity between the first image feature and the text feature, and determine the similarity between the second image feature and the text feature, and use the text image corresponding to the image feature with higher similarity as the text image for the text segment.

5. The method according to claim 1, characterized in that If the text segment is a part of the target text content, the method further includes: Acquire multiple virtual characters included in the target text content and character features corresponding to the multiple virtual characters respectively; For a target virtual character among the multiple virtual characters, determining multi-perspective character data for the target virtual character according to the character characteristics of the target virtual character, wherein the multi-perspective character data is used to describe the character characteristics of the target virtual character at different perspectives; According to the multi-view character data, a predicted text image is obtained by using an initial image generation model, wherein the initial image generation model is obtained by fusing an initial character fixed model and the image generation model; According to the difference between the predicted text picture and the target virtual character, the model parameters of the image generation model in the initial picture generation model are fixed, and the model parameters of the initial character fixed model in the initial picture generation model are adjusted to obtain a character fixed model suitable for the target virtual character; The multiple virtual characters are respectively determined as the target virtual characters, and the fixed character models corresponding to the multiple virtual characters are obtained.

6. The method according to claim 1, characterized in that If the virtual scene includes a plurality of virtual characters, the step of obtaining the virtual scene data and the virtual character data for the text segment includes: Acquire virtual scene data and multiple virtual character data for the text segment; The step of obtaining, according to the virtual character data, a fixed role model corresponding to the virtual character included in the virtual character data comprises: According to the plurality of virtual character data, obtaining fixed role models corresponding to the virtual characters included in the plurality of virtual character data respectively; The step of fusing the character fixed model and the image generation model to obtain the image generation model includes: The character fixed models corresponding to the multiple virtual characters and the image generation model are fused to obtain the image generation model.

7. The method according to claim 6, characterized in that The method further comprises: Get the difference-adjusted model; According to the text illustration for the text fragment, character features of multiple virtual characters included in the text illustration are identified through the difference adjustment model. If the similarity of the multiple character features is greater than a second similarity threshold, the virtual characters corresponding to the character features whose similarity is greater than the second similarity threshold are fine-tuned to obtain an updated text illustration.

8. The method according to claim 1, characterized in that The method further comprises: Acquire target text content, a first character prompt word, a first scene prompt word, and an associated prompt word, wherein the target text content includes a plurality of the text segments, the first scene prompt word is a question sentence for acquiring a virtual character and the character features of the virtual character, the first scene prompt word is a question sentence for acquiring the virtual scene data, and the associated prompt word is a question sentence for associating the virtual scene with the virtual character; According to the first role prompt word and the target text content, obtaining the virtual role included in the target text content and the role characteristics of the virtual role through a first text question answering model; According to the first scene prompt word and the target text content, obtaining multiple virtual scenes included in the target text content and virtual scene data corresponding to each of the virtual scenes through the first text question-answering model; According to the role characteristics of the virtual character, the virtual scene data, the associated prompt words and the target text content, the virtual characters included in each of the virtual scenes are obtained through the first text question-answering model, and the virtual character data includes the virtual characters included in each of the virtual scenes and the role characteristics of the virtual characters.

9. The method according to claim 1, characterized in that: The method further comprises: Acquire target text content, a second role prompt word and a second scene prompt word, wherein the second role prompt word is a question sentence for acquiring the virtual role data, and the second scene prompt word is a question sentence for acquiring the virtual scene data; According to the target text content, the second role prompt word and the second scene prompt word, the virtual scene data and the virtual role data for the target text content are obtained through a second text question-answering model.

10. The method according to claim 8 or 9, characterized in that: If the text question answering model also has a correction function, the method further includes: Obtaining a self-reflection prompt word, wherein the self-reflection prompt word is a question sentence used to verify the answer output by the text question answering model; According to the virtual scene data, the virtual character data and the introspection prompt words, verification is performed through the text question-answering model to obtain updated virtual scene data and updated virtual character data.

11. The method according to claim 1, characterized in that: The step of generating a text illustration for the text segment by using the illustration generation model according to the virtual character data and the target storyboard includes: According to the virtual character data, the virtual scene data and the target storyboard, a text illustration for the text segment is generated by the illustration generation model.

12. A text and picture generation device, characterized in that: The device comprises: an acquisition unit, a selection unit, a fusion unit and a generation unit; The acquisition unit is used to acquire virtual scene data and virtual character data for the text segment, wherein the virtual scene data is used to describe scene features of the virtual scene corresponding to the text segment, and the virtual character data is used to describe character features of the virtual character included in the virtual scene; The selection unit is used to obtain a target storyboard from multiple storyboards according to the virtual scene data, the popularity of the virtual scene described by the storyboard is greater than a preset popularity threshold, and the similarity between the virtual scene described by the target storyboard and the virtual scene described by the virtual scene data is greater than a first similarity threshold; The selection unit is further used to obtain, based on the virtual character data, a fixed role model corresponding to the virtual character included in the virtual character data, wherein different virtual characters correspond to different fixed role models, and the fixed role model is used to control the difference of the same virtual character in different virtual scenes to be less than a preset difference threshold; The fusion unit is used to fuse the character fixed model and the image generation model to obtain a picture generation model; The generating unit is used to generate a text illustration for the text segment through the illustration generation model according to the virtual character data and the target storyboard.

13. A computer device, characterized in that: The computer device comprises a processor and a memory: The memory is used to store a computer program and transmit the computer program to the processor; The processor is configured to execute the method according to any one of claims 1 to 11 according to the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the method is executed on a computer device, the computer device is enabled to execute the method according to any one of claims 1 to 11.

Images