Continuous role image picture generation method and device, equipment and storage medium

Abstract

The invention relates to a continuous role image picture generation method and device, equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed target text and an initial role image picture of a role, carrying out the analysis of the target text, and obtaining the plot description information of each plot structure in a story plot; according to the initial role image picture and the plot description information of each plot structure, generating a role image picture corresponding to each plot structure, and obtaining a continuous role image picture set corresponding to the target text. According to the mode, the consistency of the generated role image can be kept, the personalized continuous role picture set which is matched with the context scene and can reflect scene change and emotion and action change of the role is generated, the expression and action of the role do not need to be manually preset, and the user experience is improved. The emotion information of the scene and the role in the role image picture is dynamically adjusted according to the story plot.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a method, device, equipment and storage medium for generating continuous character image pictures. Background Art

[0002] Computer vision is an important branch of artificial intelligence, focusing on enabling computers to understand and process visual information in the human world. Image generation technology is a key area in computer vision, including virtual character generation technology, which is mainly used in character generation in game development, special effects and character generation in film and animation production, etc.

[0003] Although existing virtual character generation technologies can combine natural language processing to generate the appearance of characters, most of the existing virtual character generation technologies rely on manually configured scenes, expressions and actions, and the generated character images often cannot generate personalized continuous character images that match the context, have emotional expression capabilities and are consistent with the character image. Summary of the invention

[0004] The present application provides a method, device, equipment and storage medium for generating continuous character image images to solve the problem that existing virtual character generation technologies mostly rely on manually configured scenes, expressions and actions, and the generated character images are often unable to generate personalized continuous character images that match the context, have emotional expression capabilities and are consistent with the character image.

[0005] In a first aspect, the present application provides a method for generating continuous character image pictures, the method comprising:

[0006] Acquire a target text to be processed, wherein the target text has a storyline, and the storyline has at least one character;

[0007] Obtaining an initial character image of the character;

[0008] Parsing the target text to obtain plot description information of each plot structure in the storyline;

[0009] According to the initial character image diagram and the plot description information of each plot structure, a character image diagram corresponding to each plot structure is generated to obtain a continuous character image picture set corresponding to the target text.

[0010] In a possible implementation, the obtaining of the initial character image of the character includes:

[0011] Obtaining character image description information of the character;

[0012] The character image description information is input into a trained initial character image generation model to obtain an initial character image diagram of the character.

[0013] In a possible implementation, parsing the target text to obtain plot description information of each plot structure in the storyline includes:

[0014] Inputting the target text into a trained plot structure recognition model to obtain a plot structure recognition result output by the plot structure recognition model; wherein the plot structure recognition result includes each plot structure in the story and the text content corresponding to each plot structure;

[0015] The following processing is performed for each of the plot structures:

[0016] Identifying characters, character emotion information, and scene information contained in the plot structure from text content corresponding to the plot structure;

[0017] The plot description information of the plot structure is generated according to the emotion information and scene information of the characters contained in the plot structure.

[0018] In a possible implementation, generating the plot description information of the plot structure according to the emotion information and scene information of the characters contained in the plot structure includes:

[0019] Determining the scene information of the scene in the plot structure according to the emotional information of the character contained in the plot structure;

[0020] The plot description information of the plot structure is generated according to the characters, the emotional information of the characters, the scene information and the scene information contained in the plot structure.

[0021] In a possible implementation manner, determining the scene information of the scene in the plot structure according to the emotion information of the character contained in the plot structure includes:

[0022] Determining the intensity of the emotion of the character included in the plot structure according to the emotion information of the character included in the plot structure;

[0023] In a case where the emotional intensity of the character included in the plot structure indicates intense emotion, determining the scene information of the scene in the plot structure as a first scene;

[0024] In a case where the emotion intensity of the character included in the plot structure indicates that the emotion is not intense, the scene information of the scene in the plot structure is determined as the second scene.

[0025] In a possible implementation, generating a character image corresponding to each plot structure according to the initial character image and the plot description information of each plot structure includes:

[0026] The following processing is performed for each of the plot structures:

[0027] The plot description information of the plot structure and the initial character image are input into a trained consistent character generation model to obtain a character image corresponding to the plot structure output by the consistent character generation model.

[0028] In a possible implementation, the consistent role generation model adopts a self-attention mechanism; the step of inputting the plot description information of the plot structure and the initial role image into the trained consistent role generation model to obtain the role image corresponding to the plot structure output by the consistent role generation model includes:

[0029] For a first plot structure in the storyline, input the plot description information of the first plot structure and the initial character image into a trained consistent character generation model, obtain the character image corresponding to the first plot structure output by the consistent character generation model, and save the key vector and value vector generated in the process of generating the character image corresponding to the first plot structure;

[0030] For the non-first plot structure in the storyline, the plot description information of the non-first plot structure and the initial character image diagram are input into the consistent character generation model to obtain the character image diagram corresponding to the non-first plot structure output by the consistent character generation model using the saved key vector and value vector.

[0031] In a second aspect, the present application provides a device for generating continuous character image pictures, the device comprising:

[0032] A text acquisition module, used for acquiring a target text to be processed, wherein the target text has a storyline, and the storyline has at least one character;

[0033] A role acquisition module, used to acquire an initial role image of the role;

[0034] A text parsing module, used to parse the target text to obtain plot description information of each plot structure in the storyline;

[0035] The picture generation module is used to generate a character image corresponding to each plot structure according to the initial character image and the plot description information of each plot structure, so as to obtain a continuous character image picture set corresponding to the target text.

[0036] In a possible implementation, the role acquisition module includes:

[0037] A role acquisition unit, used to acquire role image description information of the role;

[0038] The character generation unit is used to input the character image description information into the trained initial character image generation model to obtain the initial character image diagram of the character.

[0039] In a possible implementation, the text parsing module includes:

[0040] A text parsing unit, used to input the target text into a trained plot structure recognition model to obtain a plot structure recognition result output by the plot structure recognition model; wherein the plot structure recognition result includes each plot structure in the story and the text content corresponding to each plot structure;

[0041] An information acquisition unit is used to perform the following processing for each of the plot structures: identifying the characters, the emotional information of the characters and the scene information contained in the plot structure from the text content corresponding to the plot structure;

[0042] The information generating unit is used to perform the following processing for each of the plot structures: generating plot description information of the plot structure according to the emotion information and scene information of the characters contained in the plot structure.

[0043] In a possible implementation, the information generating unit includes:

[0044] A scene information determination subunit, used to determine the scene information of the scene in the plot structure according to the emotional information of the character contained in the plot structure;

[0045] The plot description information subunit is used to generate the plot description information of the plot structure according to the characters, the emotional information of the characters, the scene information and the scene information contained in the plot structure.

[0046] In a possible implementation manner, the scene information determination subunit is specifically used to:

[0047] Determining the intensity of the emotion of the character included in the plot structure according to the emotion information of the character included in the plot structure;

[0048] In a case where the emotional intensity of the character included in the plot structure indicates intense emotion, determining the scene information of the scene in the plot structure as a first scene;

[0049] In a case where the emotion intensity of the character included in the plot structure indicates that the emotion is not intense, the scene information of the scene in the plot structure is determined as the second scene.

[0050] In a possible implementation, the picture generation module includes:

[0051] The character image generation unit is used to perform the following processing for each of the plot structures: input the plot description information of the plot structure and the initial character image diagram into the trained consistent character generation model to obtain the character image diagram corresponding to the plot structure output by the consistent character generation model.

[0052] In a possible implementation, the consistent character generation model adopts a self-attention mechanism; the character image generation unit is specifically used to:

[0053] For a first plot structure in the storyline, input the plot description information of the first plot structure and the initial character image into a trained consistent character generation model, obtain the character image corresponding to the first plot structure output by the consistent character generation model, and save the key vector and value vector generated in the process of generating the character image corresponding to the first plot structure;

[0054] For the non-first plot structure in the storyline, the plot description information of the non-first plot structure and the initial character image diagram are input into the consistent character generation model to obtain the character image diagram corresponding to the non-first plot structure output by the consistent character generation model using the saved key vector and value vector.

[0055] In a third aspect, the present application provides a device comprising: a processor and a memory, wherein the processor is used to execute a program for generating continuous character image pictures stored in the memory to implement the method for generating continuous character image pictures described in any one of the first aspects.

[0056] In a fourth aspect, the present application provides a storage medium storing one or more programs, which can be executed by one or more processors to implement the method for generating continuous character image pictures as described in any one of the first aspects.

[0057] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application obtains the target text to be processed and the initial character image of the character, parses the target text, and obtains the plot description information of each plot structure in the storyline. Based on the initial character image and the plot description information of each plot structure, a character image corresponding to each plot structure is generated to obtain a continuous character image picture set corresponding to the target text. This method can maintain the consistency of the generated character image, generate a personalized continuous character image set that matches the context and can reflect the scene changes and the character's emotions and action changes, without the need to manually pre-set the character's expressions and actions, and realize the dynamic adjustment of the scene and the character's emotional information in the character image picture according to the storyline. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0059] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0060] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0061] Figure 1 A flow chart of an embodiment of a method for generating continuous character image images provided in an embodiment of the present application.

[0062] Figure 2 This is an example diagram of the target text involved in the embodiments of the present application.

[0063] Figure 3 This is an example diagram of a set of continuous character image pictures generated by an embodiment of the present application.

[0064] Figure 4 A flow chart of another embodiment of a method for generating continuous character image images provided in an embodiment of the present application.

[0065] Figure 5 A flow chart of another embodiment of a method for generating continuous character image images provided in an embodiment of the present application.

[0066] Figure 6A diagram of a device for generating continuous character image pictures provided in an embodiment of the present application.

[0067] Figure 7 A schematic diagram of a device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0069] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0070] In order to solve the technical problem that most of the virtual character generation technologies in the prior art rely on manually configured scenes, expressions and actions, and the generated character images are often unable to generate personalized continuous character images that match the context, have emotional expression capabilities and are consistent with the character image, the present application provides a method, device, equipment and storage medium for generating continuous character image images, which can maintain the consistency of the generated character image, generate a personalized continuous character image set that matches the context scenario and can reflect scene changes and the character's emotions and action changes, without the need to manually pre-set the character's expressions and actions, and realizes dynamic adjustment of the scene and character's emotional information in the character image image according to the storyline.

[0071] Figure 1 The flowchart of the embodiment of the method for generating a continuous character image is provided in the embodiment of the present application. In one embodiment, the method is applied to a device. When the device is hardware, it can be a variety of electronic devices with a display screen, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, servers, etc. When the device is software, it can be installed in the electronic devices listed above. Figure 1 As shown, the process includes the following steps:

[0072] Step 101: Obtain a target text to be processed, wherein the target text has a storyline, and the storyline has at least one character.

[0073] The storyline includes the basic framework and development context of the story. In a storyline, there are usually one or more characters. From the perspective of character category, the characters can be human characters or animal characters. From the perspective of character authenticity, the characters can be real characters or virtual characters.

[0074] The target text to be processed may be a text description content containing one or more story plots.

[0075] In an exemplary application scenario, the target text is a text description content of a story or narrative compiled by a creator (such as a literary creator or a self-media creator).

[0076] In this exemplary application scenario, as an optional implementation, the specific implementation of obtaining the target text to be processed includes: obtaining the text description content of the storyline input or uploaded by the user, and determining the text description content as the target text to be processed.

[0077] In another exemplary application scenario, the target text is a text describing a conversation between at least two characters. Figure 2 , is an example of the target text.

[0078] In this exemplary application scenario, as an optional implementation method, the specific implementation of obtaining the target text to be processed includes: obtaining the character dialogue description text input or uploaded by the user, and determining the character dialogue description text as the target text to be processed.

[0079] As another optional implementation method, the application scenario is an online chat interaction scenario, such as an intelligent robot chat scenario, a virtual role-playing game scenario, an online consultation scenario, etc. In such scenarios, the conversation records within the historical time period can be obtained, and the text content of the conversation records can be extracted as the target text to be processed. The above historical time period can be the last half an hour or 10 minutes, or a specified historical time period, which is not limited in the embodiments of the present application.

[0080] The above exemplary scenarios are merely exemplary descriptions of scenarios to which the technical solutions provided in this application can be applied. In actual applications, the technical solutions provided in this application can also be applied to other scenarios, such as multimedia applications, image generation, etc. The embodiments of this application do not limit the specific application scenarios and the specific implementation methods of how to obtain the target text in various scenarios.

[0081] Step 102: Obtain an initial character image of the character.

[0082] Character image refers to the shaping and performance of a specific character.

[0083] The initial character image is mainly used to show the basic external features of the character (such as facial features, body features, clothing features, etc.). The initial character image has little to do with the character's current emotions, scenes, and situations.

[0084] In one embodiment, the specific implementation of obtaining the initial character image of the character includes: obtaining character image description information of the character, inputting the character image description information into a trained initial character image generation model, and obtaining the initial character image of the character.

[0085] For example, for Figure 2 The target text shown involves two characters: Shi Jianshen and Shi Mu. The character image description information of Shi Jianshen is: {young and handsome racing driver, superb skills, confident and sunny personality, charming and romantic}. The character image description information of Shi Mu is: {Shi Jianshen's girlfriend, lively and cute personality, admires Shi Jianshen, easily shy but also brave to express love}. The character image description information of the two characters is respectively input into the trained initial character image generation model to obtain the initial character image graphs of the two characters.

[0086] As an optional implementation method, the character image description information of the character is customized by the user.

[0087] As another optional implementation method, other technical means may be used, such as using a role image analysis model to extract role image description information of the roles involved in the target text from the target text.

[0088] In another embodiment, the user is allowed to select a favorite character image as the initial character image.

[0089] In addition, the initial character image of the character may also be obtained by other means, which is not limited in the embodiment of the present application.

[0090] Step 103: Parse the target text to obtain plot description information of each plot structure in the storyline.

[0091] The plot structure of a storyline refers to the arrangement and unfolding of a series of events or plots that occur in a story. It constitutes the skeleton of the story, allowing the story to unfold and develop in an orderly manner. Generally speaking, the plot structure of a storyline mainly includes four parts: beginning, middle, transition, and ending, and sometimes also includes a prologue and an epilogue. Among them, a complete storyline can include multiple middle or multiple transition parts, which are not limited in the embodiments of the present application.

[0092] The plot description information of the plot structure includes, but is not limited to: the characters contained in the plot structure, the emotional information of the characters, and the scene information.

[0093] by Figure 2 Taking the target text shown in FIG. 1 as an example, by parsing the target text, the plot structure and plot description information as shown in Table 1 below can be obtained.

[0094] Table 1

[0095]

[0096]

[0097] As for how to parse the target text and obtain the plot description information of each plot structure in the story, we will explain it in the following text. Figure 4 The illustrated embodiment is described in detail and will not be described in detail here.

[0098] Step 104: Generate a character image diagram corresponding to each plot structure based on the initial character image diagram and the plot description information of each plot structure, and obtain a continuous character image picture set corresponding to the target text.

[0099] From the description in the above steps, we can know that the initial character image diagram provides the basic image of the character, and the plot description information of each plot structure in the storyline includes the character's behavior, emotional changes, and environmental changes in the plot structure. Then, by combining the initial character image diagram and the plot description information of each plot structure, we can generate the character image diagram corresponding to each plot structure. The character image diagram corresponding to the plot structure not only provides the basic image of the character, but also includes the character's behavior, emotional changes, and environmental changes. Furthermore, by integrating the character image diagrams corresponding to all plot structures in sequence, we can form a continuous character image picture set that tells the whole story.

[0100] Here, since the character image pictures corresponding to all plot structures are generated based on the same initial character image picture, the character images in the generated character image picture set are highly consistent.

[0101] For example, see Figure 3 , is an example of a set of continuous character image pictures generated by an embodiment of the present application. Figure 3 It can be seen from the example pictures shown that the character images in the 5 character image pictures are highly consistent, and the 5 pictures correspond to the 5 plot structures in the entire storyline respectively. The information presented in each character image picture matches the real situation of the corresponding plot structure.

[0102] The technical solution provided in the embodiment of the present application obtains the target text to be processed and the initial character image of the character, parses the target text, and obtains the plot description information of each plot structure in the storyline. Based on the initial character image and the plot description information of each plot structure, a character image corresponding to each plot structure is generated to obtain a continuous character image picture set corresponding to the target text. This method can maintain the consistency of the generated character image, generate a personalized continuous character image set that matches the context and can reflect the scene changes and the character's emotions and action changes, without the need to manually pre-set the character's expression and action, and realize the dynamic adjustment of the scene and the character's emotional information in the character image according to the storyline.

[0103] See also Figure 4 , which is a flow chart of another embodiment of a method for generating continuous character image images provided in an embodiment of the present application. Figure 4 The process shown in Figure 1 Based on the process shown, the following mainly describes how to obtain the plot description information of each plot structure in the storyline, including the following steps:

[0104] Step 401: Obtain a target text to be processed, wherein the target text has a storyline, and the storyline has at least one character.

[0105] Step 402: Obtain an initial character image of the character.

[0106] For detailed description of steps 401 to 402, see Figure 1 The description of the process shown will not be repeated here.

[0107] Step 403: input the target text into the trained plot structure recognition model to obtain the plot structure recognition result output by the plot structure recognition model; wherein the plot structure recognition result includes each plot structure in the story and the text content corresponding to each plot structure.

[0108] Step 404: Identify the characters, the characters' emotional information, and the scene information contained in the plot structure from the text content corresponding to the plot structure.

[0109] The following is a unified description of step 403 and step 404:

[0110] The plot structure recognition model can divide the story text content into different text content intervals, each text content interval corresponds to a plot structure, and the text content interval corresponding to each plot structure can be regarded as the plot description content of the plot structure.

[0111] In one embodiment, the target text is input into a trained plot structure recognition model to obtain a plot structure recognition result output by the plot structure recognition model, which includes the text content corresponding to each plot structure of the plot structure, namely, the plot description content described above.

[0112] by Figure 2 For example, by inputting the target text into the trained plot structure recognition model, the following plot structure recognition results can be obtained (only some recognition results are shown as examples):

[0113] Plot structure: Beginning, the corresponding text content is:

[0114] {Shi Mu: Jian Shen! Congratulations on getting first place again!

[0115] Shi Jianshen: Mumu! Thanks to you for cheering me on, I was able to perform so well.

[0116] Shi Mu: No, no, you are the best. I was watching you racing in the audience just now, you looked so cool!}

[0117] It can be seen from the above examples that the text content corresponding to the plot structure contains the characters involved in the plot structure, the emotional information of the characters, and the scene information. Therefore, in one embodiment, the characters, the emotional information of the characters, and the scene information contained in the plot structure are identified from the text content corresponding to the plot structure.

[0118] Exemplarily, the text content corresponding to the plot structure is input into the trained emotion recognition model and scene recognition model respectively to obtain the plot description information corresponding to each plot structure in the story.

[0119] Taking the text content corresponding to the plot structure of the above example as an example, the following information is identified from the text content corresponding to the plot structure:

[0120] Role: {Tokimizuki}.

[0121] Character's emotional message: {joy}.

[0122] Scene information: {racing scene, first place}.

[0123] Step 405: Generate plot description information of the plot structure according to the emotional information of the characters and the scene information contained in the plot structure.

[0124] In one embodiment, the emotional information and scene information of the characters contained in the plot structure are used as the plot description information of the plot structure.

[0125] In another embodiment, the specific implementation of generating plot description information of a plot structure based on the emotional information of the characters and the scene information contained in the plot structure includes: determining the shot information of the scene in the plot structure based on the emotional information of the characters contained in the plot structure; generating plot description information of the plot structure based on the characters contained in the plot structure, the emotional information of the characters, the scene information and the shot information of the scene.

[0126] In this embodiment, the character's emotion information, scene information, and scene information contained in the plot structure are used as plot description information.

[0127] Among them, the field of view refers to the difference in the size of the subject presented in the camera recorder due to the different distances between the camera and the subject. The field of view can generally be divided into five types, from near to far, namely close-up, close-up, medium shot, panoramic shot, and long shot. Through the use of different field of view, a specific emotional atmosphere can be created, the layers and depth of the story can be displayed, and at the same time, the artistic appeal can be enhanced and the audience's attention can be guided.

[0128] As an optional implementation, the specific implementation of determining the scene information of the plot structure according to the emotion information of the character contained in the plot structure includes:

[0129] According to the emotional information of the characters included in the plot structure, the emotional intensity of the characters included in the plot structure is determined. When the emotional intensity of the characters included in the plot structure indicates that the emotions are intense, the scene information of the scene in the plot structure is determined as the first scene. The first scene is, for example, a close-up or a close-up; when the emotional intensity of the characters included in the plot structure indicates that the emotions are not intense, the scene information of the scene in the plot structure is determined as the second scene, and the second scene is, for example, a distant shot or a medium shot.

[0130] For example, see Figure 3 The five character image picture sets shown are of different shot sizes, wherein the first character image picture contains the character's emotion intensity as non-intense emotion, and the shot size information is medium shot, and the second character image picture contains the character's emotion intensity as intense emotion, and the shot size information is close shot.

[0131] As an optional implementation, when determining the intensity of the emotion of the character contained in the plot structure according to the emotion information of the character contained in the plot structure, a preset emotion classification information table is obtained, and the corresponding emotion classification information table is searched according to the emotion information of the character contained in the plot structure to determine the intensity of the emotion of the character contained in the plot structure. For example, the following Table 2 is an example of the emotion classification information table mentioned above:

[0132] Table 2

[0133] Strong emotions Non-violent emotions excited curious Excited joy Stimulate happy ... ...

[0134] By looking up the above Table 2, if it is determined that the emotional information of the character in the plot structure is intense emotion, then the scene information in the plot structure is determined to be a close shot or a close-up; if it is determined that the emotional information of the character in the plot structure is not intense emotion, then the scene information in the plot structure is determined to be a distant shot or a medium shot.

[0135] For example, a plot structure in the target text includes the following characters, character emotion information, and scene information at the beginning: {beginning: deep feeling, joy, racing scene, first place}. By looking up Table 2 above, it can be seen that joy is non-intense emotion information, which means that the scene information of the plot structure is long shot or medium shot.

[0136] Step 406: Generate a character image diagram corresponding to each plot structure based on the initial character image diagram and the plot description information of each plot structure, and obtain a continuous character image picture set corresponding to the target text.

[0137] For a detailed description of step 406, see Figure 1 The relevant descriptions in the process shown will not be repeated here.

[0138] pass Figure 4 The illustrated embodiment parses the target text to obtain plot description information for each plot structure in the storyline, thereby realizing dynamic identification of the plot structure and plot description information in the storyline, and facilitating the subsequent automatic generation of a character image picture set that matches the plot structure scenes, characters, and character emotional information of the storyline based on the model. There is no need for manual identification and input of the plot structure and the plot description information corresponding to the plot structure, which is more intelligent and reduces labor costs.

[0139] See also Figure 5 , which is a flow chart of an embodiment of another method for generating continuous character image pictures provided in an embodiment of the present application. Figure 5 The process shown in Figure 1 Based on the process shown, this paper mainly describes how to generate the character image diagram corresponding to each plot structure, including the following steps:

[0140] Step 501: Obtain a target text to be processed, wherein the target text has a storyline, and the storyline has at least one character.

[0141] Step 502: Obtain an initial character image of the character.

[0142] Step 503: parse the target text to obtain plot description information of each plot structure in the storyline.

[0143] For detailed description of steps 501 to 503, see Figure 1 The description of the process shown will not be repeated here.

[0144] Step 504: Input the plot description information of the plot structure and the initial character image diagram into the trained consistent character generation model to obtain the character image diagram corresponding to the plot structure output by the consistent character generation model.

[0145] In one embodiment, the above-mentioned consistent character generation model adopts a self-attention mechanism. The self-attention mechanism is a mechanism widely used in deep learning models, mainly used to process sequence data, such as text, speech, etc. The present application utilizes the processing ability of the self-attention mechanism on text, and finally generates a character image corresponding to the plot structure according to the plot description information of the plot structure, so that the generated character image can automatically generate a character image that matches the scene and emotional information of the plot structure according to the dynamic changes of the plot structure description information of the story.

[0146] The consistent role generation model in the technical solution provided in the embodiment of the present application improves the self-attention mechanism, and the specific improvements are as follows:

[0147] For the first plot structure in the story, the plot description information and the initial character image of the first plot structure are input into the trained consistent character generation model to obtain the character image corresponding to the first plot structure output by the consistent character generation model, and the key vector and value vector generated in the process of generating the character image corresponding to the first plot structure are saved. For the non-first plot structure in the story, the plot description information and the initial character image of the non-first plot structure are input into the consistent character generation model to obtain the character image corresponding to the non-first plot structure output by the consistent character generation model using the saved key vector and value vector.

[0148] In this embodiment, as a possible implementation method, the plot description information and the initial character image of the first plot structure are input into a trained consistent character generation model to generate a Q1 vector set, a k1 vector set, and a v1 vector set corresponding to the plot description information of the first plot structure. The plot description information and the initial character image of the non-first plot structure are input into the consistent character generation model to generate a Q2 vector set, and based on the k1 vector set, the v1 vector set and the Q2 vector set, a character image corresponding to the non-first plot structure is generated. By continuing to use the first key vector and value vector, the embodiment of the present application can not only improve efficiency and save computing resources, but also further improve consistency.

[0149] Based on the description of the above steps, exemplarily, the plot description information of the first plot structure is: {Beginning: (Mid-shot) Tokimi Shin is very happy, he won the first place in the racing competition at the scene of the racing competition}. The plot description information of the first plot structure and the initial character image are input into the trained consistent character generation model, and the data information of the initial character image is saved in the form of a key vector and a value vector; for example, a key vector (Key, eyes), a value vector (Value, big, round and black); wherein each key vector corresponds to a value vector, that is, it is saved in the form of a key-value pair, for example, (eyes, big, round and black), and then combined with the plot description information corresponding to the first plot structure, the character image corresponding to the first plot structure output by the consistent character generation model is obtained, such as Figure 3 The first picture shown.

[0150] For the non-first plot structure of the story, after the plot description information and the initial character image of the non-first plot structure are input into the consistent character generation model, the key vector and value vector of the initial character image saved in the first plot structure processing process are directly obtained without further processing the initial character image. Then, the character image corresponding to the non-first plot structure is output in combination with the plot description information corresponding to the non-first plot structure. For example, see Figure 3 Second picture shown.

[0151] Through the above Figure 5 The illustrated embodiment utilizes a consistent character generation model to generate, based on plot description information, a personalized character image diagram that matches each plot structure in the target text and maintains a consistent character image. This can dynamically adjust the emotions and scene changes of the generated character image diagram according to changes in the storyline, without the need for manual input of the character's emotional information and scene information, thereby reducing labor costs and realizing the intelligent generation of a character image collection.

[0152] Figure 6 A device for generating continuous character image pictures provided in an embodiment of the present application, such as Figure 6 As shown, the device comprises:

[0153] A text acquisition module 61 is used to acquire a target text to be processed, wherein the target text has a storyline, and the storyline has at least one character;

[0154] A role acquisition module 62 is used to acquire an initial role image of the role;

[0155] A text parsing module 63 is used to parse the target text to obtain plot description information of each plot structure in the storyline;

[0156] The picture generation module 64 is used to generate a character image corresponding to each plot structure according to the initial character image and the plot description information of each plot structure, so as to obtain a continuous character image picture set corresponding to the target text.

[0157] In a possible implementation, the role acquisition module 62 includes:

[0158] A role acquisition unit, used to acquire role image description information of the role;

[0159] The character generation unit is used to input the character image description information into the trained initial character image generation model to obtain the initial character image diagram of the character.

[0160] In a possible implementation, the text parsing module 63 includes:

[0161] A text parsing unit, used to input the target text into a trained plot structure recognition model to obtain a plot structure recognition result output by the plot structure recognition model; wherein the plot structure recognition result includes each plot structure in the story and the text content corresponding to each plot structure;

[0162] An information acquisition unit is used to perform the following processing for each of the plot structures: identifying the characters, the emotional information of the characters and the scene information contained in the plot structure from the text content corresponding to the plot structure;

[0163] The information generating unit is used to perform the following processing for each of the plot structures: generating plot description information of the plot structure according to the emotion information and scene information of the characters contained in the plot structure.

[0164] In a possible implementation, the information generating unit includes:

[0165] A scene information determination subunit, used to determine the scene information of the scene in the plot structure according to the emotional information of the character contained in the plot structure;

[0166] The plot description information subunit is used to generate the plot description information of the plot structure according to the characters, the emotional information of the characters, the scene information and the scene information contained in the plot structure.

[0167] In a possible implementation manner, the scene information determination subunit is specifically used to:

[0168] Determining the intensity of the emotion of the character included in the plot structure according to the emotion information of the character included in the plot structure;

[0169] In a case where the emotional intensity of the character included in the plot structure indicates intense emotion, determining the scene information of the scene in the plot structure as a first scene;

[0170] In a case where the emotion intensity of the character included in the plot structure indicates that the emotion is not intense, the scene information of the scene in the plot structure is determined as the second scene.

[0171] In a possible implementation, the picture generation module 64 includes:

[0172] The character image generation unit is used to perform the following processing for each of the plot structures: input the plot description information of the plot structure and the initial character image diagram into the trained consistent character generation model to obtain the character image diagram corresponding to the plot structure output by the consistent character generation model.

[0173] In a possible implementation, the consistent character generation model adopts a self-attention mechanism; the character image generation unit is specifically used to:

[0174] For a first plot structure in the storyline, input the plot description information of the first plot structure and the initial character image into a trained consistent character generation model, obtain the character image corresponding to the first plot structure output by the consistent character generation model, and save the key vector and value vector generated in the process of generating the character image corresponding to the first plot structure;

[0175] For the non-first plot structure in the storyline, the plot description information of the non-first plot structure and the initial character image diagram are input into the consistent character generation model to obtain the character image diagram corresponding to the non-first plot structure output by the consistent character generation model using the saved key vector and value vector.

[0176] like Figure 7 As shown, an embodiment of the present application provides a device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0177] Memory 113, used for storing computer programs;

[0178] In one embodiment of the present application, the processor 111 is used to execute the program stored in the memory 113 to implement the method for generating continuous character image pictures provided by any one of the aforementioned method embodiments, including:

[0179] Acquire a target text to be processed, wherein the target text has a storyline, and the storyline has at least one character;

[0180] Obtaining an initial character image of the character;

[0181] Parsing the target text to obtain plot description information of each plot structure in the storyline;

[0182] According to the initial character image diagram and the plot description information of each plot structure, a character image diagram corresponding to each plot structure is generated to obtain a continuous character image picture set corresponding to the target text.

[0183] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method for generating continuous character image pictures provided in any of the aforementioned method embodiments are implemented.

[0184] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0185] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0186] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0187] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A method for generating continuous character image pictures, characterized in that: The method comprises: Acquire a target text to be processed, wherein the target text has a storyline, and the storyline has at least one character; Obtaining an initial character image of the character; Parsing the target text to obtain plot description information of each plot structure in the storyline; According to the initial character image diagram and the plot description information of each plot structure, a character image diagram corresponding to each plot structure is generated to obtain a continuous character image picture set corresponding to the target text.

2. The method according to claim 1, characterized in that The step of obtaining the initial character image of the character comprises: Obtaining character image description information of the character; The character image description information is input into a trained initial character image generation model to obtain an initial character image diagram of the character.

3. The method according to claim 1, characterized in that The target text is parsed to obtain plot description information of each plot structure in the storyline, including: Inputting the target text into a trained plot structure recognition model to obtain a plot structure recognition result output by the plot structure recognition model; wherein the plot structure recognition result includes each plot structure in the story and the text content corresponding to each plot structure; The following processing is performed for each of the plot structures: Identifying characters, character emotion information, and scene information contained in the plot structure from text content corresponding to the plot structure; The plot description information of the plot structure is generated according to the emotion information and scene information of the characters contained in the plot structure.

4. The method according to claim 3, characterized in that The step of generating plot description information of the plot structure according to the emotion information and scene information of the characters contained in the plot structure comprises: Determining the scene information of the scene in the plot structure according to the emotional information of the character contained in the plot structure; The plot description information of the plot structure is generated according to the characters, the emotional information of the characters, the scene information and the scene information contained in the plot structure.

5. The method according to claim 4, characterized in that Determining the scene information of the scene in the plot structure according to the emotion information of the character contained in the plot structure includes: Determining the intensity of the emotion of the character included in the plot structure according to the emotion information of the character included in the plot structure; In a case where the emotional intensity of the character included in the plot structure indicates intense emotion, determining the scene information of the scene in the plot structure as a first scene; In a case where the emotion intensity of the character included in the plot structure indicates that the emotion is not intense, the scene information of the scene in the plot structure is determined as the second scene.

6. The method according to claim 1, characterized in that The step of generating a character image corresponding to each plot structure according to the initial character image and the plot description information of each plot structure comprises: The following processing is performed for each of the plot structures: The plot description information of the plot structure and the initial character image diagram are input into a trained consistent character generation model to obtain a character image diagram corresponding to the plot structure output by the consistent character generation model.

7. The method according to claim 6, characterized in that The consistent role generation model adopts a self-attention mechanism; the plot description information of the plot structure and the initial role image are input into the trained consistent role generation model to obtain the role image corresponding to the plot structure output by the consistent role generation model, including: For a first plot structure in the storyline, input the plot description information of the first plot structure and the initial character image into a trained consistent character generation model, obtain the character image corresponding to the first plot structure output by the consistent character generation model, and save the key vector and value vector generated in the process of generating the character image corresponding to the first plot structure; For the non-first plot structure in the storyline, the plot description information of the non-first plot structure and the initial character image diagram are input into the consistent character generation model to obtain the character image diagram corresponding to the non-first plot structure output by the consistent character generation model using the saved key vector and value vector.

8. A device for generating continuous character image pictures, characterized in that: The device comprises: A text acquisition module, used for acquiring a target text to be processed, wherein the target text has a storyline, and the storyline has at least one character; A role acquisition module, used to acquire an initial role image of the role; A text parsing module, used to parse the target text to obtain plot description information of each plot structure in the storyline; The picture generation module is used to generate a character image corresponding to each plot structure according to the initial character image and the plot description information of each plot structure, so as to obtain a continuous character image picture set corresponding to the target text.

9. A device, characterized in that: include: A processor and a memory, wherein the processor is used to execute a program for generating continuous character image pictures stored in the memory to implement the method for generating continuous character image pictures according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method for generating continuous character image pictures according to any one of claims 1-7.

Images