A method and related device for obtaining frame sequence from text

Abstract

The present application discloses a method for obtaining a frame sequence from text and a related device, which relates to the field of text-image technology. Based on the input text, the frame description text of the image frame expressing the input text and the layout information of each frame are obtained, the text embedding of the input text is obtained, and based on the reference image of each character, the image embedding of the reference image is obtained. The image embedding is fused with the layout information to obtain first fusion information, and the text embedding is fused with the layout information to obtain second fusion information. Based on the spatiotemporal self-attention mechanism, the information is fused to obtain a frame sequence including image frames. Because the first fusion information represents the first positional relationship of each image embedded in each frame, the second fusion information represents the second positional relationship of each text embedded in each frame, and the layout information of the target frame indicates the spatial information of each character in the target frame, it is possible to reduce the possibility of different characters influencing each other in the frame and ensure the consistency of the characters between frames.

Description

Technical Field

[0001] The present application relates to the field of text-generated image technology, and in particular to a method for obtaining a frame sequence from text and a related device. Background Art

[0002] Story visualization aims to generate attractive images or videos for textual narratives. With the development of text-based visualization technology using models, it is now possible to generate images containing multiple characters based on text containing multiple characters.

[0003] But the current problem is that it is difficult for the same character to remain consistent between consecutive frames. For example, the little boy's hair is black in the previous frame and red in the next frame.

[0004] Therefore, how to generate multiple characters contained in the text in one frame of image while keeping the characters consistent between different image frames is a problem that needs to be solved at present. Summary of the Invention

[0005] In view of the above problems, this application provides a method and related device for obtaining a frame sequence from text, so as to achieve the purpose of generating multiple characters contained in the text in a single frame of image while maintaining the consistency of the characters between different image frames. The specific solution is as follows:

[0006] A first aspect of the present application provides a method for obtaining a frame sequence from text, comprising:

[0007] Based on input text, acquiring frame description text and layout information of each frame, wherein the frame includes an image frame for expressing the input text, and the layout information of the target frame indicates spatial information of each character in the target frame;

[0008] Obtaining a text embedding of the input text;

[0009] Obtaining an image embedding of each character's reference image based on the reference image, wherein the character is a character represented by the input text;

[0010] fusing the image embedding with the layout information to obtain first fusion information, wherein the first fusion information represents a first positional relationship between each image embedded in each frame;

[0011] fusing the text embeddings with the layout information to obtain second fused information, wherein the second fused information represents a second positional relationship between each text embedding in each frame;

[0012] Based on the spatiotemporal self-attention mechanism, information is fused to obtain a frame sequence including the image frames, and the fused information includes the first fused information and the second fused information.

[0013] In a possible implementation, obtaining the text embedding of the input text and the image embedding of the reference image includes:

[0014] Encoding the input text into the conditional space of the stable diffusion model to obtain a text embedding of the input text;

[0015] encoding the reference image into a conditional space of a stable diffusion model to obtain an image embedding of the reference image;

[0016] The text embedding and the image embedding are aligned.

[0017] In a possible implementation, before obtaining the image embedding of the reference image, the method further includes:

[0018] Information irrelevant to the character in the reference image is removed.

[0019] In a possible implementation, fusing the image embedding with the layout information to obtain first fusion information includes:

[0020] Perform a first operation on the hidden state of the attention mechanism and each image embedding to obtain the first parameter;

[0021] performing a second operation on the first parameter and a first mask matrix to obtain a hidden state update component of each character, wherein the first mask matrix is ​​obtained based on the layout information;

[0022] The hidden state update components of each character are fused to obtain the first fusion information.

[0023] In a possible implementation, fusing the embedded features of the text with the layout information to obtain second fused information includes:

[0024] Perform the first operation on the hidden state of the attention mechanism and each text embedding to obtain the second parameter;

[0025] performing a third operation on the second parameter, an area balance matrix, and a second mask matrix to obtain a third parameter, wherein the area balance matrix is ​​used to balance the areas occupied by each character in the target frame, and the second mask matrix is ​​obtained based on the layout information;

[0026] A fourth operation is performed on the third parameter and each text embedding to obtain the second fusion information.

[0027] In a possible implementation, the information to be fused further includes: information of a character extracted from an image including a single character.

[0028] In a possible implementation, obtaining the frame description text and the layout information of each frame based on the input text includes:

[0029] The input text is output as description text and layout information of a plurality of image frames based on a language model.

[0030] A second aspect of the present application provides a device for acquiring a frame sequence from text, comprising:

[0031] A first acquisition module is configured to acquire, based on input text, frame description text and layout information of each frame, wherein the frame includes an image frame for expressing the input text, and the layout information of the target frame indicates spatial information of each character in the target frame;

[0032] A second acquisition module, configured to acquire text embedding of the input text;

[0033] a third acquisition module, configured to acquire an image embedding of each character's reference image based on the reference image of the character, the character being the character represented by the input text;

[0034] a first fusion module, configured to fuse the image embedding with the layout information to obtain first fusion information, wherein the first fusion information represents a first positional relationship between the images embedded in each frame;

[0035] a second fusion module, configured to fuse the text embeddings with the layout information to obtain second fusion information, wherein the second fusion information represents a second positional relationship between the text embeddings in the frames;

[0036] The third fusion module is used to fuse information based on the spatiotemporal self-attention mechanism to obtain a frame sequence including the image frames, and the fused information includes the first fusion information and the second fusion information.

[0037] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the method for obtaining a frame sequence from text as described in the first aspect or any implementation of the first aspect.

[0038] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0039] The memory is used to store computer programs;

[0040] The processor is used to execute the computer program so that the electronic device can implement the method for obtaining a frame sequence from text as described in the first aspect or any implementation of the first aspect.

[0041] In a fifth aspect, the present application provides a computer-readable storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the method of obtaining a frame sequence from text as described in the first aspect or any implementation of the first aspect.

[0042] By means of the above technical solution, the present application provides a method and related apparatus for obtaining a frame sequence from text. Based on input text, frame description text of image frames expressing the input text and layout information of each frame are obtained, and a text embedding of the input text is obtained. Based on a reference image of each character, an image embedding of the reference image is obtained, where the character is the character described by the input text. The image embedding is fused with the layout information to obtain first fused information, and the text embedding is fused with the layout information to obtain second fused information. Based on a spatiotemporal self-attention mechanism, this information is fused to obtain a frame sequence including image frames. Therefore, when multiple characters exist, multiple characters can be included in a single image frame. Because the first fused information represents the first positional relationship between each image embedded in each frame, the second fused information represents the second positional relationship between each text embedded in each frame, and the layout information of the target frame indicates the spatial information of each character in the target frame, the spatial information of the characters is integrated into the information used to obtain the final frame sequence. This reduces the possibility of different characters influencing each other in the frame, thereby reducing the possibility of one character being influenced by other characters, which is beneficial for ensuring the consistency of any character between frames in the case of multiple characters. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0044] Figure 1 A flow chart of a method for obtaining a frame sequence from text provided by this application;

[0045] Figure 2 This is a layout example diagram embodying the layout information provided in this application;

[0046] Figure 3 An architecture diagram for obtaining frame sequences from text based on a model provided in this application;

[0047] Figure 4 This application provides a functional example of obtaining first fusion information based on a consistent image cross-attention model and obtaining second fusion information based on a consistent text cross-attention model;

[0048] Figure 5A schematic structural diagram of a device for obtaining a frame sequence from text provided by the present application;

[0049] Figure 6 This is a schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0050] The following describes the embodiments of the present application in conjunction with the accompanying drawings. The terms used in the implementation methods of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application.

[0051] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0052] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0053] First, the models involved in the embodiments of this application are described.

[0054] Diffusion model: A generative model that can generate realistic images, videos, and other high-dimensional data. It is widely used in various fields and has achieved remarkable results in high-quality image generation, such as text-generated image models and story visualization generative models. Its main working principle includes a forward diffusion process, which gradually adds random noise from clean data to pure noise, mapping the real data distribution to a standard Gaussian distribution. The backward diffusion process starts with pure noise and gradually removes noise to obtain a clear image. Through neural network prediction and noise removal, the model gradually approximates the real image. During the training phase, the model learns to predict the noise level and understand how to remove it. During the generation phase, it starts with random noise and uses backward diffusion to gradually remove noise to generate an image. The advantages of the diffusion model include high-quality generation, flexibility (supporting conditional generation and integration with other models), and stability (less prone to problems such as mode collapse compared to generative adversarial networks).

[0055] Stable Diffusion: This is an efficient text-to-image model based on a diffusion model. It converts text descriptions into images by gradually removing noise. It excels in generating high-quality, detail-rich images while being more efficient in using computing resources.

[0056] The stable diffusion model is closely related to the diffusion model. First, they share the same underlying principles. The stable diffusion model continues the diffusion model's noise addition and removal process, and the image generation steps are consistent. Second, the stable diffusion model has been optimized and improved. By improving the model structure and utilizing latent space techniques, it reduces computational requirements and increases generation speed, making it more suitable for practical application scenarios.

[0057] The stable diffusion model is designed for text-to-image generation and is able to generate images that meet user input requirements.

[0058] Figure 1 The present invention provides a method for obtaining a frame sequence from text, comprising the following steps:

[0059] S11. Based on the input text, obtain frame description text and layout information of each frame, and obtain reference images of each character.

[0060] The frames include image frames used to express the input text. For example, image frames that can express the content described in the input text are generated from the input text, and the image frames can constitute multimedia files such as videos.

[0061] The role is the role expressed in the input text. For example, if the text content contains "girl" and "man", then "girl" and "man" are the roles.

[0062] Any frame in the image is referred to as a target frame. The layout information of the target frame indicates the number and spatial information of each character in the target frame. The spatial information includes size and position. In some examples, the size is represented by the coordinates of the four vertices of a rectangular bounding box, and the position is represented by top, bottom, left, and right.

[0063] In some implementations, S11 is implemented as follows:

[0064] 110. The user inputs a simple text into the Large Language Model (LLM), such as "Write a short story about two girls, Xiao A and Xiao B. Do not use "he", "she", "it", or "they" in the story. Generally, do not use "aperson", "they", "a girl", or "the trio" to refer to the subject. When you describe the subject, always use their name!"

[0065] 111. LLM is based on simple text and outputs stories that meet the needs of simple text, such as "In a bustling city, there lived two extraordinary girls, named Xiao A and Xiao B. Xiao A was a lively and adventurous person with bright blue eyes and curly golden hair, while Xiao B was a quiet and determined girl with long black hair and a playful smile. They were inseparable friends, always exploring new places, seeking adventures, and sharing their dreams..."

[0066] It is understandable that the text output by 111 is an extension of the simple text.

[0067] 112. The above story text is the input text. LLM divides the input text into multiple paragraphs based on the input text.

[0068] 113. Each paragraph corresponds to an image frame, and LLM outputs the picture description information of each image frame.

[0069] Screen description information includes description text and layout information.

[0070] The target frame's description text is used to describe the content of the target frame. Continuing with the previous example, it could be "They are inseparable friends, always exploring new places and seeking adventure." An example of layout information is "1. Little A, [a1, b1, c1, d1], 2. Little B, [a2, b2, c2, d2]." 1 and 2 represent the character numbers, Little A and Little B are the character names, and [a1, b1, c1, d1] and [a2, b2, c2, d2] represent the sizes of the characters in the target frame. This layout information reflects the layout of the target frame. Figure 2 As shown, Figure 2 In the figure, the first box from the left is the area represented by [a1,b1,c1,d1], and the second box is the area represented by [a2,b2,c2,d2].

[0071] The reference image is input by the user. It can be understood that the user can design the image of each character and input it in the form of a reference image to obtain a character image that meets the user's needs.

[0072] In some implementations, the description text includes a global description and a detailed description of the character. The global description is a story summary of the target frame, and the detailed description is a detailed description of the character obtained by the LLM based on its understanding of the reference image of the character.

[0073] S12. Obtain text embedding of the input text and image embedding of the reference image.

[0074] Embedding refers to embedding features.

[0075] The IP-Adapter primarily uses a pre-trained Contrastive Language-Image Pre-training (CLIP) image encoder to extract image features and normalizes the feature dimensions through linear and mapping layers. The key lies in decoupling the cross-attention module and integrating image features into the generation process of the diffusion model, enabling the model to generate images based on image cues. Its low parameter count effectively enables pre-trained text-to-image diffusion models to utilize image cues.

[0076] In some implementations, the input text is encoded using a pre-trained CLIP image encoder to obtain an encoding result that is mapped to the conditional space of the stable diffusion model, i.e., a text embedding. In other words, the input text is encoded into the conditional space of the stable diffusion model to obtain a text embedding for the input text.

[0077] In some implementations, the image embedding of the reference image is obtained by encoding the image into the conditional space of the stable diffusion model using a pre-trained CLIP image encoder.

[0078] To improve the accuracy of the characters in the resulting frame sequence, some implementations remove irrelevant character information from the reference image before deriving the image embedding. This irrelevant character information may include background, composition, and style.

[0079] In some implementations, after obtaining the text embedding and the image embedding, the text embedding and the image embedding are aligned to facilitate subsequent processing. Exemplarily, the alignment of the text embedding and the image embedding is achieved through a pre-trained IP-Adapter mapping layer.

[0080] It is understandable that implementing the above process through the IP-Adapter is only an example and is not limited to the IP-Adapter.

[0081] S13: Fusing the image embedding with the layout information to obtain first fusion information.

[0082] Because the layout information represents the size and position of each character in the target frame, after fusion with the image embedding, the image embeddings of different characters in the target frame strengthen the spatial positional relationship. In other words, the first fusion information represents the positional relationship of each image embedding in each frame. For the sake of distinction, this is referred to here as the first positional relationship.

[0083] S14: Fusing the text embedding with the layout information to obtain second fusion information.

[0084] Similarly, the second fusion information represents the positional relationship between each text embedded in each frame, which is referred to herein as the second positional relationship.

[0085] It can be understood that the positional relationship between each image embedded in each frame and the positional relationship between each text embedded in each frame can reduce the mutual interference of characters in each frame.

[0086] In this embodiment, image embedding and text embedding are fused with layout information respectively, that is, image embedding and text embedding are fused with layout information respectively, which can enhance the influence of position relationship on embedding, that is, better avoid mutual interference between characters, thereby facilitating obtaining a more accurate frame sequence.

[0087] In some implementations, S13 and S14 are implemented using a cross-attention mechanism, which will be described in the following embodiments.

[0088] S15. Based on the spatiotemporal self-attention mechanism, the information is fused to obtain a frame sequence including image frames.

[0089] The information to be fused includes first fused information and second fused information.

[0090] In some implementations, the information to be fused further includes information about each character. The information about any one character in the information about each character refers to the information about the character extracted from the image containing the single character after the image containing the single character is generated.

[0091] Exemplarily, the process of extracting information of each character includes: generating an image including the single character based on the IP-Adapter and a detailed description of the single character, and extracting the character information from each reference image through GroundedSAM.

[0092] GroundedSAM is a visual application that integrates Grounding DINO (a large model for object detection) and the Segment Anything (SAM) model. It leverages Grounding DINO's zero-shot detection capabilities to find objects in images using text input. It then leverages SAM's fine-grained segmentation capabilities and combines it with Stable Diffusion to generate text and images from the segmented regions. This model enables tasks such as automatic annotation, visual question answering, and image description generation.

[0093] The spatiotemporal self-attention mechanism is commonly used to simultaneously capture temporal and spatial information in videos. This mechanism helps the model recognize and generate dynamic changes between frames while maintaining consistent spatial details within each frame. In spatiotemporal self-attention, the model not only focuses on pixel relationships within the current frame but also exchanges information between frames to understand and generate natural motion continuity. This approach is crucial for generating high-quality videos because it ensures temporal consistency in object shape, position, and motion.

[0094] Figure 1 In the process shown, both image embedding and text embedding are integrated with layout information. The layout information represents the spatial information of each character, so it can enhance the influence of spatial information on the generated image, thereby reducing the interference between different characters in each frame, which is beneficial for maintaining the consistency of the same character between frames when generating image frames with multiple characters.

[0095] The following will explain the above process in more detail in combination with the model structure.

[0096] by Figure 3 For example, the reference image of character 1 and the reference image of character 2 are encoded into the image embedding of character 1 and the image embedding of character 2 after passing through the image encoder. The description text of frame 1 and the description text of frame 2 are encoded into the text embedding of frame 1 and the text embedding of frame 2 after passing through the text encoder.

[0097] The image encoder and the text encoder may be but are not limited to the aforementioned IP-Adapter.

[0098] The image embedding of character 1 and the image embedding of character 2, as well as the layout information of frame 1 and the layout information of frame 2, are respectively input into the consistent image cross attention model to obtain the first fusion information. Specifically, Figure 3 The text embedding of frame 1 and the text embedding of frame 2, as well as the layout information of frame 1 and the layout information of frame 2, are respectively input into the consistent text cross attention model to obtain the second fusion information. Figure 3 Provide explanation.

[0099] The hidden state guidance model outputs information about each character based on the reference image of character 1 and the reference image of character 2, as well as the fine description of frame 1 and the fine description of frame 2. For details, please refer to the specific implementation method of S15.

[0100] The information and noise data of each character serve as input to the denoising U-Net network, which then outputs the denoised data. After passing through the decoder, the denoised data generates an image frame containing both character 1 and character 2, along with the descriptive text for frame 1 and frame 2. The denoising U-Net network can employ a spatiotemporal self-attention mechanism.

[0101] in, Figure 3 The image encoder, text encoder, and denoising U-Net network in

[15] can be used as the various parts of the stable diffusion model.

[0102] based on Figure 3 The architecture shown decouples the image and text control conditions in the cross-attention calculation during the sampling phase. First, the updated hidden state is calculated between the text condition and the hidden state. Then, cross-attention is performed between each character reference image condition and the hidden state, and the hidden state is updated with a fixed weight coefficient.

[0103] To more effectively use multiple character reference images as control conditions to guide generation, a consistent image cross-attention mechanism is proposed. This mechanism leverages frame layout information (represented by each character's bounding box) to generate a mask for each character in the cross-attention score matrix. This explicitly controls the influence of different character reference images on the hidden state, thereby suppressing interference between different character references. For overlapping characters in the frame, averaging the attention scores of the overlapping regions effectively prevents fusion of character features in these overlapping regions.

[0104] Fine text (i.e., fine descriptions) is long and rich in detail. However, long text can cause semantic leakage between different subjects during the cross-attention process (for example, the description belonging to subject A is over-attentioned by subject B, interfering with B's generation). To address this, the consistent text cross-attention mechanism calculates the cross-attention mask for each subject in the latent space based on the location of the descriptions of different subjects. It then adjusts the distribution of the text cross-attention score based on the area size relationship of different subjects obtained from the layout information, ensuring that the subject is not interfered with by the descriptions of other subjects and preventing the size of the subject from interfering with the attention distribution.

[0105] Latent Guidance improves adherence to layout information and character representation during the sampling phase in multi-character generation scenarios. Before generating the final frame, each character is first generated individually, combining the detailed description of each character in the storyboard. GroundedSAM is then used to extract the character's main body from the generated image to prevent background interference. Before the final storyboard generation, the latent space representation of the single character segmented by GroundedSAM and the storyboard layout information is used to guide character generation in the early sampling stages.

[0106] like Figure 4 As shown, Figure 3 An example of the function of obtaining the first fusion information in the consistent image cross attention model is as follows:

[0107] h represents the hidden state parameter, also called the hidden state, which is passed to the cross-attention layer (here we take the cross-attention layer of the consistent image cross-attention model as an example) With each image embedding (also known as IP Image Embeddinggs) Perform the first operation to obtain the first parameter. An example of the first operation is , is the first parameter, namely the attention score matrix, d represents the pre-set scaling factor, and softMax represents the normalization operation. Figure 4 The ip in the string indicates the role.

[0108] , It is the character information (referred to as character information for short) extracted from the image including the single character output by the hidden state guidance model. is based on the hidden state update of the single character. This operation is an optional step. In the case of performing this step, the first parameter is , is to obtain the intermediate parameter of the first parameter ( Figure 4 (This is taken as an example).

[0109] The first mask matrix generated by the layout information (Layout) is , Indicates the number of subjects (i.e., characters) in the target frame, Figure 3 and Figure 4 For example, is 2. Among them, is the mask of the role ip1, It is the mask of the role ip2.

[0110] Second operation = ⊙ , is the mask of any character right The mask operation makes the reference image information of each character only take effect on the hidden state of the area where it is located, and the hidden state update component of each character is obtained. The hidden state update components of each character are fused to obtain the first fusion information. It contains two parts, Figure 4 The first part a represents , the second part b represents .

[0111] like Figure 4 As shown, Figure 3 An example of how the consistency text cross attention model obtains the second fusion information is as follows:

[0112] The hidden state (in order to distinguish, it is recorded as ) performs a first operation on each text embedding (or the result of the text embedding operation) to obtain a second parameter. An example of the first operation can be found in the method for obtaining the first parameter, and will not be repeated here.

[0113] The second parameter is combined with the area scale matrix S{S 0， S 1， S2}, the second mask matrix R{r 0， r 1， r2} performs a third operation to obtain a third parameter. Exemplarily, the third operation is to perform a matrix multiplication operation (with ), and then perform normalization operation (softMax) on the result of matrix multiplication.

[0114] Among them, the area balance matrix (Area Scale Matrix, abbreviated as S) is used to balance the area occupied by each character in the target frame. In other words, the area balance matrix S is used to balance the degree of attention score adjustment between large-area subjects and small-area subjects, enhance the attention of small subjects to their corresponding text control, and avoid losing small subjects during the generation process.

[0115] The area balance matrix S includes the area balance coefficient of each character in the target frame. The smaller the area occupied by the character, the larger the area balance coefficient. In some implementations, the reciprocal of the area balance coefficient of any character is the ratio of the area occupied by the character to the area occupied by all characters. For example, the calculation formula of the area balance coefficient of any character is , is the area balance matrix of character i, is the h dimension of the hidden state, is the w dimension of the hidden state, Represents the length of R, and Area_i represents the area occupied by the i-th IP on the compressed image or hidden state. Figure 4 middle, is the area balance matrix of character 0, taking 1.0 as an example, is the area balance matrix of character 1, taking 1.4 as an example, This is the area balance matrix of character 2, taking 1.2 as an example.

[0116] The second mask matrix R is obtained based on the layout information. For example, each column in the second mask matrix is ​​a mask of the role corresponding to the column, which is used to indicate the position of the text embedding that needs to be paid attention to in the hidden state. Figure 4In the second mask matrix, 1 indicates that the element value is 1, and -inf indicates negative infinity. is the second mask matrix of role 0, is the second mask matrix of role 1, is the second mask matrix of role 2. The second mask matrix R is obtained by: obtaining the mask matrix of each role (ip) based on the layout information, Figure 4 In this example, m0 represents the mask matrix of role 0, m1 represents the mask matrix of role 1, and m2 represents the mask matrix of role 2. The mask matrix of the i-th (if i is 0, 1 or 2) ip is flattened into The longest column is then copied c_i times (the length of the text description part corresponding to the i-th ip). In this way, the part of the i-th ip in the regional attention control matrix (i.e., the second mask matrix) R is obtained. Repeat the above operation for each ip, and then splice them together to get the complete R.

[0117] A fourth operation is performed on the third parameter and each text embedding (or a result of the text embedding operation) to obtain second fusion information.

[0118] The calculation method for fusing the first fusion information and the second fusion information is: , is the fusion result of the first fusion information and the second fusion information, is the second fusion information, It is the first fusion information.

[0119] by Figure 4 For example, the consistent image cross-attention model combines the layout information of each frame with the hidden state that incorporates the image embedding, achieving the goal of influencing the hidden state based on the layout information. Similarly, the consistent text cross-attention model can also achieve the goal of influencing the hidden state based on the layout information.

[0120] Figure 2-Figure 4 It has the following advantages:

[0121] 1. No training or fine-tuning required, fast and efficient. No complex and time-consuming training process is required, and no large amount of detailed annotation and data is required.

[0122] 2. Combining the large language model and the visual language model to generate high-quality, detailed and rich descriptions, significantly improving the generation effect.

[0123] 3. Multiple character generation capabilities, with characters highly compliant with their own reference images and text descriptions.

[0124] 4. Automatically generate stories based on user prompts in an interactive manner, reducing user usage complexity and labor costs.

[0125] The above describes a method for obtaining a frame sequence from text provided by an embodiment of the present application. The following describes an apparatus for executing the above method for obtaining a frame sequence from text.

[0126] Figure 5 This is a schematic diagram of a device for obtaining a frame sequence from text provided in an embodiment of the present application. Figure 5 As shown, the device includes: a first acquisition module, a second acquisition module, a third acquisition module, a first fusion module, a second fusion module and a third fusion module.

[0127] A first acquisition module is configured to acquire, based on input text, frame description text and layout information of each frame, wherein the frame includes an image frame for expressing the input text, and the layout information of the target frame indicates spatial information of each character in the target frame;

[0128] A second acquisition module, configured to acquire text embedding of the input text;

[0129] a third acquisition module, configured to acquire an image embedding of each character's reference image based on the reference image of the character, the character being the character represented by the input text;

[0130] a first fusion module, configured to fuse the image embedding with the layout information to obtain first fusion information, wherein the first fusion information represents a first positional relationship between the images embedded in each frame;

[0131] a second fusion module, configured to fuse the text embeddings with the layout information to obtain second fusion information, wherein the second fusion information represents a second positional relationship between the text embeddings in the frames;

[0132] The third fusion module is used to fuse information based on the spatiotemporal self-attention mechanism to obtain a frame sequence including the image frames, and the fused information includes the first fusion information and the second fusion information.

[0133] In one possible implementation, the second acquisition module acquires the text embedding by encoding the input text into a conditional space of a stable diffusion model to obtain a text embedding of the input text, and the third acquisition module acquires the image embedding of the reference image by encoding the reference image into a conditional space of a stable diffusion model to obtain an image embedding of the reference image. The third acquisition module is further configured to align the text embedding with the image embedding.

[0134] In a possible implementation, the device further includes: a removal module, configured to remove information irrelevant to the character in the reference image.

[0135] In one possible implementation, the first fusion module fuses the image embedding with the layout information to obtain the first fusion information in the following manner: performing a first operation on the hidden state of the attention mechanism and each image embedding to obtain a first parameter; performing a second operation on the first parameter and a first mask matrix to obtain a hidden state update component of each character, wherein the first mask matrix is ​​obtained based on the layout information; and fusing the hidden state update components of each character to obtain the first fusion information.

[0136] In one possible implementation, the second fusion module fuses the text embedding with the layout information to obtain the second fusion information in the following manner: performing a first operation on the hidden state of the attention mechanism and each text embedding to obtain a second parameter; performing a third operation on the second parameter and an area balance matrix and a second mask matrix to obtain a third parameter, wherein the area balance matrix is ​​used to balance the area occupied by each character in the target frame, and the second mask matrix is ​​obtained based on the layout information; performing a fourth operation on the third parameter and each text embedding to obtain the second fusion information.

[0137] In a possible implementation, the apparatus further includes an extraction module configured to extract character information from an image including a single character.

[0138] In a possible implementation, the first acquisition module is based on input text, and the method of acquiring frame description text and layout information of each frame includes: outputting the input text as description text and layout information of multiple image frames based on a language model.

[0139] The device can generate multiple characters contained in text in one frame of image, and can keep the characters consistent between different image frames.

[0140] An electronic device is also provided in an embodiment of the present application. Figure 6 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0141] like Figure 6As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 602 or programs loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing device 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0142] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 6 The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0143] A computer program product is also provided in an embodiment of the present application, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the methods for obtaining a frame sequence from text provided in the embodiments of the present application.

[0144] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any one of the methods for obtaining a frame sequence from text provided in the embodiment of the present application.

[0145] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0146] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0147] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0148] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A method for obtaining a frame sequence from text, characterized in that: include: Based on input text, acquiring frame description text and layout information of each frame, wherein the frame includes an image frame for expressing the input text, and the layout information of the target frame indicates spatial information of each character in the target frame; Obtaining a text embedding of the input text; Based on a reference image of each character input by a user, obtaining an image embedding of the reference image after removing information irrelevant to the character in the reference image, wherein the character is the character expressed by the input text; fusing the image embedding with the layout information to obtain first fusion information, wherein the first fusion information represents a first positional relationship between each image embedded in each frame; fusing the text embedding with the layout information to obtain second fused information: performing a first operation on the hidden state of the attention mechanism and each text embedding to obtain a second parameter, performing a third operation on the second parameter, an area balance matrix, and a second mask matrix to obtain a third parameter, wherein the area balance matrix is ​​used to balance the area occupied by each character in the target frame, the second mask matrix is ​​obtained based on the layout information, and performing a fourth operation on the third parameter and each text embedding to obtain the second fused information, wherein the second fused information represents a second positional relationship between each text embedding in each frame; Based on the spatiotemporal self-attention mechanism, information is fused to obtain a frame sequence including the image frames, and the fused information includes the first fusion information, the second fusion information and the information of each character. The information of any one of the characters refers to the information of the character extracted from the image containing the single character after the image including the single character is generated.

2. The method according to claim 1, characterized in that The process of obtaining the text embedding and the image embedding includes: Encoding the input text into the conditional space of the stable diffusion model to obtain a text embedding of the input text; encoding the reference image into a conditional space of a stable diffusion model to obtain an image embedding of the reference image; The text embedding and the image embedding are aligned.

3. The method according to claim 1, characterized in that The step of fusing the image embedding with the layout information to obtain first fusion information includes: Perform a first operation on the hidden state of the attention mechanism and each image embedding to obtain the first parameter; performing a second operation on the first parameter and a first mask matrix to obtain a hidden state update component of each character, wherein the first mask matrix is ​​obtained based on the layout information; The hidden state update components of each character are fused to obtain the first fused information.

4. The method according to any one of claims 1 to 3, characterized in that The step of obtaining the frame description text and the layout information of each frame based on the input text includes: The input text is output as description text and layout information of a plurality of image frames based on a language model.

5. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the method for acquiring a frame sequence from a text according to any one of claims 1 to 4.

6. An electronic device, characterized in that: comprising at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program so as to enable the electronic device to implement the method for acquiring a frame sequence from text according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the method for obtaining a frame sequence from text as claimed in any one of claims 1 to 4.

Images