Staggered image-text generation method and device, electronic equipment and storage medium

Abstract

The invention provides an interlaced image-text generation method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: inputting user demand information into an interlaced image-text generation model, and obtaining an interlaced image-text; the interlaced image-text generation model is obtained by training an initial interlaced image-text generation model based on consistency loss, and the consistency loss is determined based on a sample similarity between a first feature of a sample intermediate feature map corresponding to a current first sample image token group and a second feature of each previous sample intermediate feature map; the current first sample image token group is obtained by inputting the staggered sample text and sample image into the initial multi-mode large model. According to the method, the interlaced image-text generation model is obtained based on the consistency loss training, and the similarity between the intermediate feature maps of the samples is considered, so that the consistency between the images in the interlaced image-text generated by the interlaced image-text generation model can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for generating interlaced graphics and text. Background Art

[0002] In recent years, with the rapid development of large models in natural language processing and computer vision, multimodal generation technology has received widespread attention.

[0003] In the related art, user demand information is usually input into an autoregressive model, and interlaced graphics and text are generated through the autoregressive model. However, the consistency between images in the generated interlaced graphics and text is poor. Summary of the invention

[0004] The present invention provides a method, device, electronic device and storage medium for generating interlaced graphics and texts, so as to solve the defect that the consistency between images in the interlaced graphics and texts generated in the prior art is poor.

[0005] The present invention provides a method for generating interlaced images and texts, comprising the following steps.

[0006] Obtaining an interlaced image and text generation instruction, wherein the interlaced image and text generation instruction includes user demand information; Inputting the user demand information into an interlaced image-text generation model to obtain an interlaced image-text output by the interlaced image-text generation model; Among them, the interlaced image and text generation model is obtained by training the initial interlaced image and text generation model based on consistency loss, and the consistency loss is determined based on the sample similarity between the first feature of the sample intermediate feature map corresponding to the current first sample image token group and the second feature of each previous sample intermediate feature map, and the current first sample image token group is obtained by inputting the interlaced sample text and sample image into the initial multimodal large model in the initial interlaced image and text generation model.

[0007] According to a method for generating interlaced graphics and texts provided by the present invention, the step of inputting the user demand information into an interlaced graphics and texts generation model to obtain the interlaced graphics and texts output by the interlaced graphics and texts generation model comprises: Inputting the user demand information into the multimodal large model in the interlaced image-text generation model to obtain a current text token group and at least one current image token group output by the multimodal large model; For each of the current image token groups, determining a similarity between a first feature map corresponding to the current image token group and a second feature map corresponding to each of the previous image token groups; Determining a target current image token group from all the current image token groups based on the similarities, generating a current target image based on the target current image token group, and generating a current target text based on the current text token group; The interlaced image and text are generated based on each of the target images and each of the target texts.

[0008] According to a method for generating interlaced images and texts provided by the present invention, the method of inputting the user demand information into a multimodal large model in the interlaced image and text generation model to obtain a current text token group and at least one current image token group output by the multimodal large model includes: The user demand information is input into the multimodal large model, and the current text token group and the at least one current image token group output by the multimodal large model are obtained through an autoregressive method.

[0009] According to a method for generating interlaced images and texts provided by the present invention, the interlaced image and text generation model is trained based on the following method: Inputting the interleaved sample text and the sample image into the initial multimodal large model to obtain at least one interleaved first sample text token group and first sample image token group output by the initial multimodal large model; Inputting the current first sample image token group into the image decoder in the initial interlaced image-text generation model to obtain the current sample intermediate feature map; Extracting a first feature of the current sample intermediate feature map through the appearance feature extraction model in the initial interlaced image-text generation model, determining a sample similarity between the first feature and a second feature of each previous sample intermediate feature map, and determining a consistency loss based on each sample similarity; Determining a classification loss based on the first sample text token group and the first sample image token group; Based on the consistency loss and the classification loss, the parameters of the initial interlaced image-text generation model are iteratively optimized to obtain the interlaced image-text generation model.

[0010] According to a method for generating interlaced text and images provided by the present invention, the interlaced sample text and sample image are input into an initial multimodal large model in an initial interlaced text and image generation model, and at least one interlaced first sample text token group and first sample image token group output by the initial multimodal large model are obtained, comprising: Determine a sample token sequence based on the interleaved sample text and the sample image, wherein the sample token sequence includes an interleaved second sample text token group and a second sample image token group; Inputting the interleaved second sample text token group and the second sample image token group into the self-attention module in the first group of modules of the initial multimodal large model, and obtaining the self-attention features output by the self-attention module in an autoregressive manner; Inputting the self-attention feature into the forward feedback module in the first group of modules to obtain the hidden layer feature output by the forward feedback module in an autoregressive manner; Inputting the hidden layer features into a second group of modules to obtain hidden layer features output by the second group of modules, until obtaining hidden layer features output by a last group of modules; Based on the hidden layer features output by the last group of modules, the first sample text token group and the first sample image token group that are interlaced are obtained.

[0011] According to a method for generating interlaced images and texts provided by the present invention, the interlaced second sample text token group and the second sample image token group are input into the self-attention module in the first group of modules of the initial multimodal large model to obtain the self-attention features output by the self-attention module in an autoregressive manner, including: Inputting the interleaved second sample text token group and the second sample image token group into the self-attention module in the first group of modules to obtain text intermediate hidden layer features corresponding to each second sample text token group and image intermediate hidden layer features corresponding to each second sample image token group; Determine a memory matrix based on the intermediate hidden layer features of each text, the intermediate hidden layer features of each image, the text update strength and the image update strength in an autoregressive manner; Based on the memory matrix, the self-attention feature is determined.

[0012] According to a method for generating interlaced graphics and text provided by the present invention, the method further comprises: receiving a modification instruction for the interlaced graphics and texts, wherein the modification instruction includes user required modification information; The user requirement modification information, the user requirement information and the interlaced graphics are input into the interlaced graphics generation model to obtain the modified interlaced graphics output by the interlaced graphics generation model.

[0013] The present invention also provides a device for generating interlaced graphics and texts, comprising: An acquisition unit, used for acquiring an interlaced image and text generation instruction, wherein the interlaced image and text generation instruction includes user demand information; A first generating unit, configured to input the user demand information into an interlaced image-text generating model to obtain an interlaced image-text output by the interlaced image-text generating model; Among them, the interlaced image and text generation model is obtained by training the initial interlaced image and text generation model based on consistency loss, and the consistency loss is determined based on the sample similarity between the first feature of the sample intermediate feature map corresponding to the current first sample image token group and the second feature of each previous sample intermediate feature map, and the current first sample image token group is obtained by inputting the interlaced sample text and sample image into the initial multimodal large model in the initial interlaced image and text generation model.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for generating interlaced graphics and text as described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for generating interlaced graphics and text as described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for generating interlaced graphics and text as described above is implemented.

[0017] The interlaced image and text generation method, device, electronic device and storage medium provided by the present invention input the user demand information included in the interlaced image and text generation instruction into the trained interlaced image and text generation model to obtain the interlaced image and text output by the interlaced image and text generation model, wherein the interlaced image and text generation model is obtained by training the initial interlaced image and text generation model based on consistency loss, and the consistency loss is determined based on the sample similarity between the first feature of the sample intermediate feature map corresponding to the current first sample image token group and the second feature of each previous sample intermediate feature map. It can be seen that the present invention is based on the consistency loss determined based on the sample similarity between the first feature of the sample intermediate feature map corresponding to the current first sample image token group and the second feature of each previous sample intermediate feature map. The interlaced image and text generation model is trained by taking into account the similarity between each sample intermediate feature map, so as to improve the consistency between images in the interlaced image and text generated by the interlaced image and text generation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is one of the flow charts of the method for generating interlaced graphics and texts provided by an embodiment of the present invention.

[0020] Figure 2 This is the second flow chart of the method for generating interlaced graphics and texts provided in the embodiment of the present invention.

[0021] Figure 3 It is a flowchart of a training method for an interlaced image-text generation model provided in an embodiment of the present invention.

[0022] Figure 4 It is a schematic diagram of the framework of the initial interlaced image and text generation model provided by an embodiment of the present invention.

[0023] Figure 5 This is the third flow chart of the method for generating interlaced graphics and texts provided in the embodiment of the present invention.

[0024] Figure 6 It is one of the example diagrams of interlaced graphics and text and modification of interlaced graphics and text provided by an embodiment of the present invention.

[0025] Figure 7 This is the second example diagram of interlaced graphics and text and modification of interlaced graphics and text provided by an embodiment of the present invention.

[0026] Figure 8 This is the third example diagram of interlaced images and text and modification of interlaced images and text provided by an embodiment of the present invention.

[0027] Fig. 9 It is a schematic diagram of the structure of a device for generating interlaced graphics and texts provided in an embodiment of the present invention.

[0028] Fig.10 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Combine the following Figure 1-Figure 8 The method for generating interlaced graphics and texts of the present invention is described. The execution subject of the method for generating interlaced graphics and texts can be an electronic device such as a terminal, a tablet computer, a computer, or a device for generating interlaced graphics and texts arranged in the electronic device. The device for generating interlaced graphics and texts can be implemented by software, hardware, or a combination of the two.

[0031] Figure 1 is one of the flow charts of the method for generating interlaced graphics and texts provided by an embodiment of the present invention, such as Figure 1 As shown, the method for generating the interlaced image and text comprises the following steps: Step 101: Obtain an interlaced image and text generation instruction, wherein the interlaced image and text generation instruction includes user demand information.

[0032] For example, when a user wants to generate interlaced graphics, he or she can input user requirement information into an application loaded with a method for generating interlaced graphics and click a generation control, so that the electronic device on which the application is installed receives an interlaced graphics generation instruction including the user requirement information.

[0033] Step 102: input the user demand information into the interlaced image-text generation model to obtain the interlaced image-text output by the interlaced image-text generation model; wherein the interlaced image-text generation model is obtained by training the initial interlaced image-text generation model based on consistency loss, and the consistency loss is determined based on the sample similarity between the first feature of the sample intermediate feature map corresponding to the current first sample image token group and the second feature of each previous sample intermediate feature map, and the current first sample image token group is obtained by inputting the interlaced sample text and sample image into the initial multimodal large model in the initial interlaced image-text generation model.

[0034] For example, when the interlaced image and text generation instruction is obtained, the interlaced image and text generation instruction is parsed to obtain user demand information, the user demand information is input into a pre-trained interlaced image and text generation model, the interlaced image and text corresponding to the user demand information is output by the interlaced image and text generation model, and the generated interlaced image and text is presented to the user.

[0035] The interlaced image and text generation method provided by the present invention inputs the user demand information included in the interlaced image and text generation instruction into the trained interlaced image and text generation model to obtain the interlaced image and text output by the interlaced image and text generation model, wherein the interlaced image and text generation model is obtained by training the initial interlaced image and text generation model based on consistency loss, and the consistency loss is determined based on the sample similarity between the first feature of the sample intermediate feature map corresponding to the current first sample image token group and the second feature of each previous sample intermediate feature map. It can be seen that the present invention is based on the consistency loss determined by the sample similarity between the first feature of the sample intermediate feature map corresponding to the current first sample image token group and the second feature of each previous sample intermediate feature map. The similarity between the intermediate feature maps of each sample is taken into account, so that the consistency between the images in the interlaced image and text generated by the interlaced image and text generation model can be improved. The interlaced image and text generation method can be applied to the generation or editing tasks of picture books, and can also be applied to the generation of instructions, recipes, and teaching plans. It is rich in pictures and texts, closer to the daily generation of users, and has a wide range of application scenarios.

[0036] In one embodiment, Figure 2 FIG. 2 is a flow chart of a method for generating interlaced images and texts provided in an embodiment of the present invention. Figure 2 As shown, the above step 102 inputs the user demand information into the interlaced image and text generation model to obtain the interlaced image and text output by the interlaced image and text generation model, which can be specifically achieved by the following steps: Step 1021: input the user demand information into the multimodal large model in the interlaced image-text generation model to obtain a current text token group and at least one current image token group output by the multimodal large model.

[0037] For example, user demand information is input into the text segmenter of the interleaved image-text generation model to obtain the segmentation tokens output by the text segmenter, and the token sequence composed of the segmentation tokens is input into the multimodal large model. The multimodal large model outputs a token through the self-attention module and the forward feedback module included in several groups of modules. This token and the previous token sequence are then pieced together and input into the multimodal large model. The multimodal large model outputs the next token, and so on and so forth, through autoregressive generation, until an end identifier is output, the output of the token is ended, and finally a new token sequence is obtained. This new token sequence includes interleaved text token groups and image token groups, wherein each image token group includes at least one group of image tokens, and the new token sequence is traversed to obtain the current text token group and at least one current image token group.

[0038] Step 1022: For each of the current image token groups, determine the similarity between the first feature map corresponding to the current image token group and the second feature map corresponding to each of the previous image token groups.

[0039] For example, for each current image token group, the current image token group is input into the image decoder in the interlaced image-text generation model, the first feature map corresponding to the current image token group is output through the middle layer of the image decoder, the features of the first feature map are extracted through the apparent feature extraction model in the interlaced image-text generation model, and the similarity between the features of the first feature map and the features of the second feature maps corresponding to each of the previous image token groups is calculated, where the similarity may be cosine similarity or Pearson similarity, etc.

[0040] Step 1023: determine a target current image token group from all the current image token groups based on the similarities, generate a current target image based on the target current image token group, and generate a current target text based on the current text token group.

[0041] For example, when obtaining each similarity, all similarities are arranged in order from large to small, the first k similarities are taken, and the first k similarities are sent to the image decoder, so that the image decoder selects the target similarity from the first k similarities based on Top-k sampling, determines the current image token group corresponding to the target similarity as the target current image token group, and decodes the target current image token group to obtain the current target image; the current text token group is also input into the text decoder in the interlaced image-text generation model, and the current text token group is decoded by the text decoder to obtain the current target text.

[0042] Step 1024: Generate the interlaced image and text based on each of the target images and each of the target texts.

[0043] For example, when the target text corresponding to each text token group and the target image corresponding to each image token group are obtained, each target image and each target text are combined to generate an interlaced image and text.

[0044] In this embodiment, a target current image token group is determined from all current image token groups based on the similarities, a current target image is generated based on the target current image token group, and a current target text is generated based on the current text token group. Finally, an interlaced image and text are generated based on each target image and each target text. The similarity between the first feature map corresponding to the current image token group and the second feature map corresponding to each previous image token group is further considered, thereby further improving the consistency and coherence between images in the generated interlaced image and text.

[0045] In one embodiment, in the above step 1021, the user demand information is input into the multimodal large model in the interlaced image-text generation model to obtain the current text token group and at least one current image token group output by the multimodal large model, which can be specifically implemented in the following manner: The user demand information is input into the multimodal large model, and the current text token group and the at least one current image token group output by the multimodal large model are obtained through an autoregressive method.

[0046] For example, user demand information is input into the text segmenter of the interleaved image and text generation model to obtain the segmentation tokens output by the text segmenter, the first segmentation token is input into the multimodal large model to obtain the text token corresponding to the first segmentation token output by the multimodal large model, and then the text token corresponding to the first segmentation token and the second token are input into the multimodal large model to obtain the text token corresponding to the second token, and so on and so forth to obtain the current text token group through autoregression, and generate at least one current image token group based on the current text token group, and after traversing all the segmentation tokens, multiple interleaved text token groups and image token groups can be obtained.

[0047] In this embodiment, the current text token group and at least one current image token group output by the multimodal large model are obtained by autoregression, which improves the coherence between the text corresponding to the current text token group and the text corresponding to the previous text token group, and improves the coherence between the image corresponding to the current image token group and the image corresponding to the previous image token group.

[0048] In one embodiment, Figure 3 is a flow chart of a training method for an interlaced image-text generation model provided by an embodiment of the present invention, such as Figure 3 As shown, the interlaced image and text generation model is trained based on the following method: Step 301: Input the interleaved sample text and the sample image into the initial multimodal large model to obtain at least one interleaved first sample text token group and first sample image token group output by the initial multimodal large model.

[0049] For example, Figure 4 is a schematic diagram of the framework of the initial interlaced image and text generation model provided by an embodiment of the present invention, such as Figure 4 As shown, the initial interlaced image-text generation model includes an initial multimodal large model, a text segmenter, an image encoder, a text decoder, an image decoder and an appearance feature extraction model, wherein the initial multimodal large model includes a self-attention module and a forward feedback module.

[0050] A text-image dataset is collected, the text-image dataset includes web pages, picture book stories, instructions, recipes, etc. with interleaved texts and images, and the interleaved texts and images are preprocessed, the preprocessing includes image resizing and normalization, text segmentation and encoding, etc., finally, interleaved sample texts and sample images are obtained, for each sample text in the interleaved sample texts and sample images, the sample text is input into a text segmenter to obtain a second sample text token group corresponding to the sample text output by the text segmenter, for each sample image in the interleaved sample texts and sample images, the sample image is input into an image encoder to obtain a second sample image token group corresponding to the sample image output by the image encoder, the second sample text token group and the second sample image token group are alternately combined to obtain a sample token sequence, the sample token sequence is input into an initial multimodal large model to obtain at least one interleaved first sample text token group and first sample image token group output by the initial multimodal large model.

[0051] Step 302: Input the current first sample image token group into the image decoder in the initial interlaced image-text generation model to obtain the current sample intermediate feature map.

[0052] For example, all interlaced first sample text token groups and first sample image token groups are traversed, the current first sample image token group is input into the image decoder in the initial interlaced image-text generation model, and the current sample intermediate feature map is output through the intermediate layer of the image decoder.

[0053] Step 303: extract the first feature of the current sample intermediate feature map through the apparent feature extraction model in the initial interlaced image and text generation model, determine the sample similarity between the first feature and the second feature of each previous sample intermediate feature map, and determine the consistency loss based on each sample similarity.

[0054] For example, the appearance feature extraction model is used to extract the features of the current sample intermediate feature map output by the intermediate layer of the image decoder to obtain the first feature, and the sample similarity between the first feature and the second feature of each previous sample intermediate feature map is calculated, and then the consistency loss is determined based on the similarity of each sample by the following formula (1): : in, represents the similarity function, Represents an image decoder, which is used to restore discrete image tokens into images. Indicates sample image token groups, Indicates sample image token groups, Represents an apparent feature extraction model, which is used to extract features from the current sample intermediate feature map output by the intermediate layer of the image decoder.

[0055] It should be noted that the present invention does not limit the specific structures of the appearance feature extraction model and the image decoder. For example, the appearance feature extraction model may be dinov2, and the image decoder may be a diffusion model.

[0056] Step 304: Determine the classification loss based on the first sample text token group and the first sample image token group.

[0057] For example, when the first sample text token group and the first sample image token group are obtained, the classification loss is determined based on the difference between the first sample text token group and the second sample text token group of the sample text corresponding to the input, and the difference between the first sample image token group and the second sample image token group of the sample image corresponding to the input. , the classification loss It can be cross entropy loss.

[0058] Step 305: Based on the consistency loss and the classification loss, iteratively optimize the parameters of the initial interlaced image-text generation model to obtain the interlaced image-text generation model.

[0059] For example, when the consistency loss and classification loss are obtained, the complete loss is determined based on the following formula (2): , and based on complete loss The parameters of the initial interlaced image-text generation model are iteratively optimized to obtain the interlaced image-text generation model. Specifically, the parameters of the self-attention module and the forward feedback module in the initial interlaced image-text generation model are iteratively optimized until the performance of the interlaced image-text generation model on the validation set reaches the expected level, and finally the interlaced image-text generation model is obtained.

[0060] in, Loss coefficient, used to modulate the balance between loss functions.

[0061] It should be noted that the performance of the interlaced image-text generation model can also be evaluated based on the test set to ensure the generalization ability of the interlaced image-text generation model, and the parameters of the interlaced image-text generation model finally trained can be saved for subsequent reasoning.

[0062] In this embodiment, an apparent feature extraction model is adopted, and the calculation of sample similarity between the first feature and the second feature of each previous sample intermediate feature map is introduced. The consistency loss is determined based on the similarity of each sample, and the initial interlaced image and text generation model is constrained based on the consistency loss, so as to improve the consistency between images in the interlaced image and text generated by the trained interlaced image and text generation model.

[0063] In one embodiment, the above step 301 inputs the interleaved sample text and the sample image into the initial multimodal large model in the initial interleaved image-text generation model to obtain at least one interleaved first sample text token group and first sample image token group output by the initial multimodal large model, which can be specifically implemented by the following steps: Based on the interleaved sample text and the sample image, a sample token sequence is determined, the sample token sequence including an interleaved second sample text token group and a second sample image token group; the interleaved second sample text token group and the second sample image token group are input into the self-attention module in the first group of modules of the initial multimodal large model to obtain the self-attention features output by the self-attention module in an autoregressive manner; the self-attention features are input into the forward feedback module in the first group of modules to obtain the hidden layer features output by the forward feedback module in an autoregressive manner; the hidden layer features are input into the second group of modules to obtain the hidden layer features output by the second group of modules, until the hidden layer features output by the last group of modules are obtained; based on the hidden layer features output by the last group of modules, the interleaved first sample text token group and the first sample image token group are obtained.

[0064] For example, for each sample text in the interlaced sample text and sample image, the sample text is input into a text segmenter to obtain a second sample text token group corresponding to the sample text output by the text segmenter; for each sample image in the interlaced sample text and sample image, the sample image is input into an image encoder to obtain a second sample image token group corresponding to the sample image output by the image encoder; the second sample text token group and the second sample image token group are alternately combined to obtain the following: Figure 4 The sample token sequence 401 shown in FIG. 4 is a sample token sequence. The parts filled with diagonal lines in the sample token sequence 401 are all the second sample text token group, and the parts filled with white are all the second sample image token group. The sample token sequence is input into the self-attention module in the first group of modules of the initial multimodal large model to obtain the self-attention feature output by the self-attention module in an autoregressive manner. The self-attention feature can be expressed by the following formula (3): in, represents the self-attention feature, represents the query matrix, represents the key matrix, represents the value matrix, Represents the dimension of K, which is used to scale the dot product result to prevent the dot product value from being too large and causing the softmax function to saturate. represents the memory matrix, express The transpose of express The transpose of .

[0065] The self-attention feature is input into the forward feedback module in the first group of modules to obtain the hidden layer feature output by the forward feedback module through autoregression. The hidden layer feature can be specifically expressed by the following formula (4): in, Indicates the hidden features corresponding to the current second sample text token group and the second sample image token group, represents a forward feedback module, which usually includes two linear layers and a nonlinear activation function. Represents layer normalization operations, which are used to stabilize and accelerate the training process. Indicates the previous second sample text token group and the second sample image token group, Represents the memory matrix corresponding to the second sample text token group and the second sample image token group. represents the current query matrix, represents the current key matrix, Represents the current value matrix. Feedforward modules usually involve adjustments to the model output to improve the accuracy and consistency of the generated content.

[0066] The hidden layer features output by the first group of modules are input into the second group of modules to obtain the hidden layer features output by the second group of modules, and then the hidden layer features output by the second group of modules are input into the third group of modules until the hidden layer features output by the last group of modules are obtained, and then the hidden layer features output by the last group of modules are input into the output layer to obtain the interleaved first sample text token group and the first sample image token group output by the output layer. The interleaved first sample text token group and the first sample image token group are as follows Figure 4 In the sequence 402 shown, the parts filled with slashes in the sequence 402 are all the first sample text token group, which corresponds to the second sample text token group in the sample token sequence 401, and the parts filled with white in the sequence 402 are all the first sample image token group, which corresponds to the second sample image token group in the sample token sequence 401.

[0067] In this embodiment, a combination of multiple self-attention modules and forward feedback modules is used to represent an adaptive alternating generation mechanism, so that text and images can be naturally interspersed and alternated during the generation process. Unlike the traditional single output, this method flexibly switches between text and images according to user needs when generating content, achieving continuous and smooth interlacing of text and images. This approach not only improves the coherence of the content, but also makes the information conveyed more vivid. In addition, the autoregressive method is adopted, and the image and text generation is carried out in the same context. The previous text and previous image information are used in each generation step, so that the subsequently generated images and texts can fully refer to the previous content, so that the images and texts can achieve higher consistency in vision and semantics.

[0068] In one embodiment, the above-mentioned inputting the interleaved second sample text token group and the second sample image token group into the self-attention module in the first group of modules of the initial multimodal large model to obtain the self-attention features output by the self-attention module in an autoregressive manner can be specifically implemented in the following manner: The interleaved second sample text token group and the second sample image token group are input into the self-attention module in the first group of modules to obtain the text intermediate hidden layer features corresponding to each second sample text token group and the image intermediate hidden layer features corresponding to each second sample image token group; a memory matrix is ​​determined based on each of the text intermediate hidden layer features, each of the image intermediate hidden layer features, the text update strength and the image update strength by an autoregressive method; and the self-attention features are determined based on the memory matrix.

[0069] For example, the self-attention module adopts a memory enhancement mechanism to ensure that the previous information is not lost by maintaining and updating the external memory matrix M, thereby enhancing the ability of the self-attention module to process long sequences. Because there are two modalities, text and image, the memory matrix is ​​updated in segments. The update of the memory matrix can be specifically expressed by the following formula (5): in, Represents the memory matrix corresponding to the second sample text token group and the second sample image token group. Represents the memory matrix corresponding to the current second sample text token group and the second sample image token group, Indicates the text intermediate hidden layer features and image intermediate hidden layer features corresponding to the second sample text token group and the second sample image token group. Indicates the text intermediate hidden layer features and image intermediate hidden layer features corresponding to the current second sample text token group and the second sample image token group, When encoding text, it indicates the text update strength; when encoding images, it indicates the image update strength. When updating the memory matrix, it pays attention to the type of generated tokens. The memory matrix update weights of different types of tokens are different, that is, the text update strength and image update strength are different. Represents an update memory function, including but not limited to sliding average, GRU gated recurrent unit (GRU), attention mechanism, conditional update, nonlinear activation function, etc.

[0070] In this embodiment, the initial multimodal large model autoregressively generates an interlaced first sample text token group and a first sample image token group, and iteratively updates the memory matrix to improve the consistency between images in the interlaced text and pictures subsequently generated based on the multimodal large model.

[0071] In one embodiment, after the above step 102, the method for generating interlaced graphics and text further includes the following steps: Receive a modification instruction for the interlaced graphics, wherein the modification instruction includes user requirement modification information; input the user requirement modification information, the user requirement information and the interlaced graphics into the interlaced graphics generation model to obtain the modified interlaced graphics output by the interlaced graphics generation model.

[0072] For example, Figure 5 FIG. 3 is a flow chart of a method for generating interlaced images and texts provided in an embodiment of the present invention. Figure 5 As shown, when the user wants to generate interlaced graphics, the user demand information can be input into the application loaded with the generation method of interlaced graphics, and the generation control is clicked, so that the electronic device installed with the application receives the interlaced graphics generation instruction including the user demand information, and the interlaced graphics generation instruction is parsed to obtain the user demand information, and the user demand information is input into the pre-trained interlaced graphics generation model to obtain the interlaced graphics corresponding to the user demand information output by the interlaced graphics generation model. After the interlaced graphics are generated, if the user receives the modification instruction of the interlaced graphics, the user demand modification information included in the modification instruction is input into the intent classifier, and the user's modification requirements are classified by the intent classifier to modify the interlaced graphics more accurately. When the user's modification requirement is to modify the content of the intermediate node, the process of modifying the intermediate node is entered. If the generation of the intermediate image does not meet expectations, the user demand modification information representing the modification of the intermediate image, the user demand information input for the first time, and the generated interlaced graphics are all input into the interlaced graphics generation model, and the interlaced graphics generation model regenerates the image of the current node. Figure 6 is one of the example diagrams of interlaced graphics and text and interlaced graphics and text modification provided by an embodiment of the present invention, such as Figure 6 As shown in the figure, the user's first input user demand information is "generate a recipe for scrambled eggs with tomatoes", and the first generated interlaced graphics and texts include Figure 6 The five generation steps shown on the left, when generating a recipe, the ingredients, steps described in the text and the generated images can be accurately matched, achieving true text and picture combination. Each generation step represents a node, and each generation step includes text and images. If the user's requirement modification information is "the image of generation step 1 is not clear enough, the image of generation step 5 is better", the modified interlaced text and images include Figure 6 The five generation steps shown on the right modify the image in relation to the user's desired information.

[0073] If subsequent nodes need to be adjusted, the user demand modification information representing the need for adjustment of subsequent nodes, the user demand information input for the first time, and the generated interlaced graphics and text are all input into the interlaced graphics and text generation model. The interlaced graphics and text generation model modifies the subsequent text and images according to the user demand modification information based on the text and image before the current node. Figure 7FIG. 2 is a second example of interlaced graphics and text and interlaced graphics and text modification provided by an embodiment of the present invention. Figure 7 As shown in the figure, the user's first input user demand information is "generate a recipe for scrambled eggs with tomatoes", and the first generated interlaced graphics and texts include Figure 7 The five generation steps shown on the left each represent a node, and each generation step includes text and images. If the user needs to modify the information to "no salt, add some pepper", the modified interlaced text and images include Figure 7 The five generation steps shown on the right modify the text and images relevant to the user's required information.

[0074] If a specific node needs to be refined, the user demand modification information representing the refinement of the specific node, the user demand information input for the first time, and the generated interlaced graphics and text are all input into the interlaced graphics and text generation model. The interlaced graphics and text generation model generates refined text and images based on the text and image before the current node and the user demand modification information. Figure 8 FIG. 3 is a third example of interlaced graphics and text and interlaced graphics and text modification provided by an embodiment of the present invention. Figure 8 As shown in the figure, the user demand information input by the user for the first time is "generate the manual of the smart coffee machine", and the interlaced graphics and text generated for the first time include Figure 8 The seven parts shown on the left each represent a node, and each part includes text and images. If the user needs to modify the information to "I want to know the specific process of brewing coffee", the modified interlaced text and images include Figure 8 The five parts of the specific process of brewing coffee are shown on the right.

[0075] Ask the user whether he has any new requirements. If the user has any new requirements, adjust the interlaced image and text accordingly. If the user has no new requirements, end the current image and text generation task and output the final interlaced image and text.

[0076] In this embodiment, in order to increase the controllability and flexibility of the interlaced image and text generation model, the interlaced image and text generation model can modify the interlaced image and text based on the user demand modification information input by the user, thereby realizing the editing function of the interlaced image and text, so that the finally generated interlaced image and text is more in line with the user's personalized needs.

[0077] The following is a description of the interlaced graphics and text generation device provided by the present invention. The interlaced graphics and text generation device described below and the interlaced graphics and text generation method described above can be referenced to each other.

[0078] Fig. 9 is a schematic diagram of the structure of a device for generating interlaced graphics and texts provided by an embodiment of the present invention, such as Fig. 9As shown, the interlaced graphics and text generation device 900 includes an acquisition unit 901 and a first generation unit 902; wherein: An acquisition unit 901 is used to acquire an interlaced image and text generation instruction, wherein the interlaced image and text generation instruction includes user demand information; The first generating unit 902 is used to input the user demand information into the interlaced image and text generating model to obtain the interlaced image and text output by the interlaced image and text generating model; Among them, the interlaced image and text generation model is obtained by training the initial interlaced image and text generation model based on consistency loss, and the consistency loss is determined based on the sample similarity between the first feature of the sample intermediate feature map corresponding to the current first sample image token group and the second feature of each previous sample intermediate feature map, and the current first sample image token group is obtained by inputting the interlaced sample text and sample image into the initial multimodal large model in the initial interlaced image and text generation model.

[0079] The interlaced image and text generation device provided by the present invention inputs the user demand information included in the interlaced image and text generation instruction into the trained interlaced image and text generation model to obtain the interlaced image and text output by the interlaced image and text generation model, wherein the interlaced image and text generation model is obtained by training the initial interlaced image and text generation model based on consistency loss, and the consistency loss is determined based on the sample similarity between the first feature of the sample intermediate feature map corresponding to the current first sample image token group and the second feature of each previous sample intermediate feature map. It can be seen that the present invention is based on the consistency loss determined based on the sample similarity between the first feature of the sample intermediate feature map corresponding to the current first sample image token group and the second feature of each previous sample intermediate feature map. The interlaced image and text generation model is trained by taking into account the similarity between each sample intermediate feature map, so as to improve the consistency between images in the interlaced image and text generated by the interlaced image and text generation model.

[0080] Based on any of the foregoing embodiments, the first generating unit 902 is specifically configured to: Inputting the user demand information into the multimodal large model in the interlaced image-text generation model to obtain a current text token group and at least one current image token group output by the multimodal large model; For each of the current image token groups, determining a similarity between a first feature map corresponding to the current image token group and a second feature map corresponding to each of the previous image token groups; Determining a target current image token group from all the current image token groups based on the similarities, generating a current target image based on the target current image token group, and generating a current target text based on the current text token group; The interlaced image and text are generated based on each of the target images and each of the target texts.

[0081] Based on any of the foregoing embodiments, the first generating unit 902 is further specifically configured to: The user demand information is input into the multimodal large model, and the current text token group and the at least one current image token group output by the multimodal large model are obtained through an autoregressive method.

[0082] Based on any of the above embodiments, the interlaced image and text generation model is trained based on the following method: Inputting the interleaved sample text and the sample image into the initial multimodal large model to obtain at least one interleaved first sample text token group and first sample image token group output by the initial multimodal large model; Inputting the current first sample image token group into the image decoder in the initial interlaced image-text generation model to obtain the current sample intermediate feature map; Extracting a first feature of the current sample intermediate feature map through the appearance feature extraction model in the initial interlaced image-text generation model, determining a sample similarity between the first feature and a second feature of each previous sample intermediate feature map, and determining a consistency loss based on each sample similarity; Determining a classification loss based on the first sample text token group and the first sample image token group; Based on the consistency loss and the classification loss, the parameters of the initial interlaced image-text generation model are iteratively optimized to obtain the interlaced image-text generation model.

[0083] Based on any of the above embodiments, the step of inputting the interleaved sample text and the sample image into an initial multimodal large model in an initial interleaved image-text generation model to obtain at least one interleaved first sample text token group and first sample image token group output by the initial multimodal large model comprises: Determine a sample token sequence based on the interleaved sample text and the sample image, wherein the sample token sequence includes an interleaved second sample text token group and a second sample image token group; Inputting the interleaved second sample text token group and the second sample image token group into the self-attention module in the first group of modules of the initial multimodal large model, and obtaining the self-attention features output by the self-attention module in an autoregressive manner; Inputting the self-attention feature into the forward feedback module in the first group of modules to obtain the hidden layer feature output by the forward feedback module in an autoregressive manner; Inputting the hidden layer features into a second group of modules to obtain hidden layer features output by the second group of modules, until obtaining hidden layer features output by a last group of modules; Based on the hidden layer features output by the last group of modules, the first sample text token group and the first sample image token group that are interlaced are obtained.

[0084] Based on any of the above embodiments, the step of inputting the interleaved second sample text token group and the second sample image token group into the self-attention module in the first group of modules of the initial multimodal large model to obtain the self-attention features output by the self-attention module in an autoregressive manner includes: Inputting the interleaved second sample text token group and the second sample image token group into the self-attention module in the first group of modules to obtain text intermediate hidden layer features corresponding to each second sample text token group and image intermediate hidden layer features corresponding to each second sample image token group; Determine a memory matrix based on the intermediate hidden layer features of each text, the intermediate hidden layer features of each image, the text update strength and the image update strength in an autoregressive manner; Based on the memory matrix, the self-attention feature is determined.

[0085] Based on any of the above embodiments, the interlaced graphics and text generation device 900 further includes: A receiving unit, used for receiving a modification instruction for the interlaced graphics and texts, wherein the modification instruction includes user required modification information; The second generating unit is used to input the user requirement modification information, the user requirement information and the interlaced graphics into the interlaced graphics generation model to obtain the modified interlaced graphics output by the interlaced graphics generation model.

[0086] Fig.10 is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Fig.10 As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute the interlaced image and text generation method, the method comprising: obtaining an interlaced image and text generation instruction, the interlaced image and text generation instruction including user demand information; Inputting the user demand information into an interlaced image-text generation model to obtain an interlaced image-text output by the interlaced image-text generation model; Among them, the interlaced image and text generation model is obtained by training the initial interlaced image and text generation model based on consistency loss, and the consistency loss is determined based on the sample similarity between the first feature of the sample intermediate feature map corresponding to the current first sample image token group and the second feature of each previous sample intermediate feature map, and the current first sample image token group is obtained by inputting the interlaced sample text and sample image into the initial multimodal large model in the initial interlaced image and text generation model.

[0087] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0088] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the interlaced image and text generation method provided by the above methods, the method includes: obtaining an interlaced image and text generation instruction, the interlaced image and text generation instruction includes user demand information; Inputting the user demand information into an interlaced image-text generation model to obtain an interlaced image-text output by the interlaced image-text generation model; Among them, the interlaced image and text generation model is obtained by training the initial interlaced image and text generation model based on consistency loss, and the consistency loss is determined based on the sample similarity between the first feature of the sample intermediate feature map corresponding to the current first sample image token group and the second feature of each previous sample intermediate feature map, and the current first sample image token group is obtained by inputting the interlaced sample text and sample image into the initial multimodal large model in the initial interlaced image and text generation model.

[0089] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being implemented when executed by a processor to execute the interlaced graphics and text generation method provided by the above methods, the method comprising: obtaining an interlaced graphics and text generation instruction, the interlaced graphics and text generation instruction comprising user demand information; Inputting the user demand information into an interlaced image-text generation model to obtain an interlaced image-text output by the interlaced image-text generation model; Among them, the interlaced image and text generation model is obtained by training the initial interlaced image and text generation model based on consistency loss, and the consistency loss is determined based on the sample similarity between the first feature of the sample intermediate feature map corresponding to the current first sample image token group and the second feature of each previous sample intermediate feature map, and the current first sample image token group is obtained by inputting the interlaced sample text and sample image into the initial multimodal large model in the initial interlaced image and text generation model.

[0090] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0091] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating interlaced images and texts, characterized in that: include: Obtaining an interlaced image and text generation instruction, wherein the interlaced image and text generation instruction includes user demand information; Inputting the user demand information into an interlaced image-text generation model to obtain an interlaced image-text output by the interlaced image-text generation model; Among them, the interlaced image and text generation model is obtained by training the initial interlaced image and text generation model based on consistency loss, and the consistency loss is determined based on the sample similarity between the first feature of the sample intermediate feature map corresponding to the current first sample image token group and the second feature of each previous sample intermediate feature map, and the current first sample image token group is obtained by inputting the interlaced sample text and sample image into the initial multimodal large model in the initial interlaced image and text generation model.

2. The method for generating interlaced graphics and text according to claim 1, characterized in that: The step of inputting the user demand information into the interlaced image-text generation model to obtain the interlaced image-text output by the interlaced image-text generation model includes: Inputting the user demand information into the multimodal large model in the interlaced image-text generation model to obtain a current text token group and at least one current image token group output by the multimodal large model; For each of the current image token groups, determining a similarity between a first feature map corresponding to the current image token group and a second feature map corresponding to each of the previous image token groups; Determining a target current image token group from all the current image token groups based on the similarities, generating a current target image based on the target current image token group, and generating a current target text based on the current text token group; The interlaced image and text are generated based on each of the target images and each of the target texts.

3. The method for generating interlaced images and texts according to claim 2, characterized in that: The step of inputting the user demand information into the multimodal large model in the interlaced image-text generation model to obtain a current text token group and at least one current image token group output by the multimodal large model includes: The user demand information is input into the multimodal large model, and the current text token group and the at least one current image token group output by the multimodal large model are obtained through an autoregressive method.

4. The method for generating interlaced graphics and text according to claim 1, characterized in that: The interlaced image-text generation model is trained based on the following method: Inputting the interleaved sample text and the sample image into the initial multimodal large model to obtain at least one interleaved first sample text token group and first sample image token group output by the initial multimodal large model; Inputting the current first sample image token group into the image decoder in the initial interlaced image-text generation model to obtain the current sample intermediate feature map; Extracting a first feature of the current sample intermediate feature map through the appearance feature extraction model in the initial interlaced image-text generation model, determining a sample similarity between the first feature and a second feature of each previous sample intermediate feature map, and determining a consistency loss based on each sample similarity; Determine a classification loss based on the first sample text token group and the first sample image token group; Based on the consistency loss and the classification loss, the parameters of the initial interlaced image-text generation model are iteratively optimized to obtain the interlaced image-text generation model.

5. The method for generating interlaced graphics and text according to claim 4, characterized in that: The step of inputting the interleaved sample text and the sample image into an initial multimodal large model in an initial interleaved image-text generation model to obtain at least one interleaved first sample text token group and first sample image token group output by the initial multimodal large model comprises: Determine a sample token sequence based on the interleaved sample text and the sample image, wherein the sample token sequence includes an interleaved second sample text token group and a second sample image token group; Inputting the interleaved second sample text token group and the second sample image token group into the self-attention module in the first group of modules of the initial multimodal large model, and obtaining the self-attention features output by the self-attention module in an autoregressive manner; Inputting the self-attention feature into the forward feedback module in the first group of modules to obtain the hidden layer feature output by the forward feedback module in an autoregressive manner; Inputting the hidden layer features into a second group of modules to obtain hidden layer features output by the second group of modules, until obtaining hidden layer features output by a last group of modules; Based on the hidden layer features output by the last group of modules, the first sample text token group and the first sample image token group that are interlaced are obtained.

6. The method for generating interlaced graphics and text according to claim 5, characterized in that: The step of inputting the interleaved second sample text token group and the second sample image token group into the self-attention module in the first group of modules of the initial multimodal large model to obtain the self-attention features output by the self-attention module in an autoregressive manner includes: Inputting the interleaved second sample text token group and the second sample image token group into the self-attention module in the first group of modules to obtain text intermediate hidden layer features corresponding to each second sample text token group and image intermediate hidden layer features corresponding to each second sample image token group; Determine a memory matrix based on the intermediate hidden layer features of each text, the intermediate hidden layer features of each image, the text update strength and the image update strength in an autoregressive manner; Based on the memory matrix, the self-attention feature is determined.

7. The method for generating interlaced images and texts according to any one of claims 1 to 6, characterized in that: The method further comprises: receiving a modification instruction for the interlaced graphics and texts, wherein the modification instruction includes user required modification information; The user requirement modification information, the user requirement information and the interlaced graphics are input into the interlaced graphics generation model to obtain the modified interlaced graphics output by the interlaced graphics generation model.

8. A device for generating interlaced images and texts, characterized in that: include: An acquisition unit, used for acquiring an interlaced image and text generation instruction, wherein the interlaced image and text generation instruction includes user demand information; A first generating unit, configured to input the user demand information into an interlaced image-text generating model to obtain an interlaced image-text output by the interlaced image-text generating model; Among them, the interlaced image and text generation model is obtained by training the initial interlaced image and text generation model based on consistency loss, and the consistency loss is determined based on the sample similarity between the first feature of the sample intermediate feature map corresponding to the current first sample image token group and the second feature of each previous sample intermediate feature map, and the current first sample image token group is obtained by inputting the interlaced sample text and sample image into the initial multimodal large model in the initial interlaced image and text generation model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method for generating interlaced graphics and text as claimed in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating interlaced graphics and text as claimed in any one of claims 1 to 7 is implemented.

Images