Image-text information processing method and device and storage medium

Abstract

The invention relates to an image-text information processing method and device and a storage medium, relates to the technical field of image-text information processing, and can be applied to the financial field and the medical field. The method comprises the following steps: executing the following operations on to-be-processed image-text information by adopting an image-text processing model: extracting text feature information from the to-be-processed image-text information, and extracting continuous image feature information from the to-be-processed image-text information; performing feature fusion on the text feature information and the image feature information to obtain a fused feature vector; determining a target modal type indicated by the to-be-generated content of the to-be-processed graphic and text information; selecting a target head module associated with the target modal type from the diffusion head module and the language model head module; processing the fusion feature vector by adopting a target head module to obtain a target feature; a target decoder associated with the target header module is adopted to decode the target features to obtain target content information; and outputting the target content information of the image-text processing model.

Description

Technical Field

[0001] This invention relates to the field of graphic information processing technology, and can be applied to the financial and medical fields. In particular, it relates to a graphic information processing method, apparatus, and storage medium. Background Technology

[0002] In the field of AI content generation, two main approaches are used to achieve high-quality image-text interleaving generation and understanding. The first is a discretized autoregressive model, which converts images into discrete visual units through vector quantization to achieve unified sequence modeling with text. However, this lossy compression process loses high-frequency details of the image, leading to a decrease in generation quality and understanding accuracy. The second is a hybrid model combining autoregression and diffusion. While attempting to integrate the advantages of both generation paradigms, it suffers from a fundamental conflict between the noisy input required by the diffusion model and the clean features needed for the understanding task. This makes the model design complex and limits its flexibility and performance when handling interleaved content. It is evident that neither of these approaches can achieve high-quality image-text interleaving generation while maintaining a simple architecture. Therefore, current technology urgently needs a solution that can avoid information loss, resolve modeling conflicts, and flexibly generate coherent image-text content. Summary of the Invention

[0003] This invention provides a method, apparatus, and storage medium for processing text and image information, to at least solve the problems of code redundancy, repetitive development, and difficulty in reusing business code in related technologies. The technical solution of this invention is as follows: According to a first aspect of the present invention, a method for processing graphic information is provided. The method includes: inputting graphic information to be processed into a graphic processing model, and performing the following operations using the graphic processing model: extracting text feature information from the graphic information to be processed, and extracting continuous image feature information from the graphic information to be processed; fusing the text feature information and the image feature information into multimodal features to obtain a fused feature vector; determining the target modality type of the content to be generated in the graphic information to be processed; selecting a target head module associated with the target modality type from a diffusion head module and a language model head module according to the association relationship between the modality type and the head module; processing the fused feature vector using the target head module to obtain target features; decoding the target features using a target decoder associated with the target head module to obtain target content information; and outputting the target content information obtained after the graphic processing model has completed the operation; the target content information includes target text or target image.

[0004] As one implementation, the image-text processing model includes a text lexer, a visual autoencoder, a differentiable visual embedding module, and a text embedding module. The output of the text lexer is connected to the input of the text embedding module; the output of the visual autoencoder is connected to the input of the differentiable visual embedding module. The model extracts text feature information from the image-text information to be processed, and extracts continuous image feature information from the image-text information to be processed, including: using the text lexer to extract and process text words in the image-text information to be processed, obtaining a discrete text word sequence; using the visual autoencoder to extract and process image features in the image-text information to be processed, obtaining a continuous image feature sequence; using the differentiable visual embedding module to map the image feature sequence into a first embedding vector sequence with the same dimension as the text word sequence; and using the text embedding module to convert the text word sequence into a second embedding vector sequence.

[0005] Another implementation method, the image and text processing model also includes an autoregressive Transformer backbone network. The input of the autoregressive Transformer backbone network is connected to the output of the differentiable visual embedding module and the output of the text embedding module, respectively. Multimodal feature fusion of text feature information and image feature information is performed to obtain a fused feature vector, including: using the autoregressive Transformer backbone network to process the dependency relationship between image features and text features in the first embedding vector sequence and the second embedding vector sequence to obtain the fused feature vector.

[0006] In another implementation, the image-text processing model also includes a head module selector. The input of the head module selector is connected to the output of the autoregressive Transformer backbone network. The head module selector includes a diffusion head module and a language model head module. It determines the target modality type of the content to be generated from the image-text information to be processed. Based on the association between the modality type and the head module, it selects the target head module associated with the target modality type from the diffusion head module and the language model head module. This includes: when the target modality type is determined to be text, using the head module selector, the language model head module associated with the text modality is used as the target head module, and the feature vector is fused and input into the language model head module; when the target modality type is determined to be image, the diffusion head module associated with the image modality is used as the target head module, and the feature vector is fused and input into the diffusion head module.

[0007] Another implementation method, the image processing model also includes: a text decoder and an image decoder; the input of the text decoder is connected to the output of the language model head module; the input of the image decoder is connected to the output of the diffusion head module; the target decoder associated with the target head module is used to decode the target features to obtain the target content information, including: when the target modality type is determined to be text modality, the text decoder is used to decode the target features to obtain the target text; when the target modality type is determined to be image modality, the image decoder is used to decode the target features to obtain the target image.

[0008] Another implementation involves training the image-text processing model based on the initial processing model. The model parameters of the text lexer, visual autoencoder, and text embedding module in the initial processing model have been trained. This method also includes: using an image sample dataset, performing a first-stage training on the model parameters of the differentiable visual embedding module and the diffusion head module in the initial processing model to obtain a pre-processing model; and using a multimodal sample dataset, performing a second-stage training on the model parameters of the autoregressive Transformer backbone network, the differentiable visual embedding module, the language model head module, and the diffusion head module in the pre-processing model to obtain the image-text processing model.

[0009] Another implementation uses a differentiable visual embedding module to map the image feature sequence into a first embedding vector sequence with the same dimension as the text word sequence. This includes: based on a learnable codebook containing multiple codeword vectors, performing the following preset operations for each image feature in the image feature sequence: determining the distance between each image feature and each codeword vector in the learnable codebook; determining the normalized weight of each image feature corresponding to each codeword vector according to the distance and a preset temperature coefficient; and weighting and summing the learnable embedding weight vectors corresponding to each codeword vector in the learnable codebook according to the normalized weights to generate a first embedding vector corresponding to each image feature, thus forming the first embedding vectors of the first embedding vector sequence.

[0010] Another implementation involves performing a first-stage training, which includes: fixing the parameters of the autoregressive Transformer backbone network and the language model head module; using an image sample dataset, training the model parameters of the differentiable visual embedding module and the diffusion head module with the goal of minimizing the image diffusion loss; the image diffusion loss characterizes the difference between the noise predicted by the diffusion head module and the real noise added to the target image features; performing a second-stage training, which includes: unfreezing the parameters of the autoregressive Transformer backbone network; using a multimodal dataset with interleaved text and images, jointly optimizing the model parameters of the autoregressive Transformer backbone network, the differentiable visual embedding module, the language model head module, and the diffusion head module with the joint goal of minimizing the weighted sum of the language modeling loss, the image diffusion loss, and the cross-modal contrast loss; the language modeling loss characterizes the difference between the next text term predicted by the language model head module and the real next text term; the cross-modal contrast loss is used to bring the distance between matching text and image feature pairs closer and push the distance between mismatched text and image feature pairs further apart in the representation space of the autoregressive Transformer backbone network.

[0011] The first and second phases of training are performed alternately and iteratively until the overall loss function converges to the preset threshold.

[0012] According to a second aspect of the present invention, a text-image information processing apparatus is provided, which includes a text-image information processing method as described in the first aspect and any possible implementation thereof. The apparatus comprises: an input unit configured to input text-image information to be processed into a text-image processing model, wherein the text-image processing model performs the following operations: extracting text feature information from the text-image information to be processed, and extracting continuous image feature information from the text-image information to be processed; fusing the text feature information and the image feature information in a multimodal manner to obtain a fused feature vector; determining the target modality type of the content to be generated in the text-image information to be processed; selecting a target head module associated with the target modality type from a diffusion head module and a language model head module according to the association relationship between the modality type and the head module; processing the fused feature vector using the target head module to obtain target features; and decoding the target features using a target decoder associated with the target head module to obtain target content information; and an output unit configured to output the target content information obtained after the text-image processing model completes the operation; wherein the target content information includes target text or a target image.

[0013] According to a third aspect of the present invention, a graphic information processing system is provided, the system being configured to perform a graphic information processing method as described in the first aspect and any possible implementation thereof.

[0014] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which instructions are stored, such that when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform a text and image information processing method as described in the first aspect and any possible implementation thereof.

[0015] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including computer instructions, which, when executed on an electronic device, cause the electronic device to perform the graphic information processing method of the first aspect and any possible implementation thereof.

[0016] The technical solutions provided by the embodiments of the present invention bring at least the following beneficial effects: First, by explicitly processing continuous image feature information, the lossy discretization process caused by related vector quantization is completely avoided, thereby achieving high-fidelity preservation and processing of visual information and significantly improving the detail quality of image feature information and the accuracy of visual understanding. Second, by directly fusing multimodal features of textual and visual modal information, deep contextual understanding of text and image information is achieved in a clean feature space, ensuring that the text and image processing model can deeply grasp the overall text and image context. Third, based on the target modality type, a target head module and target decoder adapted to the target modality are dynamically selected, enabling the text and image processing model to autonomously or according to instructions decide whether to generate text or an image in the next step based on the real-time generated context. This achieves the generation of logically coherent, highly related, and structurally flexible long-sequence interlaced content, greatly expanding the application scenarios and practicality of the text and image processing model. Furthermore, after multimodal feature fusion, the above-mentioned dynamic selection mechanism selects the appropriate head module and decoder for content generation and output, which fundamentally resolves the inherent conflict between noisy input and clean understanding in the hybrid model, making the model structure simpler and the training more stable and efficient.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0019] Figure 1 This is a schematic diagram illustrating a graphic information processing system according to an exemplary embodiment; Figure 2 This is a schematic diagram illustrating a graphic information processing model according to an exemplary embodiment; Figure 3This is a flowchart illustrating a method for processing graphic information according to an exemplary embodiment; Figure 4 This is a block diagram illustrating a graphic information processing apparatus according to an exemplary embodiment; Figure 5 This is a schematic diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0021] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0022] Before providing a detailed description of the graphic information processing method provided in the embodiments of this application, let's briefly introduce the application scenarios and implementation environment involved in the embodiments of this application.

[0023] First, a brief introduction to the application scenarios involved in this application will be given.

[0024] In recent years, significant progress has been made in large-scale models that can simultaneously understand and generate multimodal content, but existing technological approaches still face many challenges that limit their application in complex real-world scenarios.

[0025] Firstly, there's the discretized autoregressive model: pioneering work by Chameleon and LWM uses Vector Quantization (VQ) to convert images into discrete visual word sequences, enabling them to be modeled uniformly within an autoregressive framework along with text words. However, this "everything is word-based" strategy has inherent flaws: VQ, as a lossy compression technique, inevitably leads to the loss of high-frequency details in the image (such as facial features and text OCR), thus affecting the final performance of the visual task and limiting the quality of the generated image and the accuracy of the understanding task.

[0026] Secondly, there are Autoregressive and Diffusion Mixed Models (AR-Diffusion Mixed Models): another type of approach, represented by Transfusion and Show-o, attempts to integrate autoregressive modeling of text and diffusion modeling of images into a single Transformer model. These models typically take noisy image patches as input and generate images through a denoising process. However, the inherent characteristics of diffusion models in handling noisy images complicate joint modeling. The conflict between needing clean images for understanding (such as image captioning) and simultaneously inputting noisy images for generation presents challenges to model design, especially when dealing with complex text-image interleaving generation tasks, where the model's flexibility and performance are affected.

[0027] Therefore, there is an urgent need for a new unified model architecture that can avoid the information loss caused by vector quantization while maintaining the simplicity and power of autoregressive models when processing sequence data, so as to flexibly and efficiently generate high-quality interlaced graphic content.

[0028] To address the aforementioned issues, this application proposes a method for processing image and text information. It improves the model structure and execution flow of the image and text information processing model and constructs a novel autoregressive (AR) architecture. This architecture, through modality-specific output heads, can simultaneously process continuous image features and discrete text terms within a unified Transformer model, thereby efficiently generating high-quality, context-coherent mixed image and text content.

[0029] Secondly, the implementation architecture involved in this application will be briefly introduced below.

[0030] Figure 1 This is a schematic diagram of a graphic information processing system 10 provided in this disclosure. For example... Figure 1 As shown, the text and image information processing system includes a server 101 and a user terminal 102. The text and image information processing system 10 can be a distributed system, with the server 101 and the user terminal 102 communicating via a wired network and / or a wireless network.

[0031] In some embodiments, the user terminal 102 can be a terminal device. The server 101 is equipped with, for example... Figure 2The image processing model shown includes a text lexer, a text embedding module, a visual autoencoder, a differentiable visual embedding module, an autoregressive Transformer backbone network, a head module selector, a text decoder, and an image decoder. The head module selector includes a diffusion head module and a language model head module. The output of the text lexer is connected to the input of the text embedding module. The output of the visual autoencoder is connected to the input of the differentiable visual embedding module. The outputs of the text embedding module and the differentiable visual embedding module are connected to the input of the autoregressive Transformer backbone network. The output of the autoregressive Transformer backbone network is connected to the inputs of the diffusion head module and the language model head module, respectively. The output of the diffusion head module is connected to the input of the image decoder, and the output of the language model head module is connected to the input of the text decoder.

[0032] The text lexer is configured to extract and process text lexes from the text and image information to be processed, and obtain discrete text lexe sequences.

[0033] The visual autoencoder is configured to extract and process image features from the text and image information to be processed, thereby obtaining a continuous sequence of image features.

[0034] The differentiable visual embedding module is configured to map the image feature sequence to a first embedding vector sequence with the same dimension as the text word sequence.

[0035] The text embedding module is configured to convert a sequence of text terms into a second embedding vector sequence.

[0036] The autoregressive Transformer backbone network is configured to process the dependencies between image features and text features in the first and second embedding vector sequences to obtain a fused feature vector.

[0037] The head module selector is configured to select a target head module from the diffusion head module and the language model head module based on the target modality type of the content to be generated from the text and image information to be processed, and to process the fused feature vector to obtain the target features.

[0038] The text decoder is configured to perform text decoding processing on the target features to obtain the target text. The image decoder is configured to perform image decoding processing on the target features to obtain the target text.

[0039] Server 101 receives text and image information to be processed from client 102. The text and image information to be processed includes the modality type of the content to be generated. Server 101 calls the text and image processing model to process the text and image information to be processed, and obtains text content or image content related to the modality type.

[0040] In some embodiments, server 101 includes or is connected to a database, which stores image and text processing models and / or image sample datasets and image and text sample datasets. Image and text sample datasets for different business types are stored in different databases for training image and text information processing models for different business types. For example, image and text sample datasets and image and text sample datasets for the financial sector, along with image and text information processing models for the financial sector, are stored in a first database, while image and text sample datasets and image and text sample datasets and image and text information processing models for the medical sector are stored in a second database. Each user terminal 102 can access the data and models stored in the database of server 101.

[0041] In other embodiments, server 101 may be a single server, or it may be a server cluster consisting of multiple servers. In some embodiments, the server cluster may also be a distributed cluster. This application does not limit the specific implementation of server 101.

[0042] The terminal device can be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) / virtual reality (VR) device, etc., that can install and use content community applications (such as Kuaishou). This disclosure does not impose any special restrictions on the specific form of the terminal device. It can interact with users through one or more methods such as keyboard, touchpad, touch screen, remote control, voice interaction, or handwriting device.

[0043] The server 101 described above can be connected to at least one terminal device. Furthermore, this application does not limit the number or type of terminal devices.

[0044] The image and text information processing method provided in this application embodiment can be applied to the aforementioned... Figure 1 The illustrated implementation architecture includes a text and image information processing system. For ease of understanding, the text and image information processing method provided in this application will be described in detail below with reference to the accompanying drawings.

[0045] Figure 3 This is a flowchart illustrating a text and image information processing method according to an exemplary embodiment, such as... Figure 3 As shown, the image and text information processing method includes the following steps.

[0046] S31, the text and image information to be processed is input into the text and image processing model, and the following operations are performed using the text and image processing model: extracting text feature information from the text and image information to be processed, and extracting continuous image feature information from the text and image information to be processed; fusing the text feature information and image feature information into multimodal features to obtain a fused feature vector; determining the target modality type of the content to be generated in the text and image information to be processed; selecting the target head module associated with the target modality type from the diffusion head module and the language model head module according to the association relationship between the modality type and the head module; using the target head module to process the fused feature vector to obtain the target feature; using the target decoder associated with the target head module to decode the target feature to obtain the target content information.

[0047] S32 outputs the target content information obtained after the image and text processing model completes the operation.

[0048] The target content information includes target text or target image.

[0049] Through the above implementation methods, firstly, by explicitly processing continuous image feature information, the lossy discretization process caused by related vector quantization is completely avoided, thereby achieving high-fidelity preservation and processing of visual information and significantly improving the detail quality of image feature information and the accuracy of visual understanding. Secondly, by directly fusing multimodal features from textual and visual modal information, deep contextual understanding of text and image information is achieved in a clean feature space, ensuring that the text-image processing model can deeply grasp the overall text-image context. Thirdly, based on the target modality type, a target head module and target decoder adapted to the target modality are dynamically selected, enabling the text-image processing model to autonomously or by instruction decide whether to generate text or an image in the next step based on the real-time generated context. This achieves the generation of logically coherent, highly related, and structurally flexible long-sequence interlaced content, greatly expanding the application scenarios and practicality of the text-image processing model. Furthermore, after multimodal feature fusion, the above-mentioned dynamic selection mechanism selects the appropriate head module and decoder for content generation and output, which fundamentally resolves the inherent conflict between noisy input and clean understanding in the hybrid model, making the model structure simpler and the training more stable and efficient.

[0050] In one implementation, the image-text processing model includes a text lexer, a visual autoencoder, a differentiable visual embedding module, and a text embedding module. The output of the text lexer is connected to the input of the text embedding module; the output of the visual autoencoder is connected to the input of the differentiable visual embedding module.

[0051] Based on this implementation method, the specific process of extracting continuous image feature information from the image and text information to be processed is as follows.

[0052] Firstly, a visual autoencoder is used to extract and process the image features in the text and image information to be processed, resulting in a continuous sequence of image features.

[0053] In this implementation step, continuous and discrete features are used as a unified input to the image processing model. Unlike related technologies, this application directly processes the continuous and lossless image block feature sequence V=[v1,...,vn] generated by the visual encoder. At the same time, the text is converted into a discrete word sequence U=[u1,...,um] through a standard text lexer. This design avoids the information bottleneck caused by VQ (quantization connection) from the source.

[0054] Secondly, a text embedding module is used to convert the text word sequence into a second embedding vector sequence.

[0055] Correspondingly, the specific process of extracting text feature information from the image and text information to be processed is as follows.

[0056] Firstly, a text lexer is used to extract and process text words from the text and image information to be processed, resulting in a discrete text word sequence.

[0057] Secondly, a differentiable visual embedding module is used to map the image feature sequence into a first embedding vector sequence with the same dimension as the text word sequence.

[0058] In this implementation step, in order to efficiently transform a pre-trained fully autoregressive model (such as Chameleon) into the architecture of this invention, a "hard-coded to soft-coded" adaptation technique is implemented. The hard VQ lookup operation based on argmin in the models of related technologies is replaced by a differentiable, softmax-based weighted summation operation in this application, which allows visual features to be smoothly embedded into the model space.

[0059] In one implementation, the image processing model also includes an autoregressive Transformer backbone network, the input of which is connected to the output of the differentiable visual embedding module and the output of the text embedding module, respectively.

[0060] In this implementation step, the image and text feature sequences processed by the embedding module are fed into a Transformer backbone network employing a causal attention mechanism. This network models the sequences in a uniform autoregressive manner, effectively capturing dependencies within images, text, and between images and text.

[0061] Based on this implementation method, multimodal feature fusion of text feature information and image feature information is performed to obtain a fused feature vector. Specifically, the following steps are included: using an autoregressive Transformer backbone network to process the dependency relationship between image features and text features in the first embedding vector sequence and the second embedding vector sequence to obtain a fused feature vector.

[0062] In one implementation, the image processing model further includes a head module selector, the input of which is connected to the output of the autoregressive Transformer backbone network; the head module selector includes a diffusion head module and a language model head module.

[0063] In this implementation, the output state f_i of the Transformer backbone network is routed to two different heads based on the modality that needs to be generated.

[0064] First, the Language Model (LM) head: a standard linear projection layer plus softmax, used to predict the next discrete text lexical.

[0065] Secondly, the Diffusion Head: a novel design for generating the next consecutive image patch feature v_(i+1). It uses the Transformer's output f_i as a condition to predict v_(i+1) through a diffusion model. The training objective of the Diffusion Head is to minimize the noise prediction error.

[0066] Based on this implementation structure, the target modality type of the text and image information to be processed is determined. According to the association between the modality type and the head module, the specific process of selecting the target head module associated with the target modality type from the diffusion head module and the language model head module is as follows.

[0067] The first approach uses a head module selector. When the target modality type is determined to be text modality, the language model head module associated with the text modality is used as the target head module, and the feature vector is fused and input into the language model head module.

[0068] The second approach uses a head module selector. When the target modality type is determined to be an image modality, the diffusion head module associated with the image modality is used as the target head module, and the feature vector is fused and input into the diffusion head module.

[0069] As one implementation, the image processing model further includes: a text decoder and an image decoder; the input of the text decoder is connected to the output of the language model head module; the input of the image decoder is connected to the output of the diffusion head module.

[0070] Based on this structure, the target decoder associated with the target header module is used to decode the target features and obtain the target content information, including the following cases.

[0071] The first approach is to use a text decoder to process the target features when the target modality is determined to be a text modality, thereby obtaining the target text.

[0072] The second approach is to use an image decoder to process the target features when the target modality is determined to be an image modality, thereby obtaining the target image.

[0073] As one implementation method, the image-text processing model is obtained based on the training of the initial processing model; the model parameters of the text lexer, visual autoencoder and text embedding module in the initial processing model have been trained.

[0074] To enhance the model's ability to handle complex interwoven text and image tasks, multimodal post-training is performed on the established base model. Specifically, the text and image processing model is trained in stages in the following manner to obtain the text and image processing model.

[0075] First, using an image sample dataset, the model parameters of the differentiable visual embedding module and the diffusion head module in the initial processing model are trained in the first stage to obtain the pre-processing model.

[0076] Secondly, using a multimodal sample dataset, the model parameters of the autoregressive Transformer backbone network, differentiable visual embedding module, language model head module, and diffusion head module in the pre-processing model are trained in the second stage to obtain the image and text processing model.

[0077] As one implementation method, a differentiable visual embedding module is used to map the image feature sequence into a first embedding vector sequence with the same dimension as the text word sequence, specifically including the following steps.

[0078] Based on a learnable codebook containing multiple codeword vectors, for each image feature in the image feature sequence, the following preset operation is performed until all image features in the image feature sequence are processed, resulting in a first embedding vector sequence containing multiple first embedding vectors.

[0079] First, determine the distance between each image feature and each codeword vector in the learnable codebook.

[0080] Secondly, based on the distance and the preset temperature coefficient, the normalized weight of each image feature corresponding to each codeword vector is determined.

[0081] Third, the learnable embedding weight vectors corresponding to each codeword vector in the learnable codebook are weighted and summed according to normalized weights to generate the first embedding vector corresponding to each image feature, thus forming the first embedding vectors of the first embedding vector sequence.

[0082] The first embedding vector sequence includes multiple first embedding vectors.

[0083] For example, the differentiable vision embedding module described above is executed by the following formula.

[0084]

[0085] The i-th image feature in the image feature sequence Mapped to the first embedding vector The image feature sequence includes K image features; Let be the j-th codeword vector in a learnable codebook containing K codewords, and d() be the distance metric function. The coefficient is a constant; For coding The corresponding learnable embedding weight vector.

[0086] This implementation not only solves the information bottleneck problem, but also makes it possible to reuse the pre-trained weights of existing models with related technologies, greatly improving the construction efficiency.

[0087] Optionally, the first phase of training includes the following steps: fixing the parameters of the autoregressive Transformer backbone network and the language model head module, using an image sample dataset, and training the model parameters of the differentiable visual embedding module and the diffusion head module with the goal of minimizing the image diffusion loss.

[0088] Among them, the image diffusion loss characterizes the difference between the noise predicted by the diffusion head module and the real noise added to the target image features.

[0089] For example, image diffusion loss It is characterized by the following formula.

[0090]

[0091] in, This represents the features of the next image patch generated from the target. The noise scheduling coefficients are predefined for the diffusion process, where t is the time step of random sampling. N(0, I) represents standard Gaussian noise; This represents a noise prediction network with parameter θ, where the model can use a small transformer; the network takes the noisy image features, time step t, and conditional f as input, and predicts the added noise. .

[0092] The second phase of training includes the following steps: unfreezing the parameters of the autoregressive Transformer backbone network, using a multimodal dataset with text and image interleaving, and jointly optimizing the model parameters of the autoregressive Transformer backbone network, the differentiable visual embedding module, the language model head module, and the diffusion head module with the joint objective of minimizing the weighted sum of language modeling loss, image diffusion loss, and cross-modal contrast loss.

[0093] Among them, the language modeling loss represents the difference between the next text word predicted by the language model head module and the actual next text word; the cross-modal contrastive loss is used to bring the distance between matching image-text feature pairs closer and push the distance between mismatched image-text feature pairs further apart in the representation space of the autoregressive Transformer backbone network.

[0094] In some implementations, the first phase of training is performed repeatedly until the goal of minimizing image diffusion loss is achieved; then the second phase of training is performed repeatedly until the joint goal is achieved.

[0095] In other implementations, the first and second phases of training are performed iteratively, alternating until the overall loss function converges to a preset threshold.

[0096] It should be noted that the various modules within the image and text processing model can be physically connected via electrical connections. Alternatively, the system can call the corresponding modules for each step in the image and text processing model according to the following sequence of operations one through seven, achieving the same functionality.

[0097] The first operation is to extract and process the text words in the image and text information to obtain a discrete sequence of text words; and the second operation is to extract and process the image features in the image and text information to obtain a continuous sequence of image features. The third operation is to map the image feature sequence into a first embedding vector sequence with the same dimension as the text word sequence; and the fourth operation is to convert the text word sequence into a second embedding vector sequence. Fifth operation: Process the dependencies between image features and text features in the first and second embedding vector sequences to obtain the fused feature vector; The sixth operation: Based on the target modality type of the content to be generated from the text and image information to be processed, select the target head module that matches the target modality type from the diffusion head module and the language model head module, process the fused feature vector, and obtain the target features; the target features are target text words or target image features. The seventh step is to decode the target features into target content information, which includes target text or target image.

[0098] Among them, the text lexer is configured to perform the first operation; the visual autoencoder is configured to perform the second operation; the differentiable visual embedding module is configured to perform the third operation; the text embedding module is configured to perform the fourth operation; the autoregressive Transformer backbone network is configured to perform the fifth operation; the head module selector is configured to perform the sixth operation; and the decoder is configured to perform the seventh operation.

[0099] In one specific implementation method, the above-mentioned graphic information processing process is described through the following specific implementation process.

[0100] First, basic model initialization: An open-source, pre-trained multimodal understanding and generation (Chameleon) model is selected as the starting point. Its visual input module is replaced with the aforementioned differentiable visual embedding module, and the encoder part of the original visual autoencoder is frozen.

[0101] Secondly, the training process: A diffusion head is added to the output of the model described in step one. This diffusion head is implemented as an MLP containing 3 residual blocks. Then, it is trained on a dataset containing 10,000 high-quality images. During training, only the parameters of the differentiable visual embedding module and the newly added diffusion head are updated. This stage uses only images as input, and the training goal is to enable the diffusion head to generate the next image block v_(i+1) based on the features f_i of the previous image block. The original VQ-VAE decoder of the multimodal understanding and generation (Chameleon) model is fine-tuned so that it can directly reconstruct images with high quality from continuous features V, achieving better reconstruction results than the original VQ-VAE.

[0102] Third, multimodal post-training: Based on the obtained model, post-training is performed using a hybrid dataset. This dataset includes LlaVA-v1.5-665K (for visual understanding) as well as JourneyDB and LAION-COCO-aesthetic (for text-to-image processing). Except for the visual autoencoder, the model parameters of all other modules are used in the training.

[0103] Through the above specific implementation methods, a unified multimodal model that performs well in multiple tasks such as interlaced text and image generation, visual understanding, and text-to-image generation can be constructed.

[0104] Therefore, this application achieves the following technical effects through its unique architecture and training strategy.

[0105] Firstly, lossless visual information processing: directly modeling continuous image features completely avoids the information loss caused by vector quantization (VQ), enabling the model to better handle visual tasks that require high-frequency details, such as OCR and face generation, significantly improving the quality of generated images and the accuracy of visual understanding.

[0106] Secondly, the modeling is simple and flexible: it adopts a unified fully autoregressive framework to process text and image data, simplifying the modeling of cross-modal dependencies. Compared with complex AR-Diffusion hybrid models, this method avoids the interference of input noise on visual understanding tasks and is more suitable for generating contextually coherent interlaced text and image content.

[0107] Third, extremely high construction efficiency: A highly efficient strategy is proposed to construct the model of this invention from existing fully autoregressive models through "hard-to-soft" adaptation and incremental fine-tuning. Only a few newly added modules need to be fine-tuned, and a powerful basic model can be obtained in a very short time.

[0108] Fourth, outstanding performance: In multiple visual understanding and generation benchmark tests, the model of this invention significantly outperforms similar baseline models, demonstrating its powerful generation capabilities and alignment with text instructions.

[0109] Fifth, it achieves a truly end-to-end unified model: by adopting a causal attention mechanism, image and text signals, as well as understanding and generation tasks, can be processed uniformly in the same autoregressive Transformer, achieving architectural simplicity and efficiency.

[0110] Sixth, powerful interlaced text and image generation and context learning capabilities: After post-training, the model can generate long, interlaced content that is logically coherent and highly related to text and images (such as storybook creation). In addition, the model also demonstrates strong context learning capabilities, enabling it to complete new text and image editing tasks based on examples without explicit instructions.

[0111] To achieve the above functions, the graphic information processing apparatus includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art will readily recognize that, based on the algorithmic steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0112] This disclosure also provides an embodiment such as Figure 4 The image and text information processing device shown includes an input unit 41, an output unit 42, and a training unit 43.

[0113] Input unit 41 is configured to input the text and image information to be processed into the text and image processing model, so that the text and image processing model can perform the following operations: extract text feature information from the text and image information to be processed, and extract continuous image feature information from the text and image information to be processed; perform multimodal feature fusion of text feature information and image feature information to obtain fused feature vector; determine the target modality type of the content to be generated in the text and image information to be processed; select the target head module associated with the target modality type from the diffusion head module and the language model head module according to the association relationship between the modality type and the head module; use the target head module to process the fused feature vector to obtain target features; use the target decoder associated with the target head module to decode the target features to obtain target content information.

[0114] Output unit 42 is configured to output the target content information obtained after the image and text processing model has completed the operation; the target content information includes target text or target image.

[0115] As one implementation, the image-text processing model includes a text lexer, a visual autoencoder, a differentiable visual embedding module, and a text embedding module; the output of the text lexer is connected to the input of the text embedding module; the output of the visual autoencoder is connected to the input of the differentiable visual embedding module; the input unit 41 is specifically configured to: use the text lexer to extract and process text words in the image-text information to be processed to obtain a discrete text word sequence; and use the visual autoencoder to extract and process image features in the image-text information to be processed to obtain a continuous image feature sequence; use the differentiable visual embedding module to map the image feature sequence into a first embedding vector sequence with the same dimension as the text word sequence; and use the text embedding module to convert the text word sequence into a second embedding vector sequence.

[0116] In another implementation, the image and text processing model further includes an autoregressive Transformer backbone network, the input of which is connected to the output of the differentiable visual embedding module and the output of the text embedding module, respectively; the input unit 41 is specifically configured to: use the autoregressive Transformer backbone network to process the dependency relationship between image features and text features in the first embedding vector sequence and the second embedding vector sequence to obtain a fused feature vector.

[0117] In another implementation, the image processing model further includes a head module selector, the input of which is connected to the output of the autoregressive Transformer backbone network; the head module selector includes a diffusion head module and a language model head module; the input unit 41 is specifically configured as follows: when the target modality type is determined to be text modality using the head module selector, the language model head module associated with the text modality is used as the target head module, and the feature vector is fused and input into the language model head module; when the target modality type is determined to be image modality using the head module selector, the diffusion head module associated with the image modality is used as the target head module, and the feature vector is fused and input into the diffusion head module.

[0118] In another implementation, the image processing model further includes a text decoder and an image decoder; the input of the text decoder is connected to the output of the language model head module; the input of the image decoder is connected to the output of the diffusion head module; the output unit 42 is specifically configured to: when the target modality type is determined to be text modality, use the text decoder to perform text decoding processing on the target features to obtain target text; when the target modality type is determined to be image modality, use the image decoder to perform image decoding processing on the target features to obtain target image.

[0119] In another implementation, the image-text processing model is trained based on the initial processing model. The model parameters of the text lexer, visual autoencoder, and text embedding module in the initial processing model have been trained. The training unit 43 is configured to: use an image sample dataset to perform a first-stage training on the model parameters of the differentiable visual embedding module and the diffusion head module in the initial processing model to obtain a pre-processing model; and use a multimodal sample dataset to perform a second-stage training on the model parameters of the autoregressive Transformer backbone network, the differentiable visual embedding module, the language model head module, and the diffusion head module in the pre-processing model to obtain the image-text processing model.

[0120] In another implementation, the input unit 41 is specifically configured to: based on a learnable codebook containing multiple codeword vectors, perform the following preset operations for each image feature in the image feature sequence: determine the distance between each image feature and each codeword vector in the learnable codebook; determine the normalized weight corresponding to each codeword vector for each image feature according to the distance and a preset temperature coefficient; and perform a weighted summation of the learnable embedding weight vectors corresponding to each codeword vector in the learnable codebook according to the normalized weights to generate a first embedding vector corresponding to each image feature, thereby forming each first embedding vector in the first embedding vector sequence.

[0121] In another implementation, training unit 43 is specifically configured to perform a first phase of training, including: fixing the parameters of the autoregressive Transformer backbone network and the language model head module, using an image sample dataset, and training the model parameters of the differentiable visual embedding module and the diffusion head module with the objective of minimizing image diffusion loss; the image diffusion loss characterizes the difference between the noise predicted by the diffusion head module and the real noise added to the target image features; training unit 43 is specifically configured to perform a second phase of training, including: unfreezing the parameters of the autoregressive Transformer backbone network, using a graph-text interleaved... The multimodal dataset aims to jointly optimize the model parameters of the autoregressive Transformer backbone network, differentiable visual embedding module, language model head module, and diffusion head module by minimizing the weighted sum of language modeling loss, image diffusion loss, and cross-modal contrast loss. The language modeling loss represents the difference between the next text word predicted by the language model head module and the true next text word. The cross-modal contrast loss is used to bring matching image-text feature pairs closer together and push away the distance between mismatched image-text feature pairs in the representation space of the autoregressive Transformer backbone network.

[0122] Regarding the apparatus in the above embodiments, the specific manner in which each unit module performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0123] Figure 5 This is a schematic diagram of an electronic device provided in this application. (For example...) Figure 5 The electronic device 50 may include at least one processor 501 and a memory 503 for storing processor-executable instructions. The processor 501 is configured to execute the instructions in the memory 503 to implement the image and text information processing method in the following embodiments.

[0124] In addition, electronic device 50 may also include communication bus 502, at least one communication interface 504, input device 506 and output device 505.

[0125] The processor 501 may be a processor (central processing unit, CPU), a microprocessor unit, an ASIC, or one or more integrated circuits for controlling the execution of the program of the present application.

[0126] The communication bus 502 may include a path for transmitting information between the aforementioned components.

[0127] Communication interface 504 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0128] Input device 506 is used to receive input signals and output device 505 is used to output signals.

[0129] Memory 503 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory may exist independently and be connected to the processing unit via a bus. Memory may also be integrated with the processing unit.

[0130] The memory 503 stores instructions for executing the scheme of this application, and the processor 501 controls the execution. The processor 501 executes the instructions stored in the memory 503 to implement the functions of the method of this application.

[0131] In a specific implementation, as one example, the processor 501 may include one or more CPUs, for example... Figure 5 CPU0 and CPU1 in the CPU.

[0132] In a specific implementation, as one example, the electronic device 50 may include multiple processors, such as... Figure 5 Processors 501 and 507 are mentioned. Each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0133] The electronic device is as follows Figure 5 The diagram includes a processor 501 and a memory 503 for storing executable instructions of the processor 501; wherein the processor 501 is configured to execute executable instructions to implement the graphic information processing method as described in any of the possible embodiments above. And it can achieve the same technical effect, so to avoid repetition, it will not be described again here.

[0134] This application also provides a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by the processor of a graphic information processing apparatus or electronic device, the graphic information processing apparatus or electronic device is able to perform the graphic information processing method as described in any of the above possible embodiments. And it can achieve the same technical effect; to avoid repetition, it will not be described again here.

[0135] This application also provides a computer program product, including a computer program or instructions, which are executed by a processor as described in any of the possible implementations of the graphic information processing method above. The same technical effects can be achieved, and to avoid repetition, further details are omitted here.

[0136] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0137] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for processing graphic and textual information, characterized in that, The method includes: The text and image information to be processed is input into the text and image processing model, which then performs the following operations: extracting text feature information from the text and image information to be processed, and extracting continuous image feature information from the text and image information to be processed; fusing the text feature information and the image feature information into multimodal features to obtain a fused feature vector; determining the target modality type of the content to be generated in the text and image information to be processed; selecting a target head module associated with the target modality type from the diffusion head module and the language model head module according to the association between the modality type and the head module; processing the fused feature vector using the target head module to obtain target features; and decoding the target features using the target decoder associated with the target head module to obtain target content information. The target content information obtained by the image processing model after completing the operation is output; the target content information includes target text or target image.

2. The image and text information processing method according to claim 1, characterized in that, The image processing model includes a text lexer, a visual autoencoder, a differentiable visual embedding module, and a text embedding module; the output of the text lexer is connected to the input of the text embedding module; the output of the visual autoencoder is connected to the input of the differentiable visual embedding module. Extracting text feature information from the text and image information to be processed, and extracting continuous image feature information from the text and image information to be processed, includes: The text lexer is used to extract and process text words in the text-image information to be processed, resulting in a discrete text word sequence; and the visual autoencoder is used to extract and process image features in the text-image information to be processed, resulting in a continuous image feature sequence. The image feature sequence is mapped to a first embedding vector sequence with the same dimension as the text word sequence using the differentiable visual embedding module; and the text embedding module is used to convert the text word sequence into a second embedding vector sequence.

3. The image and text information processing method according to claim 2, characterized in that, The image and text processing model also includes an autoregressive Transformer backbone network, the input of which is connected to the output of the differentiable visual embedding module and the output of the text embedding module, respectively. The step of fusing the text feature information and the image feature information into a multimodal feature vector to obtain a fused feature vector includes: The autoregressive Transformer backbone network is used to process the dependencies between image features and text features in the first and second embedding vector sequences to obtain the fused feature vector.

4. The image and text information processing method according to claim 2, characterized in that, The image and text processing model further includes a head module selector, the input of which is connected to the output of the autoregressive Transformer backbone network; the head module selector includes a diffusion head module and a language model head module; determining the target modality type indicated by the content to be generated in the image and text information to be processed, and selecting the target head module associated with the target modality type from the diffusion head module and the language model head module according to the association relationship between the modality type and the head module, includes: When the target modality type is determined to be text modality using the head module selector, the language model head module associated with the text modality is used as the target head module, and the fused feature vector is input to the language model head module. When the target modality type is determined to be an image modality using the head module selector, the diffusion head module associated with the image modality is used as the target head module, and the fused feature vector is input to the diffusion head module.

5. The image and text information processing method according to claim 4, characterized in that, The image processing model further includes: a text decoder and an image decoder; the input of the text decoder is connected to the output of the language model head module; the input of the image decoder is connected to the output of the diffusion head module; the target decoder associated with the target head module decodes the target features to obtain target content information, including: When the target modality type is determined to be a text modality, the text decoder is used to perform text decoding processing on the target features to obtain the target text; When the target modality type is determined to be an image modality, the image decoder is used to perform image decoding processing on the target features to obtain the target image.

6. The image and text information processing method according to claim 5, characterized in that, The image processing model is obtained by training an initial processing model. The model parameters of the text lexer, the visual autoencoder, and the text embedding module in the initial processing model have been trained; the method further includes: Using an image sample dataset, the model parameters of the differentiable visual embedding module and the diffusion head module in the initial processing model are trained in the first stage to obtain a pre-processing model; Using a multimodal sample dataset, the model parameters of the autoregressive Transformer backbone network, the differentiable visual embedding module, the language model head module, and the diffusion head module in the pre-processing model are trained in the second stage to obtain the image and text processing model.

7. The image and text information processing method according to any one of claims 2 to 6, characterized in that, The differentiable visual embedding module is used to map the image feature sequence into a first embedding vector sequence with the same dimension as the text word sequence, including: Based on a learnable codebook containing multiple codeword vectors, for each image feature in the image feature sequence, the following preset operations are performed: determining the distance between each image feature and each codeword vector in the learnable codebook; determining the normalized weight corresponding to each codeword vector for each image feature according to the distance and a preset temperature coefficient; and weighting and summing the learnable embedding weight vectors corresponding to each codeword vector in the learnable codebook according to the normalized weights to generate a first embedding vector corresponding to each image feature, thereby forming the first embedding vectors of the first embedding vector sequence.

8. The image and text information processing method according to claim 5, characterized in that, Performing the first stage of training includes: fixing the parameters of the autoregressive Transformer backbone network and the language model head module, using the image sample dataset, and training the model parameters of the differentiable visual embedding module and the diffusion head module with the goal of minimizing image diffusion loss; the image diffusion loss characterizes the difference between the noise predicted by the diffusion head module and the real noise added to the target image features; The second phase of training includes: unfreezing the parameters of the autoregressive Transformer backbone network; using a multimodal dataset with interleaved text and images, jointly optimizing the model parameters of the autoregressive Transformer backbone network, the differentiable visual embedding module, the language model head module, and the diffusion head module with the joint objective of minimizing the weighted sum of language modeling loss, image diffusion loss, and cross-modal contrast loss; the language modeling loss represents the difference between the next text word predicted by the language model head module and the actual next text word; the cross-modal contrast loss is used to narrow the distance between matching text and image feature pairs and widen the distance between mismatched text and image feature pairs in the representation space of the autoregressive Transformer backbone network.

9. A graphic information processing apparatus as described in any one of claims 1 to 8, characterized in that, The device includes: The input unit is configured to input the text and image information to be processed into the text and image processing model, so that the text and image processing model performs the following operations: extracting text feature information from the text and image information to be processed, and extracting continuous image feature information from the text and image information to be processed; fusing the text feature information and the image feature information into multimodal features to obtain a fused feature vector; determining the target modality type of the content to be generated in the text and image information to be processed; selecting a target head module associated with the target modality type from the diffusion head module and the language model head module according to the association relationship between the modality type and the head module; processing the fused feature vector using the target head module to obtain target features; and decoding the target features using the target decoder associated with the target head module to obtain target content information. The output unit is configured to output the target content information obtained by the image processing model after completing the operation; the target content information includes target text or target image.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the graphic information processing method as described in any one of claims 1-8.

Images