Knowledge injection-based text-image pre-training model processing method and text-image retrieval system

Abstract

A text-image pre-training model processing method and an online text-image retrieval method based on knowledge injection are disclosed. The processing method comprises: identifying entities contained in text by a text-image sample; fusing an embedding vector corresponding to the entities in a knowledge graph with an embedding vector of the text to obtain a first embedding vector enhanced by knowledge; generating a pseudo-image label sequence based on the first embedding vector; finding a closest vector representation of an image dictionary based on a second embedding vector corresponding to the image to generate an image label representation sequence; and adjusting parameters of the text-image pre-training model according to a loss function representing the similarity between the pseudo-image label sequence and the image label representation sequence. Thus, by introducing external knowledge about entities in the model pre-training stage, the prior information of the pre-training model is enhanced, the effectiveness of the model parameters is improved, and good image reconstruction performance can be achieved by a small-scale model, thereby realizing online deployment and real-time response of the model.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning, and in particular to a method for processing text and image pre-trained models based on knowledge injection and a text and image retrieval system. Background Technology

[0002] With the advent of the information age, multimedia data (including text, images, voice, video, etc.) on the Internet has deeply permeated all aspects of people's daily lives. How to efficiently parse effective content that follows human semantic understanding from massive amounts of multimedia data and provide accurate and relevant content feedback based on specific users' behavioral habits has become a research hotspot in academia and industry in recent years.

[0003] For example, traditional single-modal techniques for pure image or pure text retrieval, due to their limited data formats, can no longer meet the increasingly diverse needs. In contrast, receiving diverse perceptual content allows AI agents to understand things more comprehensively and efficiently, which is more in line with human multi-sensory cognition. Text-image generation is a common multimodal task that aims to generate images corresponding to given text. Existing technologies improve the performance of models in downstream text-image generation tasks by increasing the size of pre-trained models and the amount of training sample data; however, such models with billions or tens of billions of parameters are often difficult to deploy and optimize in real-world applications.

[0004] Therefore, a more optimized text-image pre-training model is needed. Summary of the Invention

[0005] One technical problem this disclosure aims to solve is to provide a knowledge-injection-based method for processing text-image pre-trained models and a text-image retrieval system. By introducing external knowledge into entities, the prior information of the pre-trained model is enhanced, making the model parameters more effective, thereby achieving good image reconstruction performance even at a small scale. Furthermore, by introducing a submodule that generates discrete sequences based on an image dictionary, the model can reconstruct high-resolution images.

[0006] According to a first aspect of this disclosure, a method for processing a text-image pre-training model based on knowledge injection is provided, comprising: identifying entities contained in the text of a text-image sample pair; fusing the embedding vector corresponding to the entity in a knowledge graph with the embedding vector of the text to obtain a first embedding vector, the first embedding vector corresponding to the fused knowledge-enhanced text embedding vector; generating a pseudo-image tag sequence based on the first embedding vector, wherein the pseudo-image tags are indices of an image dictionary; searching in the image dictionary based on a second embedding vector, and obtaining an image tag representation sequence based on the closest vector representation found, wherein the second embedding vector corresponds to the embedding vector of the image in the image-text pair; and adjusting the parameters of the text-image pre-training model according to a loss function characterizing the similarity between the pseudo-image tag sequence and the image tag representation sequence.

[0007] Optionally, the text is Chinese text, and identifying entities contained in the text in the text image sample pair includes: identifying multiple entities contained in the text, including different entities corresponding to the same Chinese character, and fusing the text embedding vector of the text with the corresponding multiple entity embedding vectors to obtain the first text embedding vector.

[0008] Optionally, fusing the text embedding vector with the corresponding multiple entity embedding vectors to obtain a first embedding vector includes: assigning weights to the multiple entity embedding vectors based on the degree of interaction between each entity embedding vector and the text embedding vector; and fusing the text embedding vector with the weighted multiple entity embedding vectors to obtain a first text embedding vector.

[0009] Optionally, the text-image pre-training model includes a first sub-model for generating image sequences and a second sub-model for image reconstruction. The method further includes: feeding the text in the text-image sample pair into the encoder of the first sub-model to obtain a text embedding vector; the first sub-model obtaining the image dictionary from the second sub-model to generate the pseudo-image tag sequence; the encoder of the second sub-model generating the image tag representation sequence based on the image dictionary; and the method further includes: the decoder of the first sub-model learning to generate a corresponding image tag representation sequence based on the pseudo-image tag sequence.

[0010] Optionally, the method further includes: training the second sub-model with the image as input sample and image reconstruction as the objective, and training the second sub-model includes: training the image dictionary, wherein the image dictionary maintains a vector representation of the image label for each input sample.

[0011] Optionally, the second sub-model is trained using block mask-based adversarial training, enabling the second sub-model to reconstruct high-resolution images.

[0012] Optionally, the parameters of the second sub-model are fixed during the parameter adjustment of the first sub-model.

[0013] According to a second aspect of this disclosure, an online text-image retrieval method is provided, comprising: acquiring text information input by a user via network transmission; feeding the text information into a knowledge-enhanced text-image pre-training model acquired according to the first aspect; acquiring image information inferred by the text-image pre-training model based on the text information; and returning the image information to the user in real time via network transmission.

[0014] Optionally, the image information inferred from the text information corresponds to a text-image synthesis (TIS) task performed by the knowledge-enhanced text-image pre-trained model. In the TIS task, the decoder of the first sub-model predicts the corresponding image tag representation sequence based on the input text, and the encoder of the second sub-model is used to reconstruct the image based on the image tag representation sequence.

[0015] According to a third aspect of this disclosure, a computing device is provided, comprising: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method described in the first aspect above.

[0016] According to a fourth aspect of this disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored, which, when executed by a processor of an electronic device, causes the processor to perform the method described in the first aspect above.

[0017] Therefore, by introducing external knowledge about entities during the model pre-training stage, the prior information of the pre-trained model is enhanced, improving the effectiveness of model parameters. This enables good image reconstruction performance even with small-scale models, making online deployment and real-time response possible. Furthermore, by introducing a reconstruction submodule that generates discrete sequences based on an image dictionary, the model can reconstruct high-resolution images. Attached Figure Description

[0018] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments thereof taken in conjunction with the accompanying drawings, wherein like reference numerals generally denote like parts.

[0019] Figure 1 A schematic flowchart of a knowledge injection-based text-image pre-training model processing method according to an embodiment of the present invention is shown.

[0020] Figure 2Examples of different Chinese entities obtained under different word segmentation are shown.

[0021] Figure 3 The general process of pre-training based on GPT and VQGAN sub-models is shown.

[0022] Figure 4 An example of a pre-trained text-image pre-trained model according to the present invention is shown.

[0023] Figure 5 An example of the text and image retrieval system of the present invention is shown.

[0024] Figure 6 A schematic flowchart of an online text and image retrieval method according to an embodiment of the present invention is shown.

[0025] Figure 7 A schematic diagram of a computing device is shown, which can be used to implement the above-described knowledge-injection-based text-image pre-training model processing method according to an embodiment of the present invention. Detailed Implementation

[0026] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0027] As mentioned earlier, how to efficiently extract effective content that follows human semantic understanding from massive amounts of multimedia data and provide accurate relevant content feedback based on the behavioral habits of specific users (including application scenarios such as search, recommendation, and advertising) has become a research hotspot in academia and industry in recent years.

[0028] Traditional single-modal technologies (such as pure image or pure text retrieval) can no longer meet the increasingly diverse needs due to their singular data format. In contrast, receiving diverse perceptual content allows AI agents to understand things more comprehensively and efficiently, which is more in line with human multi-sensory cognition. Multimodal technologies are also favored by the industry due to their superior performance in many semantically rich understanding tasks.

[0029] Text-to-image generation is a common multimodal task. Also known as text-to-image synthesis (TIS), it generates images from input text, aiming to produce images corresponding to given text.

[0030] Existing technologies improve model performance in downstream text and image generation tasks by increasing the size of pre-trained models and the amount of training sample data. However, these models, often with billions or tens of billions of parameters, are difficult to deploy and optimize in real-world applications. For example, existing technologies require 4 billion model parameters to achieve the accuracy suitable for providing online services, but such large-scale models are difficult to deploy, and the computational load and time required for inference are prohibitive for providing text-image retrieval services to ordinary users.

[0031] To apply text-image models in real-world scenarios, such as generating corresponding fashion images from user-input text descriptions in the fashion industry, this invention proposes an optimization method for text-image pre-trained models. By introducing external knowledge about entities during the model pre-training stage, the prior information of the pre-trained model is enhanced, significantly improving the effectiveness of model parameters. This allows for good image reconstruction performance even with small-scale models, making online deployment and real-time response possible. Furthermore, by introducing a reconstruction submodule that generates discrete sequences based on an image dictionary, the model can reconstruct high-resolution images.

[0032] Figure 1 A schematic flowchart illustrating a knowledge injection-based text-image pre-training model processing method according to an embodiment of the present invention is shown. Specifically, Figure 1 The method shown can be viewed as a pre-training approach for a model used to perform text and image generation tasks. Compared to existing technologies, this method improves the effectiveness of model parameters by introducing external knowledge about entities, thereby effectively reducing the model parameter size and making online deployment of the model possible.

[0033] In the field of deep learning, if the tasks of a model during training and inference are inconsistent, the model training (parameter tuning) process is called the pre-training process. In this invention, during model training, text-image sample pairs are used as input, and image token (token) representation sequences are used as output (parameter tuning is performed based on a loss function that compares the similarity between the pseudo-image token sequence corresponding to the text and the image token representation sequence). However, when performing the TIS task after model training, text is used as output, and the reconstructed image is used as output. Therefore, because the tasks during training and inference are inconsistent, the model tuning process in this invention falls under the category of "pre-training" in the field of deep learning.

[0034] In step S110, the entities contained in the text within the text-image sample pair are identified. Typically, an image and the text describing that image can be considered as a text-image sample pair. For example, consider a picture of a propeller plane parked on an airport runway (as shown below). Figure 3 and Figure 4As shown, the text description "a propeller plane is parked on the airport runway" and the image can be used as a text-image sample pair as training samples for the model. The text includes entities, which can then be identified in step S110.

[0035] Subsequently, external knowledge injection can be achieved through the fusion of embeddings (also translated as "embedded representation" or "embedded"). Specifically, in step S120, the embedding vector corresponding to the entity in the knowledge graph can be fused with the embedding vector of the text to obtain a first embedding vector. Here, the first embedding vector is a knowledge-enhanced text embedding vector, obtained by fusing the embedding vector corresponding to the entity in the knowledge graph with the embedding vector of the text. When training entity embedding vectors based on the knowledge graph, information such as relationships and attributes between entities can be learned into the corresponding embedding vectors. Compared to external knowledge injection in the form of natural language, such as adding a natural language representation of the entity's relevant triples after the natural language "a propeller plane is parked on the airport runway," such as "a propeller plane is a fixed-wing plane," directly injecting embedding vectors can introduce less irrelevant information, especially when Chinese text includes multiple semantics or multiple entities.

[0036] Here, "first" and "second" are intended to distinguish different objects of the same type, rather than to imply importance or order. In this invention, the "first embedding vector" corresponds to a text-based embedding vector, but this text embedding vector is not directly generated from the text in the sample pair; rather, it is a knowledge-enhanced text embedding vector that incorporates the entity embedding vectors corresponding to the entities contained in the original text (e.g., "fusion" here could be "addition"). The "second embedding vector," mentioned below, is an image embedding vector generated from the image in the text-image sample pair.

[0037] The text-image pre-trained language model of this invention is designed for use in Chinese scenarios. Compared to languages ​​like English, which use words as the basic unit of a sentence, Chinese sentences use characters as the basic unit, with more granular word segmentation, making it difficult for the model to understand the true semantics of the input text. This further necessitates the use of external knowledge injection in this invention.

[0038] When a text contains multiple entities, especially different entities corresponding to the same Chinese character, the present invention can also inject knowledge about different entities. Figure 2An example of obtaining different Chinese entities under different word segmentation is shown. As shown in the figure, the text "propeller plane" includes five Chinese characters, and based on different word segmentation strategies, it can include three entities: "propeller", "plane" and "propeller plane". In other words, the Chinese characters "luó (propeller)", "xuán (rotation)", "jiǎng (blade)" respectively correspond to two different entities "propeller" and "propeller plane", while the Chinese characters "fēi (air)", "jī (machine)" respectively correspond to two different entities "plane" and "propeller plane". In this case, identifying the entities contained in the text in the text-image sample pair may include: identifying a plurality of entities contained in the text, wherein the plurality of entities include different entities containing the same Chinese character. Accordingly, fusing the entity embeddings corresponding to the entities in the knowledge graph with the text embedding vector of the text to obtain a first embedding vector may include: fusing the embedding vector of the text with a plurality of corresponding entity embedding vectors to obtain the first embedding vector. In other words, for the text tokens of the Chinese characters "luó", "xuán", "jiǎng", the embedding representation of the corresponding entity "propeller" can be fused, and at the same time the embedding representation of the corresponding entity "propeller plane" can be fused. Thereby further improving the richness of content injected with knowledge.

[0039] Further, in order to avoid the influence of knowledge noise, when the same text token corresponds to multiple entities, weighted fusion can be performed on these entity embeddings. For this purpose, fusing the embedding vector of the text with a plurality of corresponding entity embedding vectors to obtain a first embedding vector may include: assigning a weight to each entity embedding vector based on the degree of interaction between each entity embedding vector and the embedding vector of the text; and fusing the embedding vector of the text with the weighted plurality of entity embedding vectors to obtain the first embedding vector. Accordingly, selective knowledge injection can be performed based on the entity representation interaction module.

[0040] In the present invention, the Chinese knowledge graph can be trained in advance, for example, TransE is used to train the Chinese knowledge graph CN-DBpedia, thereby obtaining entity representations in the knowledge graph. Then, in the model training stage, corresponding to the text input, all entities contained in the text (all entities under different word segmentation strategies) are identified, and the trained entity representations (weighted representations) are combined with the corresponding text token embedding vectors to enhance entity representations. Accordingly, the present invention models the multi-granularity of input text through linguistic knowledge, and injects entity embedding vectors derived from massive relational facts in the knowledge base into the model, specifically, injects them into the encoder of the first sub-model (the sub-model for generating image sequences, such as GPT model) included in the pre-trained model. This enables the model to accommodate more linguistic knowledge with smaller-scale parameters, thereby reducing the model size while ensuring model performance.

[0041] In addition, Figure 2The text in the image represents the corresponding English words for Chinese characters or combinations of Chinese characters under different word segmentation strategies, reflecting the multi-granularity of Chinese. It should be understood that these English words are not included in the actual model processing.

[0042] After obtaining the knowledge-enhanced text embedding vector, in step S130, a pseudo-image tag sequence is generated based on the first embedding vector, where the pseudo-image tags are indices of the image dictionary. In step S140, a search is performed in the image dictionary based on the second embedding vector corresponding to the image embedding vector of the image, and an image tag representation sequence is obtained based on the closest vector representation found. Since the final task of the pre-trained model of this invention is to achieve text-image generation, but texts of different lengths and images with different content contain different information and have different forms of representation. Therefore, in the implementation of this invention, the first embedding vector (i.e., the knowledge-enhanced text embedding vector) is converted into a discrete sequence (pseudo-image tag sequence), and the second embedding vector (i.e., the image embedding vector) is also converted into a discrete sequence (image tag representation sequence) to achieve the association from text to image, as well as the final image sequence prediction and image reconstruction.

[0043] In this invention, an image codebook is used to construct the pseudo-image tag sequence and the image tag representation sequence. The image codebook is pre-trained (described in detail below based on VQGAN) and stores the vector representation of each trained image tag (with a uniform format). In practice, for any image, an intermediate feature vector is obtained after encoding by an encoder. Then, for each encoded position in the feature vector, the nearest representation in the codebook is found, thereby converting the image into a discrete sequence of image tag representations from the codebook. To achieve text-to-image prediction, the text embedding representation also needs to be associated with the codebook, such as the sequence of pseudo-image tags generated based on text embeddings, which serves as the codebook index in this invention.

[0044] Therefore, in step S150, the parameters of the text-image pre-training model can be adjusted according to the loss function characterizing the similarity between the pseudo-image tag sequence and the image tag representation sequence. This enables the pre-training model to predict the discrete sequence of image tag representations of the input image through self-supervised learning. Consequently, when the model is deployed to perform downstream TIS tasks, it can predict the associated discrete sequence of image tag representations based on the input text, thereby achieving image reconstruction.

[0045] In this invention, the text-image pre-training model may include a first sub-model for generating image sequences and a second sub-model for image reconstruction. The first sub-model is used to learn the ability to predict image sequence representations in an input text-image pair from text sequence representations in the input text-image pair, and is implemented by GPT in a preferred embodiment of the invention. The second sub-model has the ability to generate discrete image sequences from the input image (performed by the encoder) and reconstruct images from the discrete image sequences that are as close as possible to the input image (performed by the decoder), and is implemented by VQGAN in a preferred embodiment of the invention.

[0046] Figure 3 The diagram illustrates the general process of pre-training based on GPT and VQGAN sub-models. Text from a text-image sample pair can be fed into the encoder of the first sub-model to obtain a text embedding vector. For example, as shown in the diagram, the text "A propeller plane is on an airplane runway" from a text-image pair is fed into the decoder of the GPT model, generating a text embedding vector. After fusing with the entity embedding vector, the text embedding vector yields a knowledge-enhanced text embedding vector, i.e., the first embedding vector. The first embedding vector can be converted into a sequence of pseudo-image tokens (corresponding to the indexes in the codebook) by the encoder of the second sub-model (shown as the VQGAN model) based on the codebook; alternatively, as shown in the diagram, the GPT model can obtain the codebook from the VQGAN model and convert the first embedding vector representation into a sequence of pseudo-image tokens. It should be understood that the "pseudo-image token sequence" is actually a sequence generated based on text-related embedding vectors (hence the "pseudo" image token sequence), its purpose being to predict a discrete sequence of image tokens from the text embedding vectors and thereby reconstruct the image.

[0047] Accordingly, the images in the text-image pair are fed into the encoder of the VQGAN model to generate discrete sequences of image tag representations based on codebook lookup. VQGAN then feeds the resulting text and image sequences back into the GPT model. The decoder of the GPT model then learns to generate corresponding image tag representation sequences based on the pseudo-image tag sequences. For example, backpropagation (BP) based on the loss function L is used for parameter tuning of the GPT.

[0048] In other words, this invention uses GPT to learn text-to-image sequence prediction and VQGAN for image reconstruction. During the parameter tuning of the first sub-model, the parameters of the second sub-model are fixed. In other words, GPT is trained after the VQGAN has been trained. At this point, the processing method of this invention further includes: training the second sub-model with images as input samples and image reconstruction as the objective, and training the second sub-model includes: training the image dictionary, which maintains a vector representation of the image label for each input sample. In a preferred embodiment, VQGAN is trained using block-mask-based adversarial training to enable the second sub-model to reconstruct high-resolution images.

[0049] Therefore, in the downstream task (i.e., the TIS task) execution phase after the model completes pre-training, the GPT encoder receives text input (e.g., user-input text query content), converts it into a pseudo-image tag sequence, and the GPT decoder generates a discrete sequence of predicted image tag representations. The predicted image sequence can then be generated by... Figure 3 The dashed line shows the decoder passed to VQGAN, which then reconstructs the image, especially high-resolution reconstruction (e.g., reconstruction of megapixel-level images).

[0050] To better illustrate the principles of this invention, the following will be combined with... Figure 4 A preferred embodiment of the present invention is described below. Figure 4 An example of a pre-trained text-image pre-trained model according to the present invention is shown.

[0051] Figure 4 The model pre-training shown comprises two stages: the first stage is the training of the VQGAN model (corresponding to the training of the second sub-model), and the second stage is the training of the GPT model (corresponding to the training of the first sub-model). In the first stage, the VQGAN model (corresponding to the VQGAN encoder and the image decoding layer including the VQGAN decoder in the diagram) performs vector quantization on the image. That is, for the input image, the encoder encodes the image into a fixed-length discrete sequence, and the decoding stage takes the discrete sequence as input and outputs a reconstructed image. In the second stage, the text sequence and the encoded image sequence are taken as input, and the GPT model (including a multi-head attention module and a module for fusing weighted entity embedding vectors with text embeddings and feeding them into a normalization layer) learns to generate image sequences conditioned on the text sequence. To enhance the model prior, a word lattice fusion layer is designed to introduce entity knowledge from the text into the model, assisting in the generation of corresponding entities in the image. In the word lattice fusion layer, due to the multi-granularity of the Chinese language, all possible word segmentation results are obtained for the input text, and corresponding word lattices are generated. Pre-trained entity embeddings are learned from a large-scale knowledge graph and selectively injected into the entity representations through the Entity Representation Interaction Module (ERIM) designed in this invention. With the help of fused knowledge, the Transformer model constituting the GPT can automatically regress and generate "pseudo-image tags," where the codebook is obtained from the VQGAN model. Finally, the same VQGAN model is used to decode the image.

[0052] Two-stage model training • First stage: VQGAN vector quantization During the training phase of VQGAN, an image codebook is trained using images from the dataset, with image reconstruction as the task objective. This codebook stores the vector representation of each image token. In practice, for a given image, the CNN Encoder within the VQGAN encoder part encodes it to obtain an intermediate feature vector. Then, for each encoded position in the feature vector, the nearest representation in the codebook is found, thus transforming the image into a discrete sequence represented by the image tokens in the codebook. In the subsequent second stage, the GPT model generates an image sequence conditionally based on text. This generated sequence is then input into the VQGAN decoder to reconstruct an image.

[0053] • Second stage: GPT generates image sequences based on text sequences. Knowledge Injection Conventional TIS treats all tags in the input text equally, but the inventors of this invention discovered that entities described in the text are often key information for generating images closely related to specific objects. Therefore, it is necessary to identify entities and generate text tag embeddings that incorporate pre-trained entity embeddings during Transformer training. For Chinese, different word segmentation has a significant impact on the meaning of a sentence, leading to error propagation and linguistic ambiguity. Therefore, it is possible to obtain the word fold of the input sentence, which contains all possible word segments and entities of the text.

[0054] The lattice structure represents all possible entities in a sentence, but excessive knowledge injection can lead to semantic confusion (also known as knowledge noise). To avoid interactions between representations of multiple entities at the same location, an Entity Representation Interaction Module (ERIM) is designed to selectively fuse knowledge from the lattice into the Transformer model. ∈ Represented as the first kLayer tag embedding, where Indicates the first i A tag embedding, N Let be the sequence length, and d be the dimension of the hidden representation. Let M This is the set of all possible entities that appear in a given sentence within a lattice. Further... Represented as M Middle m Pre-trained entity embeddings of each entity, and Based on Knowledge needs to be injected into the first i Entity embedding in the tag. Clearly, if the tag... i The first tag and the first m With overlapping entities, we have = Otherwise = 0. Here, pre-trained entity embeddings can be obtained from a large-scale Chinese knowledge graph using TransE. The mutual knowledge injection process is calculated as follows:

[0055] in It is aimed at The m The weight of each entity embedding, This represents a knowledge-enhanced hidden tokenized embedding after selectively injecting knowledge from multiple entity embeddings. Thus, by computing the interaction between each entity representation and its tokenized embedding, all entity representations are weighted, and knowledge injection is performed selectively.

[0056] The entire embedding sequence is further represented as The Transformer layer can be constructed in the following ways:

[0057] in ATT N Represents the attention layer. LN This represents the normalization layer, where W1 and W2 are learnable parameters.

[0058] During the GPT training phase, text and image sequences are concatenated as input, and the model is trained by maximizing the negative log-likelihood of the image portion. For image generation, an autoregressive transformer can be used to generate a series of "pseudo-image tags" based on knowledge-enhanced text embeddings. Specifically, the "pseudo-image tags" are image dictionary indices encoded by the pre-trained VQGAN model and can be represented as v = { v 1, v 2, ..., v G},in GThis is the sequence length of the image tags. Given text tags w and image tags v, the discrete sequence generated by the VQGAN encoder from the image. p ( v ):

[0059] The loss function of the model can be set as: L = [ log p ( v )] where Θ is the set of model parameters. Thus, VQGAN can decode an image from a "pseudo-image token" into image pixels. The parameters of VQGAN are fixed during model training, that is, the parameters of the VQGAN model remain unchanged when the GPT model is trained.

[0060] This invention uses a patch-based GAN optimization approach to obtain clearer reconstructed images through the VQGAN model. This invention proposes a denoising knowledge injection method that introduces entity information from an external knowledge graph into the GPT model, adding prior knowledge to the model and positively impacting the generation of entity objects in images. Therefore, in this invention, the average amount of knowledge carried by each model parameter is greater than that of the state-of-the-art models in the prior art. In other words, this invention can achieve performance comparable to or even surpassing large models with a much smaller model size. The table below shows that, on four typical Chinese datasets (denoted as datasets 1-4, which can correspond to, for example, COCO-CN, MUGE, and Flicker8k-CN), the model obtained using this invention (referred to as "this model" in the table) achieves superior performance compared to state-of-the-art models in the prior art (denoted as models 1-2, which can correspond to, for example, CogView, DALL-E, etc.). Here, the well-known Fréchet initial distance (FID) and initial score (IS) are used as indicators. Higher IS and lower FID indicate better generated image quality. In the table below, the optimal performance is shown in bold. As shown in the table below, the model of this invention achieves optimal performance on all datasets with the fewest parameters.

[0061]

[0062] Furthermore, the model obtained through the fine-tuning method of this invention can generate clear images even with a small model size. Evaluation on four Chinese datasets showed that models with 200 million and 400 million pixels of data, obtained using the method proposed in this invention, achieved the best FID (Frame-of-Image) metric. Moreover, this invention utilizes the generative capabilities of the VQGAN+GPT model to produce high-resolution images superior to other related techniques. This invention can also be implemented as an online text and image retrieval system. Figure 5 An example of the text-image retrieval system of the present invention is shown. The system includes a text query information acquisition module for acquiring text information input by the user; and an image information generation module, which sets up a knowledge-enhanced text-image pre-trained model (PLM) acquired as described above, for outputting matching image information based on the user-input text information. The text-image retrieval system of the present invention is particularly suitable for providing retrieval services for online e-commerce websites, especially in the fashion field. For example, a user can enter "men's hooded zip-up sweater" in the search bar. The PLM pre-trained according to the present invention can then perform a TIS task based on the text "men's hooded zip-up sweater," generating an image sequence corresponding to the pseudo-image tag sequence of the text "men's hooded zip-up sweater" and reconstructing the corresponding image. At this time, the second sub-model of the PLM can be trained using a massive amount of product images from the online e-commerce website, while the first sub-model can be trained using product images and product descriptions.

[0063] Based on the text information, the inference of image information corresponds to a text-image synthesis (TIS) task performed by the knowledge-enhanced text-image pre-trained model. In the TIS task, the decoder of the first sub-model predicts the corresponding image tag representation sequence based on the input text, and the encoder of the second sub-model is used to reconstruct the image based on the image tag representation sequence.

[0064] This online text and image retrieval system enables an online text and image retrieval method. Figure 6 A schematic flowchart of an online text and image retrieval method according to an embodiment of the present invention is shown.

[0065] In step S610, the text information input by the user is acquired via network transmission. For example, a user has installed an online shopping app on their smartphone and enters a text description of the product they want to buy within the app. This text description can be transmitted via network to the server of the online shopping app.

[0066] In step S620, the text information is fed into the knowledge-enhanced text-image pre-training model obtained according to the method described above. For example, the knowledge-enhanced text-image pre-training model obtained by the above fine-tuning is deployed online on the server of an online shopping app. Due to the injection of external knowledge, the text-image pre-training model of the present invention can achieve performance comparable to large models on a small scale, thus making it possible to deploy online and provide users with real-time text-image retrieval services.

[0067] In step S630, image information inferred by the text-image pre-trained model based on the text information is obtained. Then, in step S640, the image information is transmitted over the network and returned to the user in real time. For example, it may be returned to an online shopping app and displayed as a search result with a text description.

[0068] Figure 7 A schematic diagram of a computing device is shown, which can be used to implement the above-described knowledge-injection-based text-image pre-training model processing method according to an embodiment of the present invention.

[0069] See Figure 7 The computing device 700 includes a memory 710 and a processor 720.

[0070] Processor 720 may be a multi-core processor or may contain multiple processors. In some embodiments, processor 720 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, processor 720 may be implemented using custom circuitry, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0071] Memory 710 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 720 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 710 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 710 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0072] The memory 710 stores executable code. When the executable code is processed by the processor 720, the processor 720 can execute the knowledge injection-based text and image pre-training model processing method described above.

[0073] The knowledge injection-based text-image pre-training model processing method and text-image retrieval system according to the present invention have been described in detail above with reference to the accompanying drawings.

[0074] This invention proposes a denoising knowledge injection method that introduces entity information from an external knowledge graph into the GPT model, adding prior knowledge to the model and positively impacting the generation of entity objects in images. Therefore, even with a small model size, clear images can be generated. For example, through evaluation on four Chinese datasets, our proposed models with scales of 200 million and 400 million pixels achieved the best FID metric. Furthermore, this invention utilizes the generative capabilities of the VQGAN+GPT model to produce high-resolution images superior to other related techniques. This invention achieves clearer reconstructed images through patch-based GAN optimization using the VQGAN model. Furthermore, the method according to the present invention can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing the steps defined in the above-described method of the present invention.

[0075] Alternatively, the present invention can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) storing executable code (or computer program, or computer instruction code) thereon, which, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform the various steps of the method described above according to the present invention.

[0076] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both.

[0077] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0078] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for processing text-image pre-trained models based on knowledge injection, comprising: Identify entities contained in Chinese text within text image sample pairs; The embedding vector corresponding to the entity in the knowledge graph is fused with the embedding vector of the text to obtain a first embedding vector, which corresponds to the knowledge-enhanced text embedding vector obtained by fusion. A pseudo-image tag sequence is generated based on the first embedding vector, wherein the pseudo-image tag is an index of the image dictionary; Based on the second embedding vector, a search is performed in the image dictionary, and based on the closest vector representation found, an image tag representation sequence is obtained, wherein the second embedding vector corresponds to the embedding vector of the image in the text-image sample pair; and The parameters of the text-image pre-training model are adjusted based on the loss function that characterizes the similarity between the pseudo-image label sequence and the image label representation sequence.

2. The method as described in claim 1, wherein, The text is Chinese text, and the entities contained in the identified text image samples include: The text contains multiple entities, including different entities corresponding to the same Chinese character. The first embedding vector is obtained by fusing the embedding vector of the entity in the knowledge graph with the embedding vector of the text, including: The embedding vector of the text is fused with the corresponding multiple entity embedding vectors to obtain the first embedding vector.

3. The method as described in claim 2, wherein, The first embedding vector is obtained by fusing the text embedding vector with the corresponding multiple entity embedding vectors, including: Weights are assigned to each entity embedding vector based on the degree of interaction between each entity embedding vector and the text embedding vector; and The embedding vector of the text is fused with multiple weighted entity embedding vectors to obtain the first embedding vector.

4. The method of claim 1, wherein, The image-text pre-training model includes a first sub-model for generating image sequences and a second sub-model for image reconstruction, wherein the method further includes: The text in the text image sample pair is fed into the encoder of the first sub-model to obtain the embedding vector of the text, and The first sub-model obtains the image dictionary from the second sub-model for generating the pseudo-image tag sequence, and the encoder of the second sub-model generates the image tag representation sequence based on the image dictionary. The method also includes: The decoder of the first sub-model learns the corresponding image label representation sequence generated from the pseudo-image label sequence.

5. The method of claim 4, further comprising: Using images as input samples and image reconstruction as the training objective, the second sub-model is trained, and training the second sub-model includes: The image dictionary is trained, and the image dictionary contains a vector representation of the image label for each input sample.

6. The method of claim 5, wherein, The second sub-model is trained using adversarial training based on block masks, enabling the second sub-model to reconstruct high-definition images.

7. The method of claim 4, wherein, During the parameter adjustment of the first sub-model, the parameters of the second sub-model are fixed.

8. An online text and image retrieval method, comprising: Text information input by the user is obtained via network transmission; The text information is fed into a knowledge-enhanced text-image pre-training model obtained according to any one of claims 1-7; The image information inferred by the text-image pre-training model is based on the text information. as well as The image information is transmitted over the network and returned to the user in real time.

9. The online text and image retrieval method as described in claim 8, wherein, Based on the text information, the image information is inferred to correspond to the text image synthesis task performed by the knowledge-enhanced text-image pre-training model. In the text image synthesis task, the decoder of the first sub-model predicts the corresponding image tag representation sequence based on the input text, and the encoder of the second sub-model is used to reconstruct the image based on the image tag representation sequence.

10. A computing device, comprising: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-7.

11. A non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-7.

Images