A composition understanding method and system based on text inversion

Abstract

The present invention discloses a composition understanding method and system based on text inversion. The method uses an optimized text inversion algorithm to convert an input image into a pseudo-token, splices the original text with the pseudo-token, converts the spliced ​​text into a feature vector, and calculates the similarity between the text and the image. The method improves the understanding ability of the composition understanding model while reducing the dependence on hard negative samples, improves the model interaction efficiency, and reduces time consumption.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and more specifically, to a composition understanding method and system based on text inversion. Background Art

[0002] Compositional understanding aims to enable visual language models to distinguish between image and text compositional representations (i.e., representations of objects, attributes, and relationships). Compositional understanding tests (e.g., SugarCrepe) evaluate this understanding based on the degree of match between text-image pairs. Understanding systems aim to match the correct answer from similar samples. Due to the limitations of training data, previous work on Structure-CLIP focused on learning the difference between positive samples and hard negative samples through supervision.

[0003] Therefore, how to improve the understanding ability of the composition understanding model while reducing the reliance on hard negative samples and reducing time consumption has become one of the urgent problems to be solved. Summary of the invention

[0004] The object of the present invention is to provide a composition understanding method and system based on text inversion, which improves the understanding ability of the composition understanding model while reducing the dependence on hard negative samples and reducing time consumption.

[0005] The present invention provides a composition understanding system based on text inversion, by performing two-stage pre-training of a neural network on an unlabeled pure image dataset. In the first stage, the goal of the present invention is to find a pseudo-token that represents the semantics of the image structure, and this embedding is obtained through optimization-based text inversion and negative regularization loss. In the second stage, the present invention refines the above-mentioned optimization-based text inversion into a single model, with the aim of obtaining a model that can map any image to a pseudo-token. This pseudo-token obtained based on text inversion can not only accurately represent the content of the image, but also interact with other tokens. At the same time, the single model obtained by distillation in the second stage can effectively solve the time overhead brought by the optimization-based method.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A composition understanding method based on text inversion, which is used to test the ability of a model to match correct text and images, comprises the following steps:

[0008] S1: a data input step, receiving an input data pair, wherein the data pair includes an image, a matching text, and an erroneous text represented in a reassembled correct text;

[0009] S2: Data conversion step, using an optimization-based text inversion algorithm to convert the input image into a pseudo-token;

[0010] S3: Data concatenation step, using different templates to concatenate the original text and pseudo-token;

[0011] S4: vector generation step, converting the concatenated text into a feature vector using a neural network;

[0012] S5: similarity calculation step, respectively calculating the cosine similarity between the correct text and the wrong text after splicing and the image;

[0013] S6: Matching judgment step, if the similarity between the correct text and the image after splicing is greater than the similarity between the incorrect text and the image after splicing, the match is correct, otherwise the match is incorrect.

[0014] Preferably, the step S2 specifically includes:

[0015] S21: feature extraction step, the input image is passed through an image encoder to obtain image features;

[0016] S22: pseudo-token generation step, the image features are processed through an optimized iterative algorithm to obtain a pseudo-token corresponding to the image.

[0017] Preferably, the step S4 specifically includes:

[0018] S41: preprocessing text generation step, concatenating the pseudo-token with the input text to form a preprocessing text;

[0019] S42: text segmentation and tagging step, the preprocessed text is processed by a word segmenter, and the text is segmented into multiple tokens, each token corresponds to a unique number;

[0020] S43: a text token mapping step, wherein the token is sent to an embedding lookup table, which maps each token into a high-dimensional vector representation;

[0021] S44: Model vector conversion step. After completing the embedding lookup, these high-dimensional vectors are input into the text deep learning model. The text deep learning model converts the serialized embedding vectors into context-related semantic representations.

[0022] Preferably, the text deep learning model is a transformer, and the word segmenter, embedding lookup table and text deep learning model together constitute a text encoder.

[0023] On the other hand, the present invention further discloses a composition understanding system based on text inversion, which executes any of the composition understanding methods described above, and the system specifically includes the following modules:

[0024] A data input module, used for receiving an input data pair, wherein the data pair includes an image, a matching text, and an erroneous text represented in a reassembled correct text;

[0025] The data conversion module is used to convert the input image into pseudo-token using an optimized text inversion algorithm;

[0026] Data splicing module, used to splice original text and pseudo-token using different templates;

[0027] A vector generation module is used to convert the concatenated text into a feature vector using a neural network;

[0028] A similarity calculation module is used to calculate the cosine similarity between the correct text and the wrong text after splicing and the image respectively;

[0029] The matching judgment module is used to judge that if the similarity between the correct text after splicing and the image is greater than the similarity between the incorrect text after splicing and the image, the match is correct, otherwise the match is incorrect.

[0030] Preferably, the data conversion module specifically includes:

[0031] A feature extraction module is used to obtain image features by passing the input image through an image encoder;

[0032] In the pseudo-token generation module, the image features are processed through an optimized iterative algorithm to obtain the pseudo-token corresponding to the image.

[0033] Preferably, the vector generation module specifically includes:

[0034] The preprocessing text generation module is used to concatenate pseudo-token with the input text to form preprocessing text;

[0035] The text segmentation and tagging module is used to process the preprocessed text through a word segmenter and segment the text into multiple tokens, each of which corresponds to a unique number;

[0036] A text token mapping module, configured to feed the tokens into an embedding lookup table, which maps each token into a high-dimensional vector representation;

[0037] Model vector conversion module,After completing the embedding lookup, these high-dimensional vectors are input into the text deep learning model, which converts the serialized embedding vectors into context-related semantic representations.

[0038] Preferably, the text deep learning model is a transformer, and the word segmenter, embedding lookup table and text deep learning model together constitute a text encoder.

[0039] On the other hand, the present invention also discloses a model training method for composition understanding based on text inversion, which is used to train any of the test models described above, and the method comprises the following steps:

[0040] S1: Frontal text feature extraction step, randomly sampling context from PALAVRA template , and feed it into the CLIP text encoder , thus obtaining; ;

[0041] S2: Negative text feature extraction step, defining a set of negative cues to modify the original text and feeding it into the CLIP text encoder , get negative text features ;

[0042] S3: Data classification step. For an unlabeled image I, the zero-shot classification capability of CLIP is used to classify it into the categories of the dataset, where the hyperparameter k represents the k different visual atoms that are most similar to image I.

[0043] S4: Iterative generation step. During the iteration, one of the k categories most similar to image I is randomly sampled. The lightweight GPT model continues to write the text according to the prompt to obtain a title with the atom. ;

[0044] S5: Update the learning step, intervene in the embedding process, and use the newly learned embedding value Replace the vectors associated with the token characters to get the title with pseudo words ;

[0045] S6: Text encoding step: CLIP text encoder obtains text features and ,

[0046] Among them, the cosine loss is used to minimize the gap between two features: , for negative text, use the same loss to minimize and The distance between them; the total loss is: .

[0047] Preferably, the method further comprises:

[0048] S7: Text inversion step, which refines the knowledge obtained through optimized text inversion into , for each image, use CLIP's visual encoder to extract its features , and then predict the pseudo-token , a mini-batch containing N input pairs is defined as ,in It is an unlabeled image dataset, pseudo-token ;

[0049] S8: Process optimization step, using symmetric contrast loss to optimize the above process as follows:

[0050] ;

[0051] in, Used to maximize pre-generated pseudo-tokens and Predicted pseudo-token The cosine distance between

[0052] S9: Similarity minimization step, minimize the similarity between different tokens, the formula is as follows:

[0053] ;

[0054] in is a temperature parameter;

[0055] S10: Regularization step, using regularization techniques to update weights, the total loss function is:

[0056]

[0057] in, and is the loss weight.

[0058] The present invention provides a composition understanding method, system and model training method based on text inversion, which improves the understanding ability of the composition understanding model while reducing the dependence on hard negative samples and reducing time consumption. A composition understanding system based on text inversion is provided, which performs two-stage pre-training on a neural network on an unlabeled pure image dataset.

[0059] Compared with the prior art, the present invention provides a composition understanding method, system and model training method based on text inversion, which have the following beneficial effects:

[0060] 1. The input image is converted into a pseudo-token based on an optimized text inversion algorithm, which reduces the reliance on hard negative samples and improves the representation ability of the composition understanding model. It can also achieve interaction between tokens and effectively improve the interaction efficiency of tokens.

[0061] 2. Through the second stage of knowledge distillation training, the time overhead caused by the optimization-based method is effectively solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A schematic flow chart of a composition understanding method based on text inversion disclosed in the present invention;

[0063] Figure 2 A schematic diagram of the process of mapping a single image to a pseudo-token using text inversion disclosed in the present invention;

[0064] Figure 3 It is a schematic diagram of the first stage training method of the text inversion-based composition understanding model disclosed in the present invention;

[0065] Figure 4 Schematic diagram of the second-stage training method of the text inversion-based composition understanding model disclosed in the present invention. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0067] Compositionality is often considered to be a common fundamental feature of human vision and natural language. Through compositional understanding, people can understand more complex scenes by combining structured representations (i.e., objects, attributes, and relations). For example, compositional understanding can distinguish between "there is a blue towel in a white bathroom" and "there is a white towel in a blue bathroom". However, it is still a difficult task for visual language models to understand these scenes.

[0068] The composition understanding test dataset consists of triplets $(I, T^{t}, T^{n})$ or quadruplets $(I^{t}, I^{n}, T^{t}, T^{n})$, depending on the paradigm of the task. Such tests are objective and diverse, and can be used to evaluate a range of composition understanding capabilities from objects to attributes to relations. For the image-to-text paradigm, given an image, a piece of correct text, and a piece of incorrect text represented by recombining the correct text, the model's ability to match the image and the correct text is tested. For the text-to-image paradigm, given a piece of text, a matching correct image, and a wrong image represented by recombining the correct image, the model's ability to match the text and the correct image is tested. SugarCrepe is a small but complex benchmark for the image-to-text paradigm, covering 7 types of fine-grained hard negative samples and improving the bias in CREPE.

[0069] The present invention discloses a hierarchical approach to composition understanding in the image-to-text paradigm. For example, SugarCrepe is an example of such a dataset, which uses ChatGPT to generate credible and fluent hard negative samples, and then eliminates artifacts through manual verification and adversarial improvement mechanisms to minimize the identified biases. For ease of disclosure, various aspects of the present invention may be described by processing data related to SugarCrepe. However, those skilled in the art will understand that various aspects of the present invention may be performed using inputs from other paradigms.

[0070] Composition understanding can be simplified to a retrieval problem, finding the text that best matches the reference image. This process corresponds to the alignment of two different modalities, text and image. The model is trained on large-scale text-image pair data. These text-image pair data sets can be obtained directly from the Internet. The task of training this type of model mainly focuses on cross-modal contrastive learning.

[0071] Taking CLIP as an example, the core idea of ​​the CLIP model is that, given an image, the model can find the text description that best matches it, and vice versa. It first uses a visual encoder to process the input image and convert it into a high-dimensional vector representation; at the same time, it uses a text encoder to process the input text and generate the corresponding text embedding. By maximizing the similarity between correct image-text pairs and minimizing the similarity between incorrect pairs, the model gradually learns how to effectively associate visual and language information. This contrastive learning strategy enables CLIP to not only perform well in multimodal retrieval tasks, but also be generalized to various downstream tasks.

[0072] Visual language models are mainly trained and evaluated on large-scale datasets with rich compositional structures in images and text, but training on these datasets is not sufficient to address the lack of compositional understanding. Visual language models behave like bags of words, i.e., when they have a poor understanding of relations, they may make mistakes when associating objects with other attributes and show a severe lack of order sensitivity.

[0073] Although it is possible to retrain a visual language model, existing methods focus on fine-tuning visual language models such as CLIP due to the high training cost and the small number of datasets for compositional understanding. Neg-CLIP was obtained by fine-tuning on a dataset containing hard negative captions, while Structure-CLIP was trained on the VG-Attribution dataset (a dataset that exchanges hard negative types). However, creating datasets for compositional understanding is expensive because this type of data is difficult to obtain directly over the Internet. This is a resource-intensive task, especially for hard negative images, which are currently mainly manually curated and have a small data size. On the other hand, previous studies have shown that visual language models fine-tuned with hard negatives rely on expensive supervised data, which causes the models to inadvertently overfit.

[0074] Attached Figure 1 The flowchart of the composition understanding method based on text inversion 100 is shown. A composition understanding method based on text inversion is used to test the ability of a model to match correct text and image, and the method comprises the following steps:

[0075] S1: Data input step 102, receiving an input data pair, wherein the data pair includes an image, a matching text, and an erroneous text represented in a reassembled correct text;

[0076] S2: data conversion step 104, converting the input image into a pseudo-token using an optimization-based text inversion algorithm;

[0077] In one embodiment, an image is encoded using only a single pseudo-token, which contains two aspects of information. On the one hand, the pseudo-token can accurately represent the content of the image. On the other hand, the pseudo-token can interact with other tokens. This operation can use optimization-based text inversion or neural networks. However, those skilled in the art will understand that an image can be mapped to multiple pseudo-tokens;

[0078] S3: data concatenation step 106, concatenating the original text and the pseudo-token using different templates;

[0079] S4: vector generation step 108, converting the concatenated text into a feature vector using a neural network;

[0080] In one embodiment, the conversion process can be performed by a neural network. For example, the image can be provided to the Image Encoder of CLIP;

[0081] S5: Similarity calculation step 110, respectively calculating the cosine similarity between the correct text and the incorrect text after splicing and the image;

[0082] S6: Matching judgment step 112, if the similarity between the correct text after splicing and the image is greater than the similarity between the incorrect text after splicing and the image, the match is correct, otherwise the match is incorrect.

[0083] Attached Figure 2 A flowchart 200 of mapping a single image to a pseudo-token using text inversion is shown. This process converts the input image into the corresponding pseudo-token through parallel text encoding and image encoding.

[0084] First, the input image 202A is passed through an image encoder 206 to obtain image features, and the image features are then subjected to a text inversion operation 208 to obtain a pseudo-token corresponding to the image. The text inversion operation 208 can be either an iterative algorithm based on optimization or a special neural network. In the description of the present invention, the character "$" is used to represent this pseudo-token, but those skilled in the art will understand that other characters can be used to represent it. The present invention then concatenates the pseudo-token with the input text fragment 202B to form a preprocessed text 204, such as "a photo of $ that {inputcaption}". The preprocessed text 204 will be processed by a tokenizer 210A to divide the text into multiple tokens, each of which corresponds to a unique number, as shown in the attached figure. Figure 2 508, 701, 73 and the special symbol "*" shown in FIG. Then, these tags are sent to an embedding lookup table 210B, which maps each tag to a high-dimensional vector representation, for example, tag 508 is mapped to a vector , marking 701 is mapped to vector , and “*” is mapped to a special embedding vector The special embedding vector here is not actually used directly, it will be replaced by the vector from the text inversion operation 208. Those skilled in the art will understand that "*" is also a number, and its value is determined by the encoder of the model and the selected pseudo-token character.

[0085] After the embedding lookup is completed, these high-dimensional embedding vectors are input into the text deep learning model 210C (e.g., Transformer), which converts the serialized embedding vectors into context-related semantic representations. This representation captures the rich semantic information in the input text on the one hand, and contains the composition information of the image on the other hand.

[0086] From the attached Figure 2 As can be seen in FIG, the word segmenter 210A, the embedding lookup table 210B and the text deep learning model 210C together constitute the text encoder 210. The text deep learning model here is frozen, and the present invention does not make any parameter updates to it. The embodiment of the present invention relies on the visual language model CLIP, but those skilled in the art should understand that any other type of visual language model can be used to implement the same mapping process.

[0087] Attached Figure 3 The first stage training method of the composition understanding model based on text inversion is shown in the figure. Figure 4 A schematic diagram of the second-stage training method of the composition understanding model based on text inversion is shown.

[0088] For the first stage, given an image I, the training method finds Specifically, the present invention represents the image content that we want to know as the character "$". And intervenes in the embedding process, using the newly learned embedding value Replace the vectors associated with the labeled characters. Essentially, this is injecting image knowledge into the model’s vocabulary, and the model can form new sentences that contain image information just like it would with any other word.

[0089] To regulate the optimization process, contexts derived from the PALAVRA template are randomly sampled , for example, ``a photo of $'' and feed it into the CLIP text encoder , thus obtaining In order to make the pseudo-labeling have the ability to distinguish structural representations rather than simply mapping visual atoms, we define a set of negative sample prompts to modify the original text, for example, "a photo without $", which is fed into the CLIP text encoder , get negative text features Given an image I, obtain image features from the CLIP image encoder .

[0090] The goal of this stage is to obtain a pseudo-token with composition related to Figure I , so in the common embedding space of CLIP, this stage makes the distance between the image and the positive text features smaller than the distance between the image and the negative text. To achieve the purpose of this stage, this stage uses a CLIP-based ternary loss function:

[0091]

[0092] However, since image features and word token embeddings are in their own spaces, This forces the pseudo-labels into a sparse region of the CLIP label embedding space. It is not enough to generate pseudo-tokens that interact with other words in the CLIP dictionary. To overcome this problem, this stage uses regularization techniques on pseudo-tokens to restrict them to the CLIP token embedding manifold, thereby enhancing their reasoning ability.

[0093] For an unlabeled image I, the present invention uses CLIP's zero-shot classification capability to classify it into about 20K categories of the OpenImage V7 dataset, where a hyperparameter k is introduced to represent the k different visual atoms that are most similar to image I. In the iterative process, by prompting the lightweight GPT model with "A photo of a {class}that", one of the k categories that are most similar to image I is randomly sampled, and GPT continues to write the text according to the prompt to obtain a title with the atom In addition, replace {class} with the character $ to get a title with a pseudo word After that, the CLIP text encoder is used to obtain text features. and Finally, the present invention uses cosine loss to minimize the gap between the two features:

[0094]

[0095] For negative text, the same loss is minimized and This loss takes into account the context of the text, while The semantics of the image is associated, thus enhancing The ability to interact with human-generated text. In the first stage, the total loss of the invention is:

[0096]

[0097] The pseudo-labels extracted in the first stage are very effective because they not only contain the visual information of the image, but also help the model improve its composition understanding ability like words. However, the execution time of text inversion is difficult to ignore due to the characteristics of iterative optimization. To solve this problem, the present invention designs a second stage of training to obtain a plug-and-play model through knowledge distillation that can invert any image. Therefore, the present invention considers a text inversion network with three linear layers, and uses a GELU activation function and a dropout layer after each linear layer.

[0098] During the second stage of training, the present invention defines a small batch containing N input pairs as ,in It is an unlabeled image dataset. On the one hand, each image is mapped to a pseudo-token using optimized text inversion, and finally a set of pseudo-tokens are obtained. Although this process is time-consuming, it is only a one-time process. On the other hand, the knowledge obtained through optimized text inversion needs to be refined into , for each image, use CLIP's visual encoder to extract its features , and then predict the pseudo-token In this stage, the symmetric contrast loss is used to optimize this process as follows:

[0099]

[0100] in, Used to maximize pre-generated pseudo-tokens and Predicted pseudo-token The cosine distance between them is minimized while minimizing the similarity between different tokens. The formula is as follows:

[0101]

[0102] in is a temperature parameter.

[0103] To standardize The training of , uses the same regularization technique as the optimization-based text inversion. The total loss function for the weights is:

[0104]

[0105] in, and is the loss weight.

[0106] The present invention discloses a composition understanding method, system and model training method based on text inversion, which uses an optimized text inversion algorithm to convert an input image into a pseudo-token, splices the original text with the pseudo-token, converts the spliced ​​text into a feature vector, and calculates the similarity between the text and the image. The method improves the understanding ability of the composition understanding model while reducing the dependence on hard negative samples, improves the model interaction efficiency, and reduces time consumption.

[0107] Those skilled in the art may appreciate that the units of the various examples described in conjunction with the embodiments disclosed in the present invention may be implemented in the form of electronic hardware, or a combination of computer software and electronic hardware.

[0108] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0109] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. In addition, the functional units in the various embodiments of the present invention can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0110] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A composition understanding method based on text inversion, which is used to test the ability of a model to match correct text and images, and the method comprises the following steps: S1: a data input step, receiving an input data pair, wherein the data pair includes an image, a matching text, and an erroneous text represented in a reassembled correct text; S2: Data conversion step, using an optimization-based text inversion algorithm to convert the input image into a pseudo-token; S3: Data concatenation step, using different templates to concatenate the original text and pseudo-token; S4: vector generation step, converting the concatenated text into a feature vector using a neural network; S5: similarity calculation step, respectively calculating the cosine similarity between the correct text and the wrong text after splicing and the image; S6: Matching judgment step, if the similarity between the correct text and the image after splicing is greater than the similarity between the incorrect text and the image after splicing, the match is correct, otherwise the match is incorrect.

2. The composition understanding method according to claim 1, characterized in that: The step S2 specifically includes: S21: feature extraction step, the input image is passed through an image encoder to obtain image features; S22: pseudo-token generation step, the image features are processed through an optimized iterative algorithm to obtain a pseudo-token corresponding to the image.

3. The composition understanding method according to claim 1, characterized in that: The step S4 specifically includes: S41: preprocessing text generation step, concatenating the pseudo-token with the input text to form a preprocessing text; S42: text segmentation and tagging step, the preprocessed text is processed by a word segmenter, and the text is segmented into multiple tokens, each token corresponds to a unique number; S43: a text token mapping step, wherein the token is sent to an embedding lookup table, which maps each token into a high-dimensional vector representation; S44: Model vector conversion step. After completing the embedding lookup, these high-dimensional vectors are input into the text deep learning model. The text deep learning model converts the serialized embedding vectors into context-related semantic representations.

4. The composition understanding method according to claim 3, characterized in that: The text deep learning model is a transformer, and the word segmenter, embedding lookup table and text deep learning model together constitute a text encoder.

5. A composition understanding system based on text inversion, the system executing the composition understanding method according to any one of claims 1 to 4, characterized in that: The system specifically includes the following modules: A data input module, used for receiving an input data pair, wherein the data pair includes an image, a matching text, and an erroneous text represented in a reassembled correct text; The data conversion module is used to convert the input image into pseudo-token using an optimized text inversion algorithm; Data splicing module, used to splice original text and pseudo-token using different templates; A vector generation module is used to convert the concatenated text into a feature vector using a neural network; A similarity calculation module is used to calculate the cosine similarity between the correct text and the wrong text after splicing and the image respectively; The matching judgment module is used to judge that if the similarity between the correct text after splicing and the image is greater than the similarity between the incorrect text after splicing and the image, the match is correct, otherwise the match is incorrect.

6. The composition understanding system according to claim 5, characterized in that: The data conversion module specifically includes: A feature extraction module is used to obtain image features by passing the input image through an image encoder; In the pseudo-token generation module, the image features are processed through an optimized iterative algorithm to obtain the pseudo-token corresponding to the image.

7. The composition understanding system according to claim 5, characterized in that: The vector generation module specifically includes: The preprocessing text generation module is used to concatenate pseudo-token with the input text to form preprocessing text; The text segmentation and tagging module is used to process the preprocessed text through a word segmenter and segment the text into multiple tokens, each of which corresponds to a unique number; A text token mapping module, configured to feed the tokens into an embedding lookup table, which maps each token into a high-dimensional vector representation; Model vector conversion module,After completing the embedding lookup, these high-dimensional vectors are input into the text deep learning model, which converts the serialized embedding vectors into context-related semantic representations.

8. The composition understanding system according to claim 7, characterized in that: The text deep learning model is a transformer, and the word segmenter, embedding lookup table and text deep learning model together constitute a text encoder.

Images