Visual language model training method, image tag prediction method, and electronic device

Abstract

Embodiments of the present disclosure provide a visual language model training method, an image tag prediction method, and an electronic device. According to the embodiments of the present disclosure, encoding parameters respectively corresponding to an image encoder and a text encoder in a first visual language model are fixed, on the basis of initial values of visual prompts and text samples respectively corresponding to a plurality of image classification tags, a second visual language model is generated by training the first visual language model, and the visual prompts respectively corresponding to the plurality of image classification tags are mined and learned, thereby enhancing the visual representation capabilities of visual language models. Next, on the basis of the learned visual prompts, text prompts, and the text samples, a target visual language model is further trained by training the second visual language model, and adapter parameters of model adapters respectively configured for the image encoder and the text encoder are adjusted, to collaboratively optimize image feature extraction and text feature extraction. This process allows for the transfer of visual knowledge from visual prompts to text prompts.

Description

Training of visual language models, image label prediction methods, and electronic devices.

[0001] This disclosure claims priority to Chinese Patent Application No. 202411066755.1, filed with the China Patent Office on August 2, 2024, entitled “Training of Visual Language Model, Prediction Method of Image Labels and Electronic Device”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of computer technology, and in particular to a method for training a visual language model, a method for predicting image labels, an apparatus, an electronic device, a storage medium, and a computer program product. Background Technology

[0003] In online trading platforms, product tags are used in multiple business scenarios. For example, in search scenarios, when retrieving products based on key information entered by the user, it involves matching the key information used for retrieval with product tags. Similarly, when adding a batch of products to the database, it is necessary to identify the product tags corresponding to the products so that the virtual product objects can be associated with the product tags when storing them in the database.

[0004] In practical applications, an image may include one or more different objects. The classification of product objects can be achieved by multi-label classification based on the images of product objects. When using a model to implement multi-label tasks, in order to ensure the accuracy of classification prediction results, one approach is to provide a large number of labeled image samples for training the model, which requires high manual costs. Another approach is to add text prompts as model input. However, the representation of visual information in images by text information is still limited. Summary of the Invention

[0005] This disclosure provides a method for training a visual language model, a method for predicting image labels, an apparatus, an electronic device, a storage medium, and a computer program product to solve one or more of the aforementioned technical problems.

[0006] In a first aspect, embodiments of this disclosure provide a method for training a visual language model, comprising: fixing the encoding parameters corresponding to the image encoder and text encoder in a pre-trained first visual language model, and based on the initial values ​​of visual cues corresponding to multiple image classification labels and text samples, training the first visual language model to learn the visual cues corresponding to the multiple image classification labels, thereby generating a second visual language model; and based on the learned visual cues, text cues, and text samples, training the second visual language model to optimize the learned visual cues, text cues, and adapter parameters of the model adapters configured for the image encoder and text encoder, thereby generating a target visual language model; wherein the target visual language model is used to identify multiple image classification labels corresponding to an image.

[0007] Secondly, embodiments of this disclosure provide an image label prediction method applied to a target visual language model. The method includes: obtaining text prompts corresponding to a target image; predicting multiple image classification labels corresponding to the target image based on the target image and the corresponding text prompts; and generating the target visual language model using the aforementioned training method for visual language models.

[0008] Thirdly, this disclosure provides a data configuration method for a product object, the method comprising: acquiring at least one product image corresponding to the product object; calling a target visual language model to generate a product category label corresponding to the at least one product image; adding the product category label to the product object so that the product object can be marked according to the product category label when the product is retrieved or the product is added to the database; the target visual language model is generated by the above-described training method for the visual language model.

[0009] Fourthly, an electronic device includes a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method described above.

[0010] Fifthly, a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0011] Sixthly, a computer program product includes computer instructions that, when executed by a processor, implement the method described above.

[0012] According to embodiments of this disclosure, a pre-trained first visual language model is used. The encoding parameters of the image encoder and text encoder in the first visual language model are fixed, and initial values ​​for visual cues are set. Based on the initial values ​​of visual cues corresponding to various image classification labels and text samples, a second visual language model is generated by training the first visual language model. During this process, the visual cues corresponding to various image classification labels are mined and learned through the well-aligned feature space of the pre-trained visual language model, enhancing the visual representation capability of the visual language model. This eliminates the need to acquire visual knowledge through a large number of labeled image samples, thus reducing the need for extensive labeled image sample preparation during model training. Next, based on the learned visual cues, text cues, and text samples, the target visual language model is further trained by training the second visual language model. During this process, the adapter parameters of the model adapters configured in the image encoder and text encoder are adjusted to collaboratively optimize the extraction of image features and text features, while simultaneously optimizing the text cues and the learned visual cues. This process transfers the visual knowledge of the visual cues to the text cues, enhancing the visual representation capability of the text cues. The target visual language model trained through the above process can significantly improve the accuracy of prediction results when used to identify various image classification labels corresponding to an image.

[0013] The embodiments of this disclosure divide the training process of the visual language model into a visual cue learning stage and a visual knowledge transfer stage. In the visual cue learning stage, visual cues corresponding to various image classification labels are learned. In the visual knowledge transfer stage, the visual knowledge of the visual cues is transferred to the text cues. The two stages are performed sequentially during the training process. Compared with training the two stages simultaneously, this avoids the problem of convergence difficulties for randomly initialized visual cues.

[0014] This disclosure also provides a text sample generation scheme to supplement manually annotated text samples. By constructing a first prompt word using image classification labels and a first query template for generating pseudo-text samples, and inputting it into a large language model to generate pseudo-text samples, a second prompt word is constructed using a second query template for detecting pseudo-text samples in order to filter out unreliable text generated by the large language model. The second prompt word is then input into the large language model to query whether the generated pseudo-text samples are usable, thus ensuring the reliability of the text samples.

[0015] The above description is only an overview of the technical solution of this disclosure. In order to better understand the technical means of this disclosure, it can be implemented in accordance with the contents of the specification. In order to make the above and other objects, features and advantages of this disclosure more obvious and understandable, specific embodiments of this disclosure are given below. Attached Figure Description

[0016] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this disclosure and should not be construed as limiting the scope of this disclosure.

[0017] Figure 1 shows a schematic diagram of the first stage of the training method for the visual language model according to an embodiment of the present disclosure;

[0018] Figure 2 shows a schematic diagram of the second stage of the training method for the visual language model according to an embodiment of the present disclosure;

[0019] Figure 3 shows a flowchart of a training method for a visual language model according to an embodiment of the present disclosure;

[0020] Figure 4 shows a flowchart of an image tag prediction method according to an embodiment of the present disclosure;

[0021] Figure 5 shows a flowchart of a data configuration method for a commodity object according to an embodiment of the present disclosure;

[0022] Figure 6 shows a structural block diagram of a training device for a visual language model according to an embodiment of the present disclosure;

[0023] Figure 7 shows a structural block diagram of an image tag prediction device according to an embodiment of the present disclosure;

[0024] Figure 8 shows a structural block diagram of a data configuration device for a commodity object according to an embodiment of the present disclosure;

[0025] Figure 9 shows a block diagram of an electronic device used to implement embodiments of the present disclosure. Detailed Implementation

[0026] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of this disclosure. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0027] To facilitate understanding of the technical solutions of the embodiments of this disclosure, the related technologies of the embodiments of this disclosure are described below. The following related technologies are optional solutions and can be combined with the technical solutions of the embodiments of this disclosure in any way, and all of them fall within the protection scope of the embodiments of this disclosure.

[0028] Online trading platforms require the addition of category tags to products, especially in cross-border e-commerce scenarios. Due to the management of products from different regions, significant differences in image styles, and the use of different languages, product classification and recognition become more complex, placing higher demands on model accuracy. Using visual language models to perform multi-label classification tasks requires a large amount of labeled visual data for model training or the addition of text prompts, but recognition accuracy still needs improvement.

[0029] This disclosure employs a visual language model to perform multi-label classification tasks, and improves the original visual language model training process in several aspects. Generally, the training process of the visual language model in this disclosure is divided into two stages. The first stage is the visual cue learning stage, where the encoder parameters corresponding to the image encoder and text encoder of the pre-trained visual language model are fixed. That is, the parameters of the two encoders are not adjusted in this stage; instead, the well-aligned feature space of the image and text already possessed by the visual language model is fully utilized. The goal of this stage is to learn visual cues corresponding to various image classification labels. The second stage is the visual knowledge transfer stage, where the encoder parameters corresponding to the image encoder and text encoder of the visual language model trained in the first stage are still fixed. The parameters of the model adapters configured for the image encoder and text encoder are adjusted, and the extraction of image features and text features is collaboratively optimized. Simultaneously, the text cues and the learned visual cues are optimized. Through the optimization process in the second stage, the visual knowledge of the visual cues learned in the first stage can be transferred to the text cues, enhancing the visual representation ability of the text cues.

[0030] Figure 1 illustrates the first stage of the training method for the visual language model according to an embodiment of this disclosure. In this stage, diverse and comprehensive visual knowledge for each image classification label is accurately learned. For multi-label image classification tasks, as shown in Figure 1, various image classification labels for cross-border online trading platforms include person, bicycle, cup, vase, car, etc. Initial values ​​for visual cues are pre-configured for each classification label; these can be randomly generated values ​​or prior values. The parameters corresponding to the image encoder and text encoder are frozen (parameters are frozen in the figure). The visual cues are the parameters that need to be learned, hence the changing parameters shown in the figure.

[0031] The visual cues set for each image classification label are not combined with any explicit visual or text labels. The visual language model itself has a text-image alignment feature space, in which image modalities have extensive diversity. The visual knowledge learned by training the model is reflected in the visual cues. Since the visual cues in this embodiment are not pre-configured and directly used but are learned, they are not true visual cues, but rather learned pseudo-visual cues or simulated visual cues. A visual cue can be defined as P = [p1, p2, p3, ..., p N ], where p i ∈R H×W×3 Let H and W represent the visual cue for the i-th category. H and W are of equal size. N is the number of image category labels. The number of visual cues is equal to the number of image category labels.

[0032] Next, visual cues are learned using the feature space aligned with the visual language model, the collected text training data, and the frozen image encoder and text encoder. The learning process for visual cues can be formalized as follows: <Ω 对齐 ,T,φ,ψ>→ Where T represents the tagged text data or pseudo-text data collected by LLM, Ω 对齐 The feature space represents the alignment, and φ(·) and ψ(·) refer to the frozen text encoder and image encoder, respectively.

[0033] As shown in Figure 1, noun filtering was also performed on the text sample data. For example, in the text sample "A compact car with bicycles mounted on the roof," noun filtering revealed the nouns "car" and "bicycle." The rightmost column of the visual cues in the figure contains "car" and "bicycle" in its corresponding image classification labels. Therefore, the visual cues with the corresponding image classification labels "car" and "bicycle" can be paired with the text sample as a positive sample pair, while the text sample can be paired with other visual cues that do not contain "car" and "bicycle" as a negative sample pair. This word segmentation and matching method completes the construction of sample pairs. Based on these sample pairs, the model can be trained iteratively to learn the visual cues.

[0034] Figure 2 illustrates the second stage of the training method for the visual language model according to an embodiment of this disclosure. The learned visual cues are processed by an image encoder to obtain corresponding image features, while text data and text cues are processed by corresponding text encoders to extract text features. The image encoder and text encoder each use corresponding model adapters with identical structures, for example, consisting of two fully connected layers, an activation function, and a residual connection layer; this is referred to as a dual adapter. The text cues and text samples use the same model adapter, meaning the text adapter is a parameter-sharing network for both text cues and text samples. In the second stage of model training, the constructed loss function can be cross-constructed around text samples, text cues, and visual cues. This includes a first loss function representing the feature similarity between visual cues and text samples, a second loss function representing the feature similarity between visual cues and text cues, and a third loss function representing the feature similarity between text samples and text cues. This cross-construction of loss functions allows visual cues to influence the text as much as possible. During training, by optimizing the adapter parameters of the model adapters configured for visual cues, text cues, and the text encoder and image encoder respectively, the visual knowledge learned in the first stage can be transferred to the text cues.

[0035] Similar to the definition of visual cue P above, text cue can be defined as: R i =[r1,r2,...,r M ,g i S = [R1, R2, ..., R] N ]. Among them, g i r represents the text features of the i-th type of label. i Here, M represents the learnable context vector of the text prompt, and M represents the number of text prompts. The text encoder extracts text features corresponding to the text prompts and text samples, while the image encoder extracts image features corresponding to the visual prompts, as follows: I = φ(S), H = φ(T), V = ψ(P). Further processing using model adapters corresponding to the image encoder and the text encoder yields: U = (1-λ)g(V) + λV, G = (1-λ)h(H) + λH, E = (1-λ)h(I) + λI, where g(·) and h(·) represent the adapter functions for the image and text modalities, respectively, and λ∈[0,1] represents the weights between the features extracted by the adapter module and the image / text features. After extracting these features, a loss function can be used to train the model. The loss function constructed using similarity is used to preserve the similarity between the visual and text prompts, thus transferring the visual knowledge learned in the first stage to the text prompts.

[0036] The similarity matrix in the target loss function can be constructed as UE T ∈R N×N The visual and textual cues of N categories are constructed into an identity matrix, and the size of the similarity matrix is ​​equal to the number of image classification labels N. The objective loss function can be constructed as follows:

[0037] in, and represents the Softmax normalized similarity from visual cues to text cues and from text cues to visual cues, respectively. τ represents the temperature parameter. CE This represents the cross-entropy loss. The similarity between text prompts and visual prompts is considered realistic. If the text prompt t i and visual cues p j If they belong to the same category, then otherwise and The definition is the same. Ranking loss is used to obtain L. visual and L text (Similar to the similarity L below) PVP (Calculation formula), specifically, using L visual To measure the difference between text samples and visual cues, L visual It can be represented as:

[0038] Similarly, L text The difference between the text sample and the text prompt can be expressed as:

[0039] Finally, we can obtain the target loss function as: L = L vtc +L visual +L text , where L vtc The difference between text cues and visual cues is measured. In the second phase of training, the encoding parameters of the image encoder and text encoder are fixed, and the target loss function is optimized to learn the adapter parameters for visual cues, text cues, and both adapters.

[0040] The visual language model training process of this disclosure can be completed on a local or cloud computing device. The trained visual language model can be stored on the computing device or in the cloud for remote access or distributed to a terminal. The process of configuring product images can be completed on the merchant's end or the terminal of the transaction platform, where the visual language model is accessed from the terminal, local computing device, or cloud to obtain product category tags. For the visual language model training process, the executing entity can be an application, service, instance, software functional module, virtual machine (VM), container, or cloud server, or a hardware device (such as a server or terminal device) or hardware chip (such as a CPU, GPU, FPGA, NPU, AI accelerator card, or DPU) with data processing capabilities. The devices for visual language model training and image label prediction can be deployed on the computing devices of the application providers offering the corresponding services or on cloud computing platforms providing computing power, storage, and network resources. The cloud computing platform can provide services in the following models: IaaS (Infrastructure as a Service), PaaS (Platform as a Service), SaaS (Software as a Service), or DaaS (Data as a Service). Taking SaaS (Software as a Service) as an example, the cloud computing platform can utilize its own computing resources to provide the functions of visual language model training or image label prediction. The specific application architecture can be built according to service requirements. For example, the platform can provide application providers or individuals using platform resources with model-based construction services, and further invoke the models and implement online or offline image label prediction functions based on image label prediction requests submitted by relevant clients or servers.

[0041] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0042] The technical solutions of this disclosure and how they solve the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0043] This disclosure provides a method for training a visual language model. Figure 3 shows a flowchart of a visual language model training method 300 according to an embodiment of this disclosure. The method 300 may include:

[0044] In step S301, the encoding parameters corresponding to the image encoder and text encoder in the pre-trained first visual language model are fixed. Based on the initial values ​​of visual cues corresponding to various image classification labels and text samples, the first visual language model is trained to learn the visual cues corresponding to the various image classification labels, thereby generating a second visual language model.

[0045] In the first stage of learning visual cues, a second visual language model is generated by training a first visual language model. During this process, the constructed loss function can be calculated based on the similarity between the text sample and the visual cue. For an input text sample T, the pre-trained visual language model projects the last "...". <eos>"Using the features of the labels to obtain global text features, and obtaining the visual cues corresponding to each image classification label of P through a visual attention pool, which are represented as H=φ(T), V=ψ(P), H∈R" B×D , representing the normalized global text features extracted in batches, V∈N N×D Let represent the standardized global image features of N visual cues, for a specific text t in the batch. i ∈T, text t i The formula for calculating the similarity with visual cues is s. ij = <h i ,v j >, h i ∈H, representing text t i Global text features, v j ∈V, representing the global image features of the j-th visual cue.

[0046] In the first phase of training, ranking loss can be used to measure the similarity scores between text and visual cues, as well as the difference between the classification labels of the text samples. Visual cues can be learned using the loss function in the following formula: Where c+ and c- are the positive and negative labels of the sample pair, s ki and s kj These are the similarity scores for positive and negative sample pairs described in the similarity calculation formula above, where m is used to measure the difference between each pair of predicted values.

[0047] The aforementioned pre-trained visual language model is referred to as the first visual language model, specifically CLIP (Contrastive Language-Image Pre-training), which is a pre-trained model based on contrastive text-image pairs. Its training process is roughly divided into three stages: using the Transformer structure or a series of variants as the basic structure to extract text features; using CNN models such as ResNet or structures such as Transformer to extract image features; and further establishing a correlation between the information of these two modalities, namely, aligning the features extracted from the two modalities of text and image.

[0048] In the process of training the first visual language model, there are various methods for feature alignment. One approach is contrastive learning, which trains the model to bring paired images and text closer together in the feature space while distancing unrelated samples. Examples include image-based contrastive learning, image-and-text contrastive learning, or image-text-label contrastive learning. Another approach is generative task-based alignment, where the model is trained to perform image generation, text generation, or cross-modal generation to learn semantic features. Examples include mask-based image modeling, mask-based language modeling, or mask-based cross-modal modeling. Finally, feature alignment can be based on alignment objectives, matching features between images and text. Examples include global image-and-text matching or local image region-and-word matching.

[0049] Because the distribution of images and text varies across different business scenarios, pre-trained visual language models require adjustments when used in specific business scenarios. One approach is based on cue-based adjustments, which involves adding text cue adjustments during model input, allowing the model to adapt to the scenario's needs without fine-tuning. Another approach is based on feature adapter adjustments, which involves adding a lightweight model adapter to the visual language model architecture for fine-tuning. In this embodiment, in addition to image input, text cue input is also included during model input. Furthermore, a model adapter that adapts to both image encoders and text encoders is added to the visual language model architecture. After learning visual cue, the model is trained again based on the learned visual cue, text cue, and text samples. During training, in addition to optimizing the learned visual cue and text cue, the parameters of the model adapter are also adjusted to ensure that the model adapter is trained simultaneously.

[0050] In step S302, based on the learned visual cues, text cues, and text samples, the second visual language model is trained to optimize the adapter parameters of the learned visual cues, text cues, and the model adapters configured for the image encoder and text encoder, respectively, to generate a target visual language model; wherein, the target visual language model is used to identify multiple image classification labels corresponding to the image.

[0051] According to embodiments of this disclosure, a pre-trained first visual language model is used. The encoding parameters of the image encoder and text encoder in the first visual language model are fixed, and initial values ​​for visual cues are set. Based on the initial values ​​of visual cues corresponding to various image classification labels and text samples, a second visual language model is generated by training the first visual language model. During this process, the good image and text feature alignment space of the pre-trained visual language model is used to mine and learn visual cues corresponding to various image classification labels, enhancing the visual representation capability of the visual language model. This eliminates the need to acquire visual knowledge through a large number of labeled image samples, thus reducing the need for extensive labeled image sample preparation during model training. Next, based on the learned visual cues, text cues, and text samples, the target visual language model is further trained by training the second visual language model. During this process, the adapter parameters of the model adapters configured in the image encoder and text encoder are adjusted to collaboratively optimize image and text feature extraction, while simultaneously optimizing both the text cues and the learned visual cues. This process transfers the visual knowledge of the visual cues to the text cues, enhancing the visual representation capability of the text cues. The target visual language model trained through the above process can significantly improve the accuracy of prediction results when used to identify various image classification labels corresponding to an image.

[0052] The embodiments of this disclosure divide the training process of the visual language model into a visual cue learning stage and a visual knowledge transfer stage. In the visual cue learning stage, visual cues corresponding to various image classification labels are learned. In the visual knowledge transfer stage, the visual knowledge of the visual cues is transferred to the text cues. The two stages are performed sequentially during the training process. Compared with training the two stages simultaneously, this avoids the problem of convergence difficulties for randomly initialized visual cues.

[0053] In one possible implementation, before training a first visual language model to learn the visual cues corresponding to various image classification labels based on initial values ​​of visual cues and text samples corresponding to multiple image classification labels, positive and negative sample pairs can be constructed based on the similarity relationship between the initial visual cues and text samples. In one example, the sample construction process can be implemented through noun filtering. When constructing positive and negative sample pairs based on the similarity relationship between the initial visual cues and text samples, the text samples can first be segmented, and the corresponding image classification labels can be determined. That is, given a visual cue and a text sample, noun filtering is performed on the text sample to obtain the nouns included in the text sample. If the noun is included in the word set corresponding to a certain image classification label, then the text sample corresponding to that noun can be considered as a positive sample pair with the visual cues corresponding to the image classification label of that word set; otherwise, it is a negative sample pair. Furthermore, the initial visual cues corresponding to the image classification labels related to the segmentation and the corresponding text samples are constructed as positive sample pairs, while the initial visual cues corresponding to the image classification labels unrelated to the segmentation and the corresponding text samples are constructed as negative sample pairs.

[0054] In one possible implementation, the step of learning the visual cues corresponding to the various image classification labels by training the first visual language model based on the initial values ​​of the visual cues and text samples corresponding to the various image classification labels includes: learning the visual cues corresponding to the various image classification labels by training the first visual language model based on the positive and negative sample pairs of the constructed initial visual cues and text samples.

[0055] In one possible implementation, the target loss function constructed when training the second visual language model includes: a first loss function representing the feature similarity between visual cues and text samples, a second loss function representing the feature similarity between visual cues and text cues, and a third loss function representing the feature similarity between text samples and text cues.

[0056] In the visual knowledge transfer stage, this disclosure constructs multiple loss functions, which respectively characterize the feature similarity between visual cues and text samples, the feature similarity between visual cues and text cues, and the feature similarity between text samples and text cues. By controlling the value of the loss function, the model is optimized, thereby controlling the transfer of visual knowledge from visual cues to text cues and optimizing the adapter parameters.

[0057] In one possible implementation, the adapter parameters for optimizing the learned visual cues, text cues, and model adapters configured for the image encoder and text encoder, respectively, by training the second visual language model include:

[0058] The second visual language model is trained iteratively, and the adapter parameters of the visual cues, text cues, and model adapter are adjusted in each training round so that the target loss function used to train the second visual language model satisfies the set optimization conditions.

[0059] In one possible implementation, the text sample includes at least a pseudo-text sample, and the method further includes: constructing a first prompt word based on the image classification label and a first query template for generating pseudo-text samples, and using the first prompt word to input a large language model to generate pseudo-text samples; constructing a second prompt word using a second query template for detecting pseudo-text samples, and using the second prompt word to input a large language model to query whether the generated pseudo-text samples are available; and extracting available pseudo-text samples.

[0060] This disclosure also provides a text sample generation scheme to supplement manually annotated text samples. By constructing a first prompt word using image classification labels and a first query template for generating pseudo-text samples, and inputting it into a large language model to generate pseudo-text samples, a second prompt word is constructed using a second query template for detecting pseudo-text samples in order to filter out unreliable text generated by the large language model. The second prompt word is then input into the large language model to query whether the generated pseudo-text samples are usable, thus ensuring the reliability of the text samples.

[0061] The following provides an application example after model training from the perspective of a target visual language model. This disclosure provides an image label prediction method. Figure 4 shows a flowchart of an image label prediction method 400 according to an embodiment of this disclosure, applied to a target visual language model. The method 400 may include:

[0062] In step S401, the text prompt corresponding to the target image is obtained. In step S402, multiple image classification labels corresponding to the target image are predicted based on the target image and the corresponding text prompt; the target visual language model is generated using the visual language model training method of the above embodiment.

[0063] In one possible implementation, the target visual language model includes a text encoder and an image encoder. The step of predicting multiple image classification labels corresponding to the target image based on the target image and the corresponding text prompt includes: extracting image features of the target image using the image encoder, and extracting text features corresponding to the text prompt using the text encoder; wherein the image features and text features correspond to an aligned feature space; and predicting multiple image classification labels of the target image based on the image features and text features.

[0064] Next, considering specific application scenarios of product objects, the role of the trained target visual language model and the corresponding predicted image classification labels in practical application scenarios is given. This disclosure provides a data configuration method for product objects. Figure 5 shows a flowchart of a data configuration method 500 for product objects according to an embodiment of this disclosure. This method 500 may include:

[0065] In step S501, at least one product image corresponding to the product object is obtained. In step S502, the target visual language model is invoked to generate product category tags corresponding to the at least one product image. In step S503, the product category tags are added to the product object so that the product object can be marked according to the product category tags when the product is retrieved or added to the database; the target visual language model is generated by the training method of the visual language model in the above embodiment.

[0066] Based on the image classification label generation process, this disclosure embodiment can also provide an image classification method, including: first, acquiring a target image; second, calling a target visual language model to generate an image classification label corresponding to the target image; wherein, the target visual language model is generated by the training method of the visual language model in the above embodiment.

[0067] Corresponding to the examples and method embodiments provided in this disclosure, this disclosure also provides a training device for a visual language model. Figure 6 shows a structural block diagram of a visual language model training device 600 according to an embodiment of this disclosure. The device 600 may include:

[0068] The visual cue learning module 601 is used to fix the encoding parameters corresponding to the image encoder and text encoder in the pre-trained first visual language model, and learn the visual cue corresponding to the various image classification labels by training the first visual language model based on the initial values ​​of the visual cue and text samples corresponding to the various image classification labels, thereby generating a second visual language model; the parameter optimization module 602 is used to optimize the learned visual cue, text cue, and adapter parameters of the model adapters configured for the image encoder and text encoder respectively by training the second visual language model based on the learned visual cue, text cue, and text samples, thereby generating a target visual language model; wherein, the target visual language model is used to identify the various image classification labels corresponding to the image.

[0069] In one possible implementation, the apparatus further includes: a sample construction module, configured to construct positive sample pairs and negative sample pairs based on the similarity relationship between the initial visual cues and the text samples, before learning the visual cues corresponding to the multiple image classification labels by training the first visual language model based on the initial values ​​of the visual cues and the text samples.

[0070] In one possible implementation, the sample construction module includes: a word segmentation submodule, used to perform word segmentation on the text samples and determine the image classification labels corresponding to the word segmentation; a first sample construction submodule, used to construct a positive sample pair between the initial visual cues corresponding to the image classification labels related to the word segmentation and the text samples corresponding to the word segmentation; and a second sample construction submodule, used to construct a negative sample pair between the initial visual cues of the image classification labels unrelated to the word segmentation and the text samples corresponding to the word segmentation.

[0071] In one possible implementation, the visual cue learning module is specifically used to learn the visual cues corresponding to the various image classification labels by training the first visual language model based on positive and negative sample pairs of the constructed initial visual cues and text samples.

[0072] In one possible implementation, the target loss function constructed when training the second visual language model includes: a first loss function representing the feature similarity between visual cues and text samples, a second loss function representing the feature similarity between visual cues and text cues, and a third loss function representing the feature similarity between text samples and text cues.

[0073] In one possible implementation, the parameter optimization module is specifically used to iteratively train the second visual language model and adjust the adapter parameters of the visual cue, text cue, and model adapter in each training round so that the target loss function used to train the second visual language model satisfies the set optimization conditions.

[0074] In one possible implementation, the text sample includes at least a pseudo-text sample, and the apparatus further includes: a pseudo-text sample generation module, configured to construct a first prompt word based on the image classification label and a first query template for generating pseudo-text samples, and use the first prompt word to input a large language model to generate pseudo-text samples; and a pseudo-text sample query module, configured to construct a second prompt word using a second query template for detecting pseudo-text samples, and use the second prompt word to input a large language model to query whether the generated pseudo-text samples are available, and extract available pseudo-text samples.

[0075] Corresponding to the examples and method embodiments provided in this disclosure, this disclosure also provides an image label prediction device applied to a target visual language model. Figure 7 shows a structural block diagram of an image label prediction device 700 according to an embodiment of this disclosure. The device 700 may include:

[0076] The text prompt acquisition module 701 is used to acquire the text prompt corresponding to the target image; the classification label prediction module 702 is used to predict multiple image classification labels corresponding to the target image based on the target image and the corresponding text prompt; the target visual language model is generated by the training method of the visual language model in the above embodiment.

[0077] In one possible implementation, the target visual language model includes a text encoder and an image encoder. The classification label prediction module is specifically used to extract image features of the target image using the image encoder, and to extract text features corresponding to the text prompt using the text encoder; wherein the image features and text features correspond to an aligned feature space; and to predict multiple image classification labels of the target image based on the image features and text features.

[0078] Corresponding to the examples and method embodiments provided in this disclosure, this disclosure also provides a data configuration device for a commodity object. Figure 8 shows a structural block diagram of a data configuration device 800 for a commodity object according to an embodiment of this disclosure. The device 800 may include:

[0079] The object image acquisition module 801 is used to acquire at least one product image corresponding to the product object; the category label generation module 802 is used to call the target visual language model to generate product category labels corresponding to the at least one product image; the label adding module 803 is used to add the product category labels to the product object so that the product object can be marked according to the product category labels when the product is retrieved or the product is put into the database; the target visual language model is generated by the training method of the visual language model in the above embodiment.

[0080] Based on the image classification label generation process, this disclosure embodiment can also provide an image classification device, including: an image acquisition module for acquiring a target image; and a classification label generation module for calling a target visual language model to generate an image classification label corresponding to the target image; wherein the target visual language model is generated by the training method of the visual language model in the above embodiment.

[0081] According to embodiments of this disclosure, a pre-trained first visual language model is used. The encoding parameters of the image encoder and text encoder in the first visual language model are fixed, and initial values ​​for visual cues are set. Based on the initial values ​​of visual cues corresponding to various image classification labels and text samples, a second visual language model is generated by training the first visual language model. During this process, the good image and text feature alignment space of the pre-trained visual language model is used to mine and learn visual cues corresponding to various image classification labels, enhancing the visual representation capability of the visual language model. This eliminates the need to acquire visual knowledge through a large number of labeled image samples, thus reducing the need for extensive labeled image sample preparation during model training. Next, based on the learned visual cues, text cues, and text samples, the target visual language model is further trained by training the second visual language model. During this process, the adapter parameters of the model adapters configured in the image encoder and text encoder are adjusted to collaboratively optimize image and text feature extraction, while simultaneously optimizing both the text cues and the learned visual cues. This process transfers the visual knowledge of the visual cues to the text cues, enhancing the visual representation capability of the text cues. The target visual language model trained through the above process can significantly improve the accuracy of prediction results when used to identify various image classification labels corresponding to an image.

[0082] The embodiments of this disclosure divide the training process of the visual language model into a visual cue learning stage and a visual knowledge transfer stage. In the visual cue learning stage, visual cues corresponding to various image classification labels are learned. In the visual knowledge transfer stage, the visual knowledge of the visual cues is transferred to the text cues. The two stages are performed sequentially during the training process. Compared with training the two stages simultaneously, this avoids the problem of convergence difficulties for randomly initialized visual cues.

[0083] This disclosure also provides a text sample generation scheme to supplement manually annotated text samples. By constructing a first prompt word using image classification labels and a first query template for generating pseudo-text samples, and inputting it into a large language model to generate pseudo-text samples, a second prompt word is constructed using a second query template for detecting pseudo-text samples in order to filter out unreliable text generated by the large language model. The second prompt word is then input into the large language model to query whether the generated pseudo-text samples are usable, thus ensuring the reliability of the text samples.

[0084] The functions of each module in the apparatus of this embodiment can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0085] Figure 9 is a block diagram of an electronic device used to implement embodiments of the present disclosure. As shown in Figure 9, the electronic device includes a memory 901 and a processor 902. The memory 901 stores a computer program that can run on the processor 902. When the processor 902 executes the computer program, it implements the methods described in the above embodiments. The number of memories 901 and processors 902 can be one or more.

[0086] The electronic device also includes:

[0087] The communication interface 903 is used to communicate with external devices and exchange and transmit data.

[0088] If the memory 901, processor 902, and communication interface 903 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 9, but this does not indicate that there is only one bus or one type of bus.

[0089] Optionally, in a specific implementation, if the memory 901, processor 902, and communication interface 903 are integrated on a single chip, then the memory 901, processor 902, and communication interface 903 can communicate with each other through an internal interface.

[0090] This disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods provided in this disclosure.

[0091] This disclosure also provides a computer program product, which includes a computer program that, when executed by a processor, implements the methods provided in any embodiment of this disclosure.

[0092] This disclosure also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device on which the chip is installed to perform the methods provided in this disclosure.

[0093] This disclosure also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in this disclosure.

[0094] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0095] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0096] In the above embodiments, implementation can be achieved, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0097] In the description of this disclosure, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this disclosure, as well as the features of those different embodiments or examples.

[0098] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0099] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0100] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0101] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0102] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0103] The above description is merely an exemplary embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this disclosure, and these should all be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.< / eos>

Claims

A method for training a visual language model, comprising: Fixing the respective encoding parameters of the image encoder and the text encoder in the pre-trained first visual language model, learning the respective visual cues of the plurality of image classification labels based on the initial values of the respective visual cues and the text samples of the plurality of image classification labels, and generating a second visual language model by training the first visual language model; Based on the learned visual cues, text cues and text samples, the second visual language model is trained to optimize the learned visual cues, text cues and adapter parameters of the model adapter configured by the image encoder and the text encoder, and a target visual language model is generated; wherein the target visual language model is used to identify a plurality of image classification labels corresponding to an image. The method of claim 1, wherein, Before the method of learning the respective visual cues of the plurality of image classification labels based on the initial values of the respective visual cues and the text samples of the plurality of image classification labels by training the first visual language model, the method further comprises: According to the similarity relationship between the initial visual cues and the text samples, a positive sample pair and a negative sample pair are constructed. The method of claim 2, wherein, The positive sample pair and the negative sample pair are constructed according to the similarity relationship between the initial visual cues and the text samples, comprising: Carrying out word segmentation processing on the text sample, and determining the image classification label corresponding to the word segmentation; The initial visual cues corresponding to the image classification label related to the word segmentation and the text sample corresponding to the word segmentation are constructed as a positive sample pair; The initial visual cues of the image classification label irrelevant to the word segmentation and the text sample corresponding to the word segmentation are constructed as a negative sample pair. The method of claim 2, wherein, The method of learning the respective visual cues of the plurality of image classification labels based on the initial values of the respective visual cues and the text samples of the plurality of image classification labels by training the first visual language model, comprising: Based on the constructed positive sample pair and negative sample pair of the initial visual cues and text samples, the respective visual cues of the plurality of image classification labels are learned by training the first visual language model. The method according to any one of claims 1 to 4, wherein The target loss function constructed when training the second visual language model includes: a first loss function representing the feature similarity between the visual cues and the text samples, a second loss function representing the feature similarity between the visual cues and the text cues, and a third loss function representing the feature similarity between the text samples and the text cues. The method according to any one of claims 1 to 5, wherein The method of optimizing the learned visual cues, text cues and adapter parameters of the model adapter configured by the image encoder and the text encoder by training the second visual language model, comprising: Iteratively training the second visual language model and adjusting the visual cues, text cues and adapter parameters of the model adapter in each training round to make the target loss function for training the second visual language model meet the set optimization condition. The method according to any one of claims 1 to 6, wherein The text sample at least includes a pseudo text sample, and the method further comprises: According to the image classification label and the first query template used to generate the pseudo text sample, a first prompt word is constructed, and the first prompt word is input into a large language model to generate a pseudo text sample; A second prompt word is constructed using a second query template for detecting pseudo-text samples, and a large language model is input with the second prompt word to query whether the generated pseudo-text samples are available; The available pseudo-text samples are extracted. A method for predicting an image label, applied to a target visual language model, the method comprising: obtaining a text prompt corresponding to a target image; predicting a plurality of image classification labels corresponding to the target image based on the target image and the corresponding text prompt; the target visual language model is generated by the training method of the visual language model of any one of claims 1-7. According to the method of claim 8, the target visual language model comprises a text encoder and an image encoder, and the method for predicting a plurality of image classification labels corresponding to the target image based on the target image and the corresponding text prompt comprises: extracting image features of the target image using an image encoder, and extracting text features corresponding to the text prompt using a text encoder; wherein the image features and the text features correspond to an aligned feature space; predicting a plurality of image classification labels of the target image based on the image features and the text features. A data configuration method of a commodity object, the method comprising: obtaining at least one commodity image corresponding to a commodity object; calling a target visual language model to generate commodity classification labels corresponding to the at least one commodity image; adding the commodity classification labels to the commodity object, so that the commodity object is marked according to the commodity classification labels when the commodity object is searched or stored; the target visual language model is generated by the training method of the visual language model of any one of claims 1-7. An image classification method, the method comprising: obtaining a target image; calling a target visual language model to generate image classification labels corresponding to the target image; the target visual language model is generated by the training method of the visual language model of any one of claims 1-7. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method of any one of claims 1-11 when executing the computer program. A computer-readable storage medium having a computer program stored therein, wherein the computer program is executed by a processor to implement the method of any one of claims 1-11. A computer program product comprising computer instructions, wherein the computer instructions are executed by a processor to implement the method of any one of claims 1-11.

Images