Visual language model training method, image tag prediction method and electronic equipment

Abstract

The embodiment of the invention provides a visual language model training method, an image tag prediction method and electronic equipment. According to the embodiment of the invention, a second visual language model is generated by training a first visual language model by using corresponding coding parameters in an image encoder and a text encoder in a fixed first visual language model and based on initial values and text samples of visual prompts respectively corresponding to various image classification tags; visual prompts respectively corresponding to various image classification labels are mined and learned, and the visual representation capability of a visual language model is enhanced. Further training a target visual language model by training a second visual language model based on learned visual prompts, text prompts and text samples, adjusting adapter parameters of model adapters configured for an image encoder and a text encoder respectively, and performing collaborative optimization on image feature extraction and text feature extraction; the process enables visual knowledge of the visual cue to be transferred to the text cue.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for training a visual language model, a method for predicting an image label, a device, an electronic device, a storage medium, and a computer program product. Background Art

[0002] Multiple business scenarios in online trading platforms involve the use of product tags. For example, in a search scenario, when retrieving products based on key information entered by the user, it involves matching the key information used for retrieval with the product tags. For another example, when adding a batch of products to the warehouse, it is necessary to identify the product tags corresponding to the products so that the virtual product objects can be associated with the product tags when stored in the database.

[0003] In practical applications, an image may include one or more different objects. The classification of commodity objects can be achieved through multi-label classification based on the images of commodity objects. When using models to implement multi-label tasks, in order to ensure the accuracy of classification prediction results, one way is to provide a large number of labeled image samples for training the model, which requires high labor costs; another way is to add text prompts as model input, but the representation of the visual information of the image by text information is still limited. Summary of the invention

[0004] The embodiments of the present application provide a visual language model training method, an image label prediction method, device, electronic device, storage medium and computer program product to solve one or more of the above-mentioned technical problems.

[0005] In a first aspect, an embodiment of the present application provides a method for training a visual language model, comprising: fixing the corresponding encoding parameters in an image encoder and a text encoder in a pre-trained first visual language model, and based on the initial values ​​and text samples of the visual cues corresponding to the multiple image classification labels, training the first visual language model to learn the visual cues corresponding to the multiple image classification labels, thereby generating a second visual language model; based on the learned visual cues, text cues and text samples, training the second visual language model to optimize the learned visual cues, text cues and adapter parameters of the model adapters configured by the image encoder and the text encoder, thereby generating a target visual language model; wherein the target visual language model is used to identify multiple image classification labels corresponding to an image.

[0006] In the second aspect, an embodiment of the present application provides a method for predicting image labels, which is applied to a target visual language model. The method includes: obtaining a text prompt corresponding to a target image; predicting multiple image classification labels corresponding to the target image based on the target image and the corresponding text prompt; and the target visual language model is generated by the training method of the visual language model described above.

[0007] In a third aspect, an embodiment of the present application provides a data configuration method for a commodity object, the method comprising: obtaining at least one commodity image corresponding to the commodity object; calling a target visual language model to generate a commodity classification label corresponding to the at least one commodity image; adding the commodity classification label to the commodity object for commodity retrieval or marking the commodity object according to the commodity classification label when the commodity is put into storage; the target visual language model is generated by the training method of the visual language model described above.

[0008] In a fourth aspect, an electronic device includes a memory, a processor, and a computer program stored in the memory, wherein the processor implements the above-described method when executing the computer program.

[0009] In a fifth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method described above is implemented.

[0010] In a sixth aspect, a computer program product includes computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0011] According to the embodiment of the present application, a pre-trained first visual language model is used, the encoding parameters corresponding to the image encoder and the text encoder in the first visual language model are fixed, the initial value of the visual prompt is set, and the initial value and text sample of the visual prompt corresponding to the multiple image classification labels are used to generate the second visual language model by training the first visual language model. In this process, the visual prompts corresponding to the multiple image classification labels are mined and learned through the feature space of the pre-trained visual language model with good image and text feature alignment, and the visual representation ability of the visual language model is enhanced. There is no need to obtain visual knowledge through a large number of learning labeled image samples, so that a large number of labeled image samples do not need to be prepared during model training. Next, based on the learned visual prompts, text prompts and text samples, the target visual language model is further trained by training the second visual language model. In this process, the adapter parameters of the model adapters configured by the image encoder and the text encoder are adjusted, and the extraction of image features and text features is collaboratively optimized. At the same time, the text prompts and the learned visual prompts are optimized. This process transfers the visual knowledge of the visual prompts to the text prompts, and enhances the visual representation ability of the text prompts. The target visual language model obtained by the above process training is used to identify the multiple image classification labels corresponding to the image, which can significantly improve the accuracy of the prediction results.

[0012] The embodiment of the present application divides the training process of the visual language model into a visual cue learning stage and a visual knowledge transfer stage. In the visual cue learning stage, the visual cues corresponding to multiple image classification labels are learned. In the visual knowledge transfer stage, the visual knowledge of the visual cues is transferred to the text cues. The two stages are performed sequentially during the training process. Compared with training the two stages at the same time, the problem of difficult convergence of randomly initialized visual cues is avoided.

[0013] The embodiment of the present application also provides a scheme for generating text samples to supplement manually annotated text samples. A first prompt word is constructed by combining an image classification label with a first query template for generating pseudo-text samples, and the pseudo-text samples are generated by inputting the first prompt word into a large language model. Furthermore, in order to filter out unreliable text generated by the large language model, a second prompt word is constructed using a second query template for detecting pseudo-text samples, and the second prompt word is input into the large language model to query whether the pseudo-text samples generated are usable, thereby ensuring the reliability of the text samples.

[0014] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0016] Figure 1 A schematic diagram showing the first stage of the method for training a visual language model according to an embodiment of the present application is shown;

[0017] Figure 2 A schematic diagram showing the second stage of the method for training a visual language model according to an embodiment of the present application is shown;

[0018] Figure 3 A flowchart of a method for training a visual language model according to an embodiment of the present application is shown;

[0019] Figure 4 A flowchart of a method for predicting image labels according to an embodiment of the present application is shown;

[0020] Figure 5 A flowchart showing a method for configuring data of a commodity object according to an embodiment of the present application is shown;

[0021] Figure 6 A structural block diagram of a visual language model training device according to an embodiment of the present application is shown;

[0022] Figure 7 A structural block diagram of an image label prediction device according to an embodiment of the present application is shown;

[0023] Figure 8 A structural block diagram showing a data configuration device for a commodity object according to an embodiment of the present application is shown;

[0024] Fig. 9 A block diagram of an electronic device used to implement an embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the concept or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature and not restrictive.

[0026] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0027] Online trading platforms involve the need to add classification labels to commodity objects, especially in scenarios involving cross-border online trading platforms. Since they involve commodity management in different regions, image styles vary greatly, and text languages ​​in different languages ​​are involved, the classification and recognition of commodity objects is more complex, which places higher requirements on the accuracy of model recognition. Using visual language models to perform multi-label classification tasks requires providing a large amount of labeled visual data for model training, or adding text prompts, but recognition accuracy still needs to be improved.

[0028] The embodiment of the present application adopts a visual language model to perform multi-label classification tasks, and has made many improvements to the training process of the original visual language model. Overall, the process of training the visual language model in the embodiment of the present application is divided into two stages. The first stage is the visual cue learning stage, and the encoder parameters corresponding to the image encoder and the text encoder of the pre-trained visual language model are fixed. That is to say, the parameters of the two encoders are not adjusted at this stage, but the feature space of good image and text alignment that the visual language model already has is fully utilized. The goal of this stage is to learn the visual cues corresponding to multiple image classification labels; the second stage is the visual knowledge transfer stage, and the encoder parameters corresponding to the image encoder and the text encoder of the visual language model trained in the first stage are still fixed, and the model adapters configured for the image encoder and the text encoder are adjusted. The extraction of image features and text features is collaboratively optimized, and the text cues and the learned visual cues are optimized at the same time. Through the optimization process of the second stage, the visual knowledge of the visual cues learned in the first stage can be transferred to the text cues, and the visual representation ability of the text cues can be enhanced.

[0029] refer to Figure 1 The diagram shows the first stage of the training method of the visual language model of the embodiment of the present application. In this stage, the diverse and comprehensive visual knowledge of each image classification label is accurately learned. For image multi-label classification tasks, such as Figure 1 As shown in the figure, the various image classification labels of the cross-border online trading platform include person, bicycle, cup, vase, car, etc. For each classification label, the initial value of the visual cue is pre-configured, which can be a randomly generated value or a priori value. The corresponding parameters of the image encoder and text encoder are frozen (frozen parameters in the figure). The visual cue needs to be learned, so it is the changing parameter in the figure.

[0030] The visual cues set for each image classification label are not combined with any explicit visual or text labels. The visual language model itself has a text-image aligned feature space in which the image modalities have a wide variety. The visual knowledge learned in the image through the training model is reflected in the visual cues. Since the visual cues in the embodiments of the present application are not pre-configured for direct use but acquired through learning, they are not real visual cues, but rather a learned pseudo-visual cue or a simulated visual cue. The visual cue can be defined as P = [p1, p2, p3, ..., p N ], where p i ∈R H×W×3 represents the visual cue of the i-th category, H and W are of equal size, N is the number of image classification labels, and the number of visual cues is equal to the number of image classification labels.

[0031] Next, we use the aligned feature space of the visual language model, the collected text training data, and the frozen image encoder and text encoder to learn visual cues. The learning process of visual cues can be formalized as follows: 对齐 ,T,φ,ψ>→ , where T represents the labeled text data or pseudo text data collected by LLM, Ω 对齐 denotes the aligned feature space, φ(·) and ψ(·) refer to the frozen text encoder and image encoder, respectively.

[0032] according to Figure 1 As shown, noun filtering is also performed on the text sample data. For example, in the text sample "A compact car with bicycles mounted on the roof" in the figure, the nouns car and bicycle included in the text sample are obtained through noun filtering. The word set of the image classification label corresponding to the visual cues in the rightmost column of the figure includes car and bicycle. Therefore, the visual cues corresponding to the image classification labels of car and bicycle and the text sample can be used as a pair of positive sample pairs, and the text sample and other visual cues of image classification labels that do not include car and bicycle can be used as a pair of negative sample pairs. The sample pairs are constructed by this word segmentation matching method, and the model is trained based on the sample pairs, so that the visual cues can be learned iteratively.

[0033] like Figure 2 The schematic diagram of the second stage of the training method of the visual language model of the embodiment of the present application is shown. The above-mentioned learned visual prompts are used to obtain corresponding image features through an image encoder, and the text data and text prompts are respectively extracted by corresponding text encoders to obtain text features. Wherein the image encoder and the text encoder respectively use corresponding model adapters, and the structures of the two model adapters are the same, for example, they are composed of two fully connected layers, an activation function and a residual connection layer, which is called a dual adapter. The text prompt and the text sample use the same model adapter, that is to say, the text adapter is a parameter sharing network for text prompts and text samples. In the model training of the second stage, the constructed loss function can be cross-constructed around the text sample, the text prompt and the visual prompt, including a first loss function representing the feature similarity between the visual prompt and the text sample, a second loss function representing the feature similarity between the visual prompt and the text prompt, and a third loss function representing the feature similarity between the text sample and the text prompt. By this cross-construction loss function, the visual prompt can affect the text as much as possible. In the training process, by optimizing the adapter parameters of the model adapters configured by the visual prompt, the text prompt, and the text encoder and the image encoder, the visual knowledge of the visual prompt learned in the first stage can be transferred to the text prompt.

[0034] Similar to the definition of the visual cue P above, the textual cue can be defined as: R i =[r1,r2,...,r M ,g i ],S=[R1,R2,...,R N ]. Among them, g i Represents the text feature of the i-th label, r i is the learnable context vector of the text prompt, and M represents the number of text prompts. The text features corresponding to the text prompt and text sample can be extracted through the text encoder, and the image features corresponding to the visual prompt can be extracted through the image encoder, which correspond to: I = φ(S), H = φ(T), V = ψ(P). Through further processing by the model adapter corresponding to the image encoder and the model adapter corresponding to the text encoder, we can obtain: U = (1-λ)g(V) + λV, G = (1-λ)h(H) + λH, E = (1-λ)h(I) + λI, where g(·) and h(·) represent the adapter functions of the image and text modalities, respectively, and λ∈[0,1] represents the weight between the features extracted by the adapter module and the image features / text features. After extracting the above features, the loss function can be further used to train the model. The loss function constructed with similarity is used to retain the similarity between the visual prompt and the text prompt, so that the visual knowledge of the visual prompt learned in the first stage can be transferred to the text prompt.

[0035] The similarity matrix in the target loss function can be constructed as UE T ∈R N×N , the visual cues and textual cues of N-class labels are constructed as the identity matrix, the size of the similarity matrix is ​​equal to the number of image classification labels N, and the target loss function can be constructed as:

[0036]

[0037]

[0038]

[0039] in, and denotes the Softmax normalized similarity from visual cues to textual cues and from textual cues to visual cues, respectively. τ denotes the temperature parameter. l CE represents the cross entropy loss. Represents the authenticity of the similarity between the textual prompt and the visual prompt. If the textual prompt t i and visual cues j belong to the same category, then otherwise and The definition of is the same. Using ranking loss to obtain L visual and L text (Similar to the similarity L below PVP Calculation formula), specifically, using L visual To measure the difference between text samples and visual cues, L visual It can be expressed as:

[0040]

[0041]

[0042] Similarly, L text Measuring the difference between a text sample and a text prompt can be expressed as:

[0043]

[0044]

[0045] Finally, we can get the target loss function: L = L vtc +L visual +L text , where L vtc Measuring the difference between textual cues and visual cues. In the second stage of training, the encoding parameters corresponding to the image encoder and the text encoder are also fixed, and the objective loss function is optimized to learn the visual cues, textual cues, and adapter parameters of the two adapters.

[0046] The visual language model training process of the embodiment of the present application can be completed on a local or cloud computing device, and the trained visual language model can be stored in a computing device or cloud for remote call, or sent to a terminal for use. The process of data configuration of the product image can be completed at the merchant end or the terminal of the trading platform, wherein the visual language model of the terminal or the local computing device or the cloud is called to obtain the product classification label. For the visual language model training process, the execution subject can be an application, service, instance, software-based functional module, virtual machine (VirtualMachine, VM), container or cloud server, etc., or a hardware device with data processing function (such as a server or terminal device) or hardware chip (such as CPU, GPU, FPGA, NPU, AI acceleration card or DPU), etc. The devices for implementing visual language model training and image label prediction can be deployed on the computing equipment of the application party that provides the corresponding services or on the cloud computing platform that provides computing power, storage and network resources. The mode of providing services to the outside world by the cloud computing platform can be IaaS (Infrastructure as a Service), PaaS (Platform as a Service), SaaS (Software as a Service) or DaaS (Data as a Service). Taking the platform providing SaaS software as a service (Software as a Service) as an example, the cloud computing platform can use its own computing resources to provide the function execution of visual language model training or image label prediction, and the specific application architecture can be built according to service requirements. For example, the platform can provide a construction service based on the above model to the application party or individual using the platform resources, and further call the above model and implement the function of online or offline image label prediction based on the prediction request of the image label submitted by the relevant client or server and other devices.

[0047] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0048] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0049] The present application embodiment provides a method for training a visual language model, such as Figure 3 FIG. 1 is a flowchart of a method 300 for training a visual language model according to an embodiment of the present application. The method 300 may include:

[0050] In step S301, the corresponding encoding parameters in the image encoder and the text encoder in the pre-trained first visual language model are fixed, and based on the initial values ​​and text samples of the visual cues corresponding to the multiple image classification labels, the visual cues corresponding to the multiple image classification labels are learned by training the first visual language model to generate a second visual language model.

[0051] In the first stage of learning visual cues, the second visual language model is generated by training the first visual language model. During this process, the constructed loss function can be calculated based on the similarity between the text sample and the visual cues. For the input text sample T, the pre-trained visual language model is used to project the last " <eos>"The features of the mark are used to obtain the global text features, and the visual cues corresponding to each image classification label of P are obtained through the visual attention pool, which is represented by H = φ (T), V = ψ (P), H ∈ R B×D , represents the normalized global text features extracted in batches, V∈N N×D , represents the normalized global image features of N visual cues, for a specific text t in the batch i ∈T, text t i The similarity with the visual cue is calculated as s ij = <h i ,v j >, h i ∈H, represents the text t i The global text feature, v j ∈V, represents the global image feature of the jth visual cue.

[0052] In the first stage of training, a ranking loss can be used to measure the similarity scores of textual cues and visual cues and the difference between the classification labels of text samples. Visual cues can be learned through the loss function in the following formula: Where c+ and c- are the positive and negative labels of the sample pair, s ki and kj is the similarity of the positive sample pair and the negative sample pair described in the similarity calculation formula above, and m is used to measure the difference between each pair of prediction values.

[0053] The above-mentioned pre-trained visual language model is recorded as the first visual language model, which can be specifically CLIP (Contrastive Language-Image Pre-training), which is a pre-training model based on contrastive text-image pairs. Its training process is roughly divided into three steps: using the Transformer structure or its series of variants as the basic structure to extract text features, using CNN models such as ResNet or structures such as Transformer to extract image features, and further associating the information of the two modalities of text and image, that is, aligning the features extracted from the two modalities of text and image.

[0054] Among them, in the process of training to obtain the first visual language model, there are many ways to align features. A method based on contrastive learning can be used to train the model through contrastive learning so that it can shorten the distance between paired images and texts in the feature space, while pushing the distance between unrelated samples away. For example, it can be based on image contrastive learning, based on image and text contrastive learning, or based on image, text and label contrastive learning. Feature alignment can also choose a method based on generation tasks, by training the model for image generation, text generation or cross-modal generation to learn semantic features, for example, it can be based on mask image modeling, mask language modeling, or mask cross-modal modeling, etc. Feature alignment can also use an alignment method based on the purpose of alignment, by matching the features of the image and text to align the features, such as global image and text matching or local image area and word matching.

[0055] Due to the differences in the distribution of images and texts in different business scenarios, the pre-trained visual language model needs to undergo some adjustments when used in specific business scenarios. One way is to adjust based on prompts, by adding text prompts to the model input so that the model can adapt to the needs of the scene without fine-tuning; another way is to adjust based on feature adapters, by adding lightweight model adapters to the architecture of the visual language model for fine-tuning; in the embodiment of the present application, in addition to the input of images, the model input also includes the input of text prompts. At the same time, a model adapter that is suitable for both image encoders and text encoders is added to the architecture of the visual language model. After learning the visual prompts, the model is trained again based on the learned visual prompts, text prompts and text samples. During the training process, in addition to optimizing the learned visual prompts and text prompts, the parameters of the model adapter are also adjusted so that the model adapter can be trained at the same time.

[0056] In step S302, based on the learned visual cues, text cues and text samples, the learned visual cues, text cues and adapter parameters of the model adapters respectively configured by the image encoder and the text encoder are optimized by training the second visual language model to generate a target visual language model; wherein the target visual language model is used to identify multiple image classification labels corresponding to the image.

[0057] According to the embodiment of the present application, a pre-trained first visual language model is used, the encoding parameters corresponding to the image encoder and the text encoder in the first visual language model are fixed, the initial value of the visual prompt is set, and the initial value and text sample of the visual prompt corresponding to the multiple image classification labels are used to generate the second visual language model by training the first visual language model. In this process, the visual prompts corresponding to the multiple image classification labels are mined and learned through the good image and text feature alignment space of the pre-trained visual language model, and the visual representation ability of the visual language model is enhanced. There is no need to obtain visual knowledge through a large number of learning labeled image samples, so that a large number of labeled image samples do not need to be prepared during model training. Next, based on the learned visual prompts, text prompts and text samples, the target visual language model is further trained by training the second visual language model. In this process, the adapter parameters of the model adapters configured by the image encoder and the text encoder are adjusted, and the image feature and text feature extraction are collaboratively optimized. At the same time, the text prompts and the learned visual prompts are optimized. This process transfers the visual knowledge of the visual prompts to the text prompts, and enhances the visual representation ability of the text prompts. The target visual language model obtained by the above process training is used to identify the multiple image classification labels corresponding to the image, which can significantly improve the accuracy of the prediction results.

[0058] The embodiment of the present application divides the training process of the visual language model into a visual cue learning stage and a visual knowledge transfer stage. In the visual cue learning stage, the visual cues corresponding to multiple image classification labels are learned. In the visual knowledge transfer stage, the visual knowledge of the visual cues is transferred to the text cues. The two stages are performed sequentially during the training process. Compared with training the two stages at the same time, the problem of difficult convergence of randomly initialized visual cues is avoided.

[0059] In a possible implementation, before learning the visual cues corresponding to the various image classification labels respectively by training the first visual language model based on the initial values ​​of the visual cues and the text samples, positive sample pairs and negative sample pairs can also be constructed according to the similarity between the initial visual cues and the text samples. In one example, the sample construction process can be implemented by noun filtering. When constructing the positive sample pairs and negative sample pairs according to the similarity between the initial visual cues and the text samples, the text samples can be firstly segmented, and the image classification labels corresponding to the segmentation can be determined. That is, given a visual cue and a text sample, the text sample is filtered for nouns to obtain the nouns included in the text sample. If the noun is included in a word set corresponding to a certain image classification label, it can be considered that the text sample corresponding to the extracted noun and the visual cues corresponding to the image classification label corresponding to the word set are a pair of positive samples, otherwise, they are a pair of negative samples. Further, the initial visual cues corresponding to the image classification label related to the segmentation and the text samples corresponding to the segmentation are constructed as positive sample pairs, and the initial visual cues of the image classification label unrelated to the segmentation and the text samples corresponding to the segmentation are constructed as negative sample pairs.

[0060] In a possible implementation, the initial values ​​of the visual cues and text samples corresponding to the multiple image classification labels respectively, and learning the visual cues corresponding to the multiple image classification labels by training the first visual language model include: based on the constructed positive sample pairs and negative sample pairs of initial visual cues and text samples, learning the visual cues corresponding to the multiple image classification labels respectively by training the first visual language model.

[0061] In one possible implementation, the target loss function constructed when training the second visual language model includes: a first loss function characterizing the feature similarity between visual cues and text samples, a second loss function characterizing the feature similarity between visual cues and text cues, and a third loss function characterizing the feature similarity between text samples and text cues.

[0062] The embodiment of the present application constructs a variety of loss functions in the visual knowledge transfer stage, which respectively correspond to the feature similarity between visual cues and text samples, the feature similarity between visual cues and text cues, and the feature similarity between text samples and text cues. The model is optimized by controlling the value of the loss function, controlling the transfer of visual knowledge from visual cues to text cues while optimizing the adapter parameters.

[0063] In a possible implementation, optimizing the learned visual cues, text cues, and adapter parameters of the model adapters respectively configured by the image encoder and the text encoder by training the second visual language model includes:

[0064] The second visual language model is iteratively trained, and the visual prompt, the text prompt, and the adapter parameters of the model adapter are adjusted in each round of training so that the target loss function used to train the second visual language model satisfies the set optimization condition.

[0065] In a possible implementation, the text sample includes at least a pseudo text sample, and the method further includes: constructing a first prompt word according to the image classification label and a first query template for generating the pseudo text sample, and using the first prompt word to input a large language model to generate the pseudo text sample; constructing a second prompt word using a second query template for detecting the pseudo text sample, and using the second prompt word to input a large language model to query whether the generated pseudo text sample is available; and extracting the available pseudo text sample.

[0066] The embodiment of the present application also provides a scheme for generating text samples to supplement manually annotated text samples. A first prompt word is constructed by combining an image classification label with a first query template for generating pseudo-text samples, and the pseudo-text samples are generated by inputting the first prompt word into a large language model. Furthermore, in order to filter out unreliable text generated by the large language model, a second prompt word is constructed using a second query template for detecting pseudo-text samples, and the second prompt word is input into the large language model to query whether the pseudo-text samples generated are usable, thereby ensuring the reliability of the text samples.

[0067] Next, an application example after model training is given from the perspective of the target visual language model. The embodiment of the present application provides a method for predicting image labels, such as Figure 4 The flowchart of a method 400 for predicting an image tag according to an embodiment of the present application is shown, which is applied to a target visual language model. The method 400 may include:

[0068] In step S401, a text prompt corresponding to a target image is obtained. In step S402, multiple image classification labels corresponding to the target image are predicted based on the target image and the corresponding text prompt; the target visual language model is generated by the visual language model training method of the above embodiment.

[0069] In one possible implementation, the target visual language model includes a text encoder and an image encoder, and predicting multiple image classification labels corresponding to the target image based on the target image and the corresponding text prompt includes: using the image encoder to extract image features of the target image, and using the text encoder to extract text features corresponding to the text prompt; wherein the image features and text features correspond to aligned feature spaces; and predicting multiple image classification labels of the target image based on the image features and text features.

[0070] Next, combined with the specific application scenarios of commodity objects, the role of the target visual language model obtained through training and the corresponding predicted image classification labels in actual application scenarios is given. The embodiment of the present application provides a data configuration method for commodity objects, such as Figure 5 The flowchart of a method 500 for configuring data of a commodity object according to an embodiment of the present application is shown. The method 500 may include:

[0071] In step S501, at least one product image corresponding to a product object is obtained. In step S502, a target visual language model is called to generate a product classification label corresponding to the at least one product image. In step S503, the product classification label is added to the product object so that the product object is marked according to the product classification label when the product is retrieved or put into storage; the target visual language model is generated by the training method of the visual language model of the above embodiment.

[0072] Based on the generation process of image classification labels, an embodiment of the present application can also provide an image classification method, including: first, obtaining a target image; second, calling a target visual language model to generate an image classification label corresponding to the target image; wherein the target visual language model is generated by the training method of the visual language model of the above embodiment.

[0073] Corresponding to the above examples and method embodiments provided in the embodiments of the present application, the embodiments of the present application also provide a training device for a visual language model. Figure 6 The structure block diagram of a visual language model training device 600 according to an embodiment of the present application is shown. The device 600 may include:

[0074] A visual cue learning module 601 is used to fix the encoding parameters corresponding to the image encoder and the text encoder in the pre-trained first visual language model, and based on the initial values ​​and text samples of the visual cues corresponding to the multiple image classification labels, learn the visual cues corresponding to the multiple image classification labels by training the first visual language model to generate a second visual language model; a parameter optimization module 602 is used to optimize the learned visual cues, text cues and adapter parameters of the model adapters configured by the image encoder and the text encoder respectively based on the learned visual cues, text cues and text samples by training the second visual language model to generate a target visual language model; wherein the target visual language model is used to identify multiple image classification labels corresponding to an image.

[0075] In one possible implementation, the device also includes: a sample construction module, which is used to construct positive sample pairs and negative sample pairs according to the similarity relationship between the initial visual cues and text samples, before learning the visual cues corresponding to the multiple image classification labels respectively by training the first visual language model.

[0076] In a possible implementation, the sample construction module includes: a word segmentation submodule, which is used to perform word segmentation on text samples and determine image classification labels corresponding to the word segmentations; a first sample construction submodule, which is used to construct the initial visual cues corresponding to the image classification labels related to the word segmentations and the text samples corresponding to the word segmentations into positive sample pairs; and a second sample construction submodule, which is used to construct the initial visual cues of the image classification labels unrelated to the word segmentations and the text samples corresponding to the word segmentations into negative sample pairs.

[0077] In a possible implementation, the visual cue learning module is specifically used to learn the visual cues corresponding to the multiple image classification labels respectively by training the first visual language model based on the constructed initial visual cues and positive sample pairs and negative sample pairs of text samples.

[0078] In one possible implementation, the target loss function constructed when training the second visual language model includes: a first loss function characterizing the feature similarity between visual cues and text samples, a second loss function characterizing the feature similarity between visual cues and text cues, and a third loss function characterizing the feature similarity between text samples and text cues.

[0079] In one possible implementation, the parameter optimization module is specifically used to iteratively train the second visual language model and adjust the adapter parameters of the visual cues, text cues, and model adapter in each round of training so that the target loss function used to train the second visual language model meets the set optimization conditions.

[0080] In a possible implementation, the text sample includes at least a pseudo text sample, and the device further includes: a pseudo text sample generation module, used to construct a first prompt word according to the image classification label and a first query template for generating the pseudo text sample, and use the first prompt word to input a large language model to generate the pseudo text sample; a pseudo text sample query module, used to construct a second prompt word using a second query template for detecting the pseudo text sample, and use the second prompt word to input a large language model to query whether the generated pseudo text sample is available, and extract the available pseudo text sample.

[0081] Corresponding to the above examples and method embodiments provided in the embodiments of the present application, the embodiments of the present application also provide an image tag prediction device, which is applied to a target visual language model. Figure 7 FIG. 7 is a structural block diagram of an image label prediction device 700 according to an embodiment of the present application. The device 700 may include:

[0082] The text prompt acquisition module 701 is used to obtain the text prompt corresponding to the target image; the classification label prediction module 702 is used to predict multiple image classification labels corresponding to the target image based on the target image and the corresponding text prompt; the target visual language model is generated by the visual language model training method of the above embodiment.

[0083] In a possible implementation, the target visual language model includes a text encoder and an image encoder, and the classification label prediction module is specifically used to use the image encoder to extract image features of the target image, and use the text encoder to extract text features corresponding to the text prompt; wherein the image features and text features correspond to aligned feature spaces; and multiple image classification labels of the target image are predicted based on the image features and text features.

[0084] Corresponding to the above examples and method embodiments provided in the embodiments of the present application, the embodiments of the present application also provide a data configuration device for a commodity object. Figure 8 The structure diagram of a data configuration device 800 for a commodity object according to an embodiment of the present application is shown. The device 800 may include:

[0085] The object image acquisition module 801 is used to acquire at least one product image corresponding to the product object; the classification label generation module 802 is used to call the target visual language model to generate a product classification label corresponding to the at least one product image; the label adding module 803 is used to add the product classification label to the product object, so as to mark the product object according to the product classification label when the product is retrieved or put into storage; the target visual language model is generated by the training method of the visual language model of the above embodiment.

[0086] Based on the generation process of image classification labels, an embodiment of the present application can also provide an image classification device, including: an image acquisition module, used to acquire a target image; a classification label generation module, used to call a target visual language model to generate an image classification label corresponding to the target image; wherein the target visual language model is generated by the training method of the visual language model of the above embodiment.

[0087] According to the embodiment of the present application, a pre-trained first visual language model is used, the encoding parameters corresponding to the image encoder and the text encoder in the first visual language model are fixed, the initial value of the visual prompt is set, and the initial value and text sample of the visual prompt corresponding to the multiple image classification labels are used to generate the second visual language model by training the first visual language model. In this process, the visual prompts corresponding to the multiple image classification labels are mined and learned through the good image and text feature alignment space of the pre-trained visual language model, and the visual representation ability of the visual language model is enhanced. There is no need to obtain visual knowledge through a large number of learning labeled image samples, so that a large number of labeled image samples do not need to be prepared during model training. Next, based on the learned visual prompts, text prompts and text samples, the target visual language model is further trained by training the second visual language model. In this process, the adapter parameters of the model adapters configured by the image encoder and the text encoder are adjusted, and the image feature and text feature extraction are collaboratively optimized. At the same time, the text prompts and the learned visual prompts are optimized. This process transfers the visual knowledge of the visual prompts to the text prompts, and enhances the visual representation ability of the text prompts. The target visual language model obtained by the above process training is used to identify the multiple image classification labels corresponding to the image, which can significantly improve the accuracy of the prediction results.

[0088] The embodiment of the present application divides the training process of the visual language model into a visual cue learning stage and a visual knowledge transfer stage. In the visual cue learning stage, the visual cues corresponding to multiple image classification labels are learned. In the visual knowledge transfer stage, the visual knowledge of the visual cues is transferred to the text cues. The two stages are performed sequentially during the training process. Compared with training the two stages at the same time, the problem of difficult convergence of randomly initialized visual cues is avoided.

[0089] The embodiment of the present application also provides a scheme for generating text samples to supplement manually annotated text samples. A first prompt word is constructed by combining an image classification label with a first query template for generating pseudo-text samples, and the pseudo-text samples are generated by inputting the first prompt word into a large language model. Furthermore, in order to filter out unreliable text generated by the large language model, a second prompt word is constructed using a second query template for detecting pseudo-text samples, and the second prompt word is input into the large language model to query whether the pseudo-text samples generated are usable, thereby ensuring the reliability of the text samples.

[0090] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0091] Fig. 9 FIG. 1 is a block diagram of an electronic device used to implement an embodiment of the present application. Fig. 9 As shown, the electronic device includes: a memory 901 and a processor 902. The memory 901 stores a computer program that can be run on the processor 902. When the processor 902 executes the computer program, the method in the above embodiment is implemented. The number of the memory 901 and the processor 902 can be one or more.

[0092] The electronic device also includes:

[0093] The communication interface 903 is used to communicate with external devices and perform data exchange transmission.

[0094] If the memory 901, the processor 902 and the communication interface 903 are implemented independently, the memory 901, the processor 902 and the communication interface 903 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0095] Optionally, in a specific implementation, if the memory 901, the processor 902 and the communication interface 903 are integrated on a chip, the memory 901, the processor 902 and the communication interface 903 can communicate with each other through an internal interface.

[0096] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0097] An embodiment of the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the method provided in any embodiment of the present application is implemented.

[0098] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0099] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the present application.

[0100] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0101] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0102] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0103] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0104] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0105] Any process or method described in the flow chart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0106] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or used in combination with these instruction execution systems, devices or apparatuses.

[0107] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0108] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0109] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.< / eos>

Claims

1. A method for training a visual language model, comprising: Fixing the encoding parameters corresponding to the image encoder and the text encoder in the pre-trained first visual language model, based on the initial values ​​and text samples of the visual cues corresponding to the multiple image classification labels, training the first visual language model to learn the visual cues corresponding to the multiple image classification labels, and generating a second visual language model; Based on the learned visual cues, text cues and text samples, the target visual language model is generated by training the second visual language model to optimize the learned visual cues, text cues and adapter parameters of the model adapters configured by the image encoder and the text encoder respectively; wherein the target visual language model is used to identify multiple image classification labels corresponding to the image.

2. The method according to claim 1, wherein: Before learning the visual cues corresponding to the plurality of image classification labels respectively by training the first visual language model based on the initial values ​​and text samples of the visual cues corresponding to the plurality of image classification labels respectively, the method further comprises: Positive sample pairs and negative sample pairs are constructed according to the similarity relationship between the initial visual cues and the text samples.

3. The method according to claim 2, wherein: The constructing of positive sample pairs and negative sample pairs according to the similarity relationship between the initial visual prompt and the text sample comprises: Perform word segmentation on the text sample and determine the image classification labels corresponding to the word segmentation; The initial visual cues corresponding to the image classification labels related to the word segmentation and the text samples corresponding to the word segmentation are constructed as positive sample pairs; The initial visual cues of image classification labels that are irrelevant to the word segmentation and the text samples corresponding to the word segmentation are constructed as negative sample pairs.

4. The method according to claim 2, wherein: The initial values ​​and text samples of the visual cues corresponding to the multiple image classification labels respectively, and learning the visual cues corresponding to the multiple image classification labels respectively by training the first visual language model include: Based on the constructed initial visual cues and positive sample pairs and negative sample pairs of text samples, the visual cues corresponding to the multiple image classification labels are learned by training the first visual language model.

5. The method according to claim 1, wherein: The target loss function constructed when training the second visual language model includes: a first loss function representing the feature similarity between visual cues and text samples, a second loss function representing the feature similarity between visual cues and text cues, and a third loss function representing the feature similarity between text samples and text cues.

6. The method according to claim 1, wherein: The step of optimizing the learned visual cues, text cues, and adapter parameters of the model adapters configured by the image encoder and the text encoder, respectively, by training the second visual language model includes: The second visual language model is iteratively trained, and the visual prompt, the text prompt, and the adapter parameters of the model adapter are adjusted in each round of training so that the target loss function used to train the second visual language model satisfies the set optimization condition.

7. The method according to claim 1, wherein: The text sample at least includes a pseudo text sample, and the method further includes: Constructing a first prompt word according to the image classification label and a first query template for generating a pseudo text sample, and using the first prompt word to input a large language model to generate a pseudo text sample; Constructing a second prompt word using a second query template for detecting pseudo text samples, and inputting the second prompt word into a large language model to query whether the generated pseudo text sample is usable; Extract usable pseudo-text samples.

8. A method for predicting an image label, applied to a target visual language model, the method comprising: Get the text prompt corresponding to the target image; Based on the target image and the corresponding text prompt, predict multiple image classification labels corresponding to the target image; The target visual language model is generated by the visual language model training method described in any one of claims 1-7.

9. The method according to claim 8, wherein the target visual language model comprises a text encoder and an image encoder, and the predicting of multiple image classification labels corresponding to the target image based on the target image and the corresponding text prompt comprises: Extracting image features of the target image using an image encoder, and extracting text features corresponding to the text prompt using a text encoder; wherein the image features and text features correspond to aligned feature spaces; Predicting multiple image classification labels of the target image based on the image features and the text features.

10. A method for configuring data of a commodity object, the method comprising: Obtain at least one product image corresponding to the product object; Invoking a target visual language model to generate a product classification label corresponding to the at least one product image; The commodity classification label is added to the commodity object so as to mark the commodity object according to the commodity classification label when the commodity is retrieved or put into storage; the target visual language model is generated by the visual language model training method described in any one of claims 1-7.

11. A method for image classification, the method comprising: Get the target image; Calling a target visual language model to generate an image classification label corresponding to the target image; The target visual language model is generated by the visual language model training method described in any one of claims 1-7.

12. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 11 when executing the computer program.

13. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

14. A computer program product, comprising computer instructions, wherein when the computer instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images