Model training method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a model training method and device, electronic equipment and a readable storage medium, and belongs to the technical field of artificial intelligence. The method comprises: based on a first feature vector and a second feature vector, training a first model to obtain a second model, the first feature vector being a feature vector corresponding to an object area in a first sample image; based on a second sample image, determining a first-layer feature vector and a second-layer feature vector corresponding to the second model and a third feature vector, the second sample image including an image and a text, and the third feature vector being a feature vector corresponding to an image area in the second sample image; performing vector fusion processing on the first-layer feature vector and the third feature vector to obtain a fourth feature vector, and performing vector fusion processing on the second-layer feature vector and the third feature vector to obtain a fifth feature vector; and training the second model based on the fourth feature vector and the fifth feature vector to obtain a third model.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a model training method, device, electronic device and readable storage medium. Background Art

[0002] With the continuous development of artificial intelligence technology, the application of conversational large language models (LLM) is becoming more and more widespread. However, traditional LLM can only input text and output text, and it is often impossible to fully describe things only through text description. Therefore, based on LLM, a multimodal language large model (Multimodal LLM, MLLM) was further developed. MLLM supports multiple modal inputs, such as simultaneous input of text, voice, images and videos, and converts these multimodal signals into feature vectors in the language space, and outputs the final text results based on these feature vectors.

[0003] However, when MLLM performs feature retrieval, it is based on all feature vectors obtained by converting the input multimodal signal, so it cannot achieve fine-grained feature retrieval; and the output of MLLM is still text. This results in poor effect of feature retrieval through the model. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a model training method, device, electronic device and readable storage medium, which can improve the effect of feature retrieval through the model.

[0005] In a first aspect, an embodiment of the present application provides a model training method, the method comprising: training a first model based on a first eigenvector and a second eigenvector to obtain a second model, the first eigenvector being a eigenvector corresponding to an object area in a first sample image, the second eigenvector being a eigenvector corresponding to an image included in a first sample set, and the Euclidean distance with the first eigenvector satisfying a threshold requirement, and the first sample set comprising a first sample image; determining a first-layer eigenvector and a second-layer eigenvector and a third eigenvector corresponding to the second model based on the second sample image, the second sample image comprising an image and text, and the third eigenvector being a eigenvector corresponding to an image area in the second sample image; performing vector fusion processing on the first-layer eigenvector and the third eigenvector to obtain a fourth eigenvector, and performing vector fusion processing on the second-layer eigenvector and the third eigenvector to obtain a fifth eigenvector; training the second model based on the fourth eigenvector and the fifth eigenvector to obtain a third model.

[0006] In a second aspect, an embodiment of the present application provides a model training device, which includes: a training module, a determination module and a processing module; the training module is used to train the first model based on a first feature vector and a second feature vector to obtain a second model, wherein the first feature vector is a feature vector corresponding to an object area in a first sample image, and the second feature vector is a feature vector corresponding to an image included in a first sample set, and the Euclidean distance with the first feature vector satisfies a threshold requirement, and the first sample set includes a first sample image; the determination module is used to determine the first-layer feature vector and the second-layer feature vector, as well as the third feature vector corresponding to the second model based on the second sample image, the second sample image includes an image and text, and the third feature vector is a feature vector corresponding to the image area in the second sample image; the processing module is used to perform vector fusion processing on the first-layer feature vector and the third feature vector determined by the determination module to obtain a fourth feature vector, and perform vector fusion processing on the second-layer feature vector and the third feature vector determined by the determination module to obtain a fifth feature vector; the training module is also used to train the second model based on the fourth feature vector and the fifth feature vector obtained by processing by the processing module to obtain a third model.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program, which is stored in a storage medium and executed by at least one processor to implement the method described in the first aspect.

[0011] In an embodiment of the present application, the first model can be trained based on the first feature vector and the second feature vector to obtain the second model, the first feature vector is the feature vector corresponding to the object area in the first sample image, the second feature vector is the feature vector corresponding to the image included in the first sample set, and the Euclidean distance with the first feature vector meets the threshold requirement, and the first sample set includes the first sample image; and based on the second sample image, the first layer feature vector and the second layer feature vector, as well as the third feature vector corresponding to the second model are determined, the second sample image includes an image and text, and the third feature vector is the feature vector corresponding to the image area in the second sample image; and the first layer feature vector and the third feature vector are vector fused to obtain the fourth feature vector, and the second layer feature vector and the third feature vector are vector fused to obtain the fifth feature vector; and based on the fourth feature vector and the fifth feature vector, the second model is trained to obtain the third model. Through this scheme, on the one hand, because the model can be trained based on the feature vector of the object area in the image, the trained model can pay more attention to the local object information in the image, and can pay attention to the fine-grained object information when performing feature retrieval, thereby realizing fine-grained feature retrieval. On the other hand, since the trained model can be further subjected to vector fusion processing and the feature vector obtained after the vector fusion processing is used to continue model training, the representation capability of the feature vector can be improved, so that the model finally trained can support both text-image dialogue and feature vector retrieval. This can improve the effect of feature retrieval through the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a flowchart of a model training method provided by some embodiments of the present application;

[0013] Figure 2 is a flowchart of a model training method provided by some embodiments of the present application;

[0014] Figure 3 It is a schematic diagram of constructing training prompt words in the model training method provided in some embodiments of the present application;

[0015] Figure 4 is a flowchart of a model training method provided by some embodiments of the present application;

[0016] Figure 5 is a flowchart of a model training method provided by some embodiments of the present application;

[0017] Figure 6 is a schematic diagram of training a second model in a model training method provided in some embodiments of the present application;

[0018] Figure 7is a flowchart of a model training method provided by some embodiments of the present application;

[0019] Figure 8 is a schematic diagram of a method for determining a first eigenvector in a model training method provided in some embodiments of the present application;

[0020] Fig. 9 is a schematic diagram of a model training device provided in some embodiments of the present application;

[0021] Fig.10 is a schematic diagram of an electronic device provided by some embodiments of the present application;

[0022] Fig.11 It is a hardware schematic diagram of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0024] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0025] The term "indication" in this application can be a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, operations to be performed, or request results in the sent indication; an indirect indication can be understood as the receiver determining the corresponding information according to the indication sent by the sender, or making a judgment and determining the operation to be performed or the request result according to the judgment result.

[0026] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0027] First, some nouns or terms involved in the specification and claims of this application are explained below.

[0028] LLM: A large model for predicting text tokens built using multi-layer transformers. It can input text and output text. It is currently widely used in various natural language processing tasks such as dialogue systems, machine translation, and document writing.

[0029] MLLM: Based on LLM, it combines multimodal information and can input text, images, voice, etc. It can be used for various tasks of multimodal understanding, such as picture-text dialogue, image recognition, and voice recognition.

[0030] Multimodal retrieval large model: The retrieval large model further constructed by combining the advantages of MLLM not only supports multimodal dialogue, but also supports feature retrieval.

[0031] Feature encoder: It is generally a multi-layer neural network, whose function is to encode the original image or language signal into a set of vector representations.

[0032] Feature vector (Embedding): An image or language signal is converted into a set of vectors after passing through a feature encoder. Each element of the vector is a floating point number.

[0033] Explicit label association: After labeling a sample, associate other samples with the same label, and use the sample's label to find and recall other samples with the corresponding label.

[0034] Implicit vector association: After the samples are vectorized, related samples are associated based on the Euclidean distance between vectors. The closer the distance, the more related they are. Related samples are recalled through the closest distance.

[0035] Feature retrieval and recall: The above-mentioned labeling and vectorization methods can be used to obtain the features of the sample and build a feature pool. Given a query feature, the most relevant features can be recalled from the feature pool.

[0036] Prompt: The prompt template for large model training is an input method that limits the input of the large model to a specific range, thereby better controlling the output results of the model and better allowing the large model to complete the task as required.

[0037] The model training method, device, electronic device and readable storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0038] The model training method provided in the embodiment of the present application can be applied to model training for picture-text feature recommendation scenarios or picture-text question-answering scenarios, etc.

[0039] Exemplarily, the first model can be trained based on the first eigenvector and the second eigenvector to obtain the second model, wherein the first eigenvector is a eigenvector corresponding to an object area in the first sample image, and the second eigenvector is a eigenvector corresponding to an image included in the first sample set, whose Euclidean distance with the first eigenvector satisfies a threshold requirement, and the first sample set includes the first sample image; and based on the second sample image, the first-layer eigenvector and the second-layer eigenvector and the third eigenvector corresponding to the second model are determined, the second sample image includes an image and text, and the third eigenvector is a eigenvector corresponding to the image area in the second sample image; and the first-layer eigenvector and the third eigenvector are vector fused to obtain a fourth eigenvector, and the second-layer eigenvector and the third eigenvector are vector fused to obtain a fifth eigenvector; and based on the fourth eigenvector and the fifth eigenvector, the second model is trained to obtain the third model.

[0040] In this way, when a user needs to retrieve a certain image and text sample, a image and text sample to be retrieved can be input into the above-mentioned third model, and then the third model can output at least one image and text retrieval result matching the image and text sample; when a user needs to ask a question about the content in a certain image and text sample, the image and text sample and the question text can be input into the above-mentioned third model, and then the third model can output the answer text corresponding to the question text, as well as at least one image and text retrieval result matching the image and text sample and the question text.

[0041] Through the above scheme, on the one hand, since the model can be trained based on the feature vector of the object area in the image, the trained model can pay more attention to the local object information in the image, and can focus on fine-grained object information when performing feature retrieval, thereby achieving fine-grained feature retrieval. On the other hand, since the trained model can be subjected to vector fusion processing, and the trained model can continue to be trained based on the feature vector obtained after the vector fusion processing, the representation ability of the feature vector can be improved, so that the final trained model can support both image-text dialogue and feature vector retrieval. In this way, the effect of feature retrieval through the model can be improved.

[0042] It should be noted that the model training method provided in the embodiments of the present application can be executed by a model training device, an electronic device, or a functional module in an electronic device, etc. In some embodiments of the present application, the model training method provided in the embodiments of the present application is described by taking an electronic device executing the model training method as an example.

[0043] Figure 1 FIG. 1 is a flow chart showing a model training method provided by some embodiments of the present application. Figure 1 As shown, the model training method provided by some embodiments of the present application may include the following steps 101 to 104.

[0044] Step 101: The electronic device trains a first model based on a first eigenvector and a second eigenvector to obtain a second model.

[0045] Among them, the above-mentioned first feature vector is a feature vector corresponding to the object area in the first sample image, and the above-mentioned second feature vector is a feature vector corresponding to the image included in the first sample set, and the Euclidean distance with the first feature vector meets the threshold requirement, and the first sample set includes the first sample image.

[0046] In some embodiments of the present application, the first sample image may be a common objects in context (COCO) image or a segmented business scene image.

[0047] In some embodiments of the present application, the object area may be an image area corresponding to at least one object in the first sample image.

[0048] In some embodiments of the present application, the first sample set may be a sample set including the first sample image; the first sample set may include at least one sample image and object segmentation mask information corresponding to each sample image.

[0049] For example, the first sample set includes 2.3 million sample image-object segmentation mask information pairs, of which the number of COCO image samples is 900,000 and the number of segmentation business scene image samples is 1.4 million.

[0050] It should be noted that some sample images in the first sample set may not have corresponding object segmentation mask information. In this case, an open source segmentation model (Segment Anything Model, SAM) may be used to extract the object segmentation mask information corresponding to these sample images.

[0051] In some embodiments of the present application, all feature vectors corresponding to the images included in the first sample set are feature vectors converted from all object segmentation mask information in the first sample set. Each object segmentation mask information can be converted to obtain a feature vector.

[0052] In some embodiments of the present application, the above-mentioned satisfying the threshold requirement may include: being less than or equal to a Euclidean distance threshold.

[0053] In some embodiments of the present application, the above-mentioned Euclidean distance threshold may be preset by the system, or may be arbitrarily set according to actual usage requirements.

[0054] It should be noted that the Euclidean distance, also known as the Euclidean distance, is the true distance between two points in an m-dimensional space, where m is an integer greater than 1.

[0055] It can be understood that the closer the Euclidean distance between two eigenvectors is, the greater the similarity between the two eigenvectors is; conversely, the farther the Euclidean distance between two eigenvectors is, the less similarity between the two eigenvectors is.

[0056] In some embodiments of the present application, the first model may be LLM or MLLM.

[0057] The specific method for determining the first eigenvector and the second eigenvector will be described in detail in the following embodiments, and will not be described here in order to avoid repetition.

[0058] In some embodiments of the present application, Figure 1 ,like Figure 2 As shown, the above step 101 can be specifically implemented through the following steps 101a to 101c.

[0059] Step 101a: The electronic device converts the first feature vector into a first representation vector, and converts the second feature vector into a second representation vector.

[0060] In some embodiments of the present application, the first feature vector may be converted into the first representation vector, and the second feature vector may be converted into the second representation vector, through a 4-layer image converter (also referred to as an object encoder).

[0061] For example, assuming that the first eigenvector is represented by V mask , then V mask Through the above 4-layer image converter, the hidden vector of the last layer can be used as the first representation vector V obj = Transformer(V mask )[4th_hidden_state].

[0062] For example, assuming that the second eigenvector is represented by R mask , then R mask Through the above 4-layer image converter, the hidden vector of the last layer can be used as the second representation vector R obj = Transformer(R mask )[4th_hidden_state].

[0063] It should be noted that the representation vector is the final representation vector of the similar objects.

[0064] Step 101b: The electronic device determines a training prompt word based on the first representation vector and the second representation vector.

[0065] In some embodiments of the present application, the role of the above-mentioned training prompt words is to enable the model to effectively learn the sorting relationship between similar objects.

[0066] In some embodiments of the present application, the specific form of the training prompt word can be: "[V obj ]What is this, Top[Related sorting index][R obj ][Object label name] similarity score is [correlation score].

[0067] Among them, V obj That is, the first representation vector, R obj That is, the second representation vector mentioned above; [V obj ] is the query feature vector used for retrieval, and V obj Replace; [R obj ] To retrieve the closest object feature vector, use R obj Replace; [Related sort index] indicates the order calculated based on the Euclidean distance of the vectors, for example, 1, 2, 3, etc.; [Object label name] indicates V obj The tag name, for example, cat or dog; [Relevance score] indicates V obj and Robj The normalized Euclidean distance between .

[0068] For example, Figure 3 As shown, the segmented image region is the image region of a golden retriever, and its corresponding vector is V obj , and based on V obj Find the vector R of similar dog species in the dog category obj , for example, Golden Retriever, Labrador Retriever and Cocker Spaniel, etc. obj Middle and V obj The most similar dog is the golden retriever, followed by the Labrador and the Cocker Spaniel. The corresponding [Relevance Sorting Index] are 1, 2, and 3; the [Relevance Score] are 0.99, 0.8, and 0.6. Therefore, the training prompt word constructed here is: [V obj ]What is this, Top1[R obj ]The similarity score of Golden Retriever is 0.99.

[0069] Step 101c: The electronic device trains the first model based on the training prompt words to obtain the second model.

[0070] In some embodiments of the present application, the first model may be trained based on the training prompt words by using a large language model to predict the next token, and the model weights may be updated using a back propagation algorithm to obtain the second model.

[0071] In some embodiments of the present application, since the first representation vector and the second representation vector can be determined based on the first feature vector and the second feature vector respectively, and then the training prompt words are determined based on the first representation vector and the second representation vector, and the model is trained based on the training prompt words, the trained model can pay more attention to fine-grained object information, thereby improving the retrieval effect of the model.

[0072] Step 102: The electronic device determines a first-layer feature vector, a second-layer feature vector, and a third feature vector corresponding to a second model based on the second sample image.

[0073] The second sample image includes an image and text, and the third feature vector is a feature vector corresponding to an image region in the second sample image.

[0074] In some embodiments of the present application, the above-mentioned second sample image may be a sample pair including an image and text, and the second sample image may be a sample image in a second sample set, and the second sample set may include picture-text question-and-answer sample pairs and comparative learning sample pairs, wherein the comparative learning sample pairs can be automatically constructed using the click behavior of Internet users, and a large number of samples can be collected for training.

[0075] For example, the co-occurrence score can be used To construct the sample pairs in the above sample pair set, the co-occurrence score

[0076] In which, define n i is a sample point for a retrieval, containing relevant content {t i ,d i ,I i}, respectively the title, description and image of the sample point, n A and n B represents two specific sample points A and B. So the collinearity score above represents the co-occurrence score from node A to node B, represents the set of users who viewed node A and then clicked node B. u It means that a user u has viewed the total number of sample points. With the above definition, and calculating the co-occurrence scores of all node pairs, we can get a specific node n i The co-occurrence score set S ni ={S ni→nj |i≠j}, and sort the co-occurrence scores of the set from large to small to get node n i The most relevant nodes are used to construct contrastive learning sample pairs. ni The calculations are all based on user behavior, so a large number of training samples can be obtained without manual labeling.

[0077] The following is a rational explanation of a specific method for the electronic device to determine the first-layer feature vector and the second-layer feature vector based on the second sample image.

[0078] For example, assuming that the content of the above sample pair is {t i ,d i ,I i}, then the electronic device can {t i ,d i ,I i} are spliced ​​together to form cat(t i ,d i ,I i), and execute the forward calculation process of the large language model to obtain the last layer feature vector and the middle layer feature vector of the second model, wherein the middle layer feature vector can be the first layer feature vector, and the last layer feature vector can be the second layer feature vector; for example, if the LLM used is a 32-layer converter, the last layer feature vector is The feature vector of the middle layer is the hidden state vector of the 16th layer, which is In addition, the above image vector needs to be retained, that is, V = image encoder (I i ).

[0079] Step 103: The electronic device performs vector fusion processing on the first layer feature vector and the third feature vector to obtain a fourth feature vector, and performs vector fusion processing on the second layer feature vector and the third feature vector to obtain a fifth feature vector.

[0080] In some embodiments of the present application, the above-mentioned vector fusion processing may also be referred to as cross-layer vector fusion processing, which is used to make the two fused sets of vectors pay extra attention to image information, thereby improving the image and text retrieval effect.

[0081] In some embodiments of the present application, Figure 1 ,like Figure 4 As shown, the above step 103 can be specifically implemented through the following step 103a.

[0082] Step 103a: The electronic device performs vector fusion processing on the first layer feature vector and the third feature vector through a fusion function to obtain a fourth feature vector, and performs vector fusion processing on the second layer feature vector and the third feature vector to obtain a fifth feature vector.

[0083] In some embodiments of the present application, the above fusion function may also be called a cross-layer vector fusion function, and the fusion function may be a sigmoid function.

[0084] It should be noted that the sigmoid function is an S-shaped function commonly seen in biology, also known as an S-shaped growth curve. In information science, due to its monotonic and inverse monotonic properties, the sigmoid function is often used as an activation function for neural networks to map variables between 0 and 1. In addition, the sigmoid function also has important applications in logistic regression and is sometimes referred to as a logistic function. The characteristic of this function is that its derivative can be expressed by the function itself. The sigmoid function is not only used in logistic regression in machine learning, but is also widely used in deep learning as an activation function for the hidden layer of a neural network and the output layer of a two-category classification.

[0085] In some embodiments of the present application, the electronic device may obtain the fourth eigenvector and the fifth eigenvector by the following formula (1):

[0086]

[0087] in, represents the fourth eigenvector mentioned above, represents the fifth eigenvector, v is the third eigenvector, [,] represents the cascade operation, W and b represent the learnable weights and biases of the linear layer, and * represents the element-wise product. From the above calculation, we can see that the final fusion feature incorporates the third eigenvector v through the sigmoid function, so that the final feature pays extra attention to the image information and improves the image and text retrieval effect.

[0088] In some embodiments of the present application, since the electronic device can perform vector fusion processing on the first-layer feature vector and the third feature vector through the above-mentioned fusion function, and perform vector fusion processing on the second-layer feature vector and the third feature vector, the third feature vector can be fused in, so that the final feature pays additional attention to image information, thereby improving the image and text retrieval effect.

[0089] Step 104: The electronic device trains the second model based on the fourth eigenvector and the fifth eigenvector to obtain a third model.

[0090] In some embodiments of the present application, the first-layer feature vector is a feature vector of the first layer of the second model, and the second-layer feature vector is a feature vector of the second layer of the second model. Figure 1 ,like Figure 5 As shown, the above step 104 can be specifically implemented through the following steps 104a to 104c.

[0091] Step 104a: The electronic device determines a contrastive learning loss value of the first layer based on the fourth eigenvector.

[0092] Step 104a: The electronic device determines a contrastive learning loss value of the second layer based on the fifth eigenvector.

[0093] In some embodiments of the present application, the electronic device may determine the contrastive learning loss value of the first layer and the contrastive learning loss value of the second layer by the following formula (2):

[0094]

[0095] Among them, 2B is n i Construct the total batch of similar sample pairs, That is the most relevant comparison sample, sim(a,b)=a T b / (|a||b|), γ is a trainable parameter. L v is the contrastive learning loss value of the first layer, L m is the contrastive learning loss value of the second layer, and α is a constant used to adjust L v and L m The contribution of L is the final total contrastive learning loss value. Based on the final L and using the back propagation algorithm, the weight of MLLRM can be updated. After training is completed, it can be used for subsequent reasoning.

[0096] Step 104a: The electronic device trains the second model based on the contrastive learning loss value of the first layer and the contrastive learning loss value of the second layer to obtain a third model.

[0097] Among them, the contrastive learning loss value is used to train the retrieval ability of the model.

[0098] In some embodiments of the present application, after determining the contrastive learning loss value of the first layer and the contrastive learning loss value of the second layer, the electronic device can add the contrastive learning loss value of the first layer, the contrastive learning loss value of the second layer and the cross entropy loss value to obtain a final loss value, and use the final loss value to train the second model to obtain the third model. The cross entropy loss value is used to train the model's text-to-picture dialogue capability.

[0099] For example, Figure 6 As shown, the prompt words used for the contrast loss value are for n i Do compression, that is, the input is "compressed sample n i is a sentence", and the output is the final sample vector, which is used to convert cat(t i ,d i ,I i ) into a suitable feature vector. The hint word used in the cross entropy loss value is based on n i The contrastive learning loss value training and the cross entropy loss value training are performed alternately, that is, this batch is contrastive learning, and the next batch is cross entropy. In this way, the trained model can have both picture-text dialogue capabilities and picture-text retrieval capabilities.

[0100] In some embodiments of the present application, since the electronic device can train the above-mentioned second model based on the contrastive learning loss value, the trained above-mentioned third model can have both image-text dialogue capabilities and image-text retrieval capabilities, thereby enhancing the functionality of the model.

[0101] In the model training method provided in the embodiment of the present application, on the one hand, since the model can be trained based on the feature vector of the object area in the image, the trained model can pay more attention to the local object information in the image, and can focus on fine-grained object information when performing feature retrieval, thereby achieving fine-grained feature retrieval. On the other hand, since the trained model can be subjected to vector fusion processing, and the trained model can continue to be trained based on the feature vector obtained after the vector fusion processing, the representation ability of the feature vector can be improved, so that the model finally obtained by training can support both text-image dialogue and feature vector retrieval. In this way, the effect of feature retrieval through the model can be improved.

[0102] In some embodiments of the present application, Figure 1 ,like Figure 7 As shown, before the above step 101, the model training method provided in some embodiments of the present application may also include the following steps 105 to 107.

[0103] Step 105: The electronic device obtains an image feature vector of the first sample image and mask information of the object region.

[0104] In some embodiments of the present application, the object region is the object region in the first sample image.

[0105] In some embodiments of the present application, the electronic device may obtain the above-mentioned image feature vector through an image encoder, and obtain the above-mentioned mask information through a segmentation model or an existing segmentation annotation.

[0106] Step 106: The electronic device performs a binary operation on the image feature vector based on the mask information of the object area to obtain a first feature vector.

[0107] For example, Figure 8 As shown, the mask information O mask is a binary matrix, where the shaded pixels represent the mask. After being converted to a binary matrix, the value at the mask is 1 and the value at the non-mask is 0. Then, the object region feature vector in the above image feature vector V can be extracted through binary operation and determined as the above first feature vector, denoted as V mask =V[flatten(O mask )].

[0108] Step 107: The electronic device determines, as a second feature vector, a feature vector whose Euclidean distance with the first feature vector meets a threshold requirement among feature vectors corresponding to the images included in the first sample set.

[0109] Exemplarily, the electronic device can extract feature vectors from the object images and object mask information in the first sample set, and construct vector pools of different types according to categories (for example, cats or dogs, etc.), and then select the first feature vector as the query vector, and retrieve similar object vectors from the same type of pool according to the Euclidean distance between vectors, which are the second feature vectors, and can be sorted from large to small according to the similarity score.

[0110] In some embodiments of the present application, since the electronic device can determine the above-mentioned first feature vector based on the image feature vector of the sample image and the mask information of the image area, the first feature vector determined can accurately reflect the fine-grained information of the object through the object area information in the sample image, and determine a similar second feature vector with reference to the first feature vector. In this way, based on the first feature vector and the second feature vector, the trained model can pay more attention to fine-grained information.

[0111] In some embodiments of the present application, after the above step 104, the model training method provided by some embodiments of the present application may also include the following steps 108 and 109.

[0112] Step 108: The electronic device inputs the image and text samples to be retrieved into the third model.

[0113] In some embodiments of the present application, the above-mentioned image and text samples to be retrieved may be samples including images and texts.

[0114] Step 109: The electronic device outputs at least one graphic sample through the third model.

[0115] The similarity between each of the at least one image and text sample and the image and text sample to be retrieved is greater than or equal to a similarity threshold.

[0116] In some embodiments of the present application, the above similarity threshold may be preset by the system, or may be arbitrarily set according to actual usage requirements.

[0117] Exemplarily, through the above-mentioned third model, a sample vector pool can be constructed, and all the image and text samples are executed through the forward calculation process of the model to obtain a vector pool of all samples. The image and text samples viewed by the user also obtain sample vectors in the same way and serve as query vectors. Based on the distance between the vectors, several samples with the smallest distance (i.e., several closest samples) are found from the full sample vector pool as at least one similar image and text sample to be recalled. In addition, users can also ask questions about the image and text samples they have viewed based on the third model, such as asking what the main content of this sample is. The large model can automatically help summarize and improve the user experience. In this way, the large model can simultaneously realize the functions of image and text question and answer and image and text sample similarity retrieval. On this basis, these similar samples can be further recommended to users in subsequent lists. Because they are content that users are interested in, users will continue to watch, thereby improving user stickiness and experience.

[0118] In some embodiments of the present application, since the image and text samples to be retrieved are input into the above-mentioned third model and at least one image and text sample is output through the third model, the retrieval and question-answering of image and text samples can be realized through the third model to meet the user's various needs.

[0119] The model training method provided by some embodiments of the present application includes a training stage and a prediction stage. The training stage can also include: Training stage 1: a training stage based on mask information, so that the model focuses on local object information in the image, for example, some important entities such as animals, plants or people in the image; Training stage 2: a training stage based on image-text question-answering and cross-layer multi-branch fusion comparative learning, while improving the model's question-answering and feature retrieval capabilities. After the model training is completed, it can be used in business scenarios, namely the prediction stage. The prediction stage is also divided into two sub-stages, namely: 1. Use the MLLRM model to recall image and text samples similar to user interest samples; 2. Recommend similar image and text samples to target users.

[0120] Exemplarily, the first training stage may include the following steps:

[0121] Step 1011: prepare the first-stage image samples I, including COCO, refCOCOg and images of segmented business scenes, as well as object segmentation mask information; a total of 2.3 million image-segmentation pairs, of which the COCO sample size is 900,000 and the business scene sample size is 1.4 million.

[0122] Step 1012: The image I is decoded by an image encoder to obtain a feature vector V of the image, and the image I is decoded by a segmentation model or an existing segmentation annotation to obtain mask information O of the object region. mask .

[0123] Among them, due to O maskIt is a binary matrix. The feature vector of the object area in V can be extracted through binary operation, which is recorded as V mask =V[flatten(O mask )].

[0124] Step 1013: Calculate the representation vector V of the fine-grained object in the original image obj , which is used to put the features of fine-grained objects into the model for training together, and improve the model's ability to focus on fine-grained objects. The specific implementation process is:

[0125] The V obtained in step 1012 is mask Through a 4-layer image converter, the hidden vector of the last layer is used as the representation vector V of the object obj = Transformer(V mask )[4th_hidden_state].

[0126] Step 1014: V mask As the benchmark object feature vector, similar objects are retrieved from the training set in step 1011 for subsequent model sorting training to improve the model's ability to distinguish similar objects. The specific implementation process is as follows:

[0127] For the object images and object mask information in the training set, extract V according to step 1013 mask And build different types of vector pools according to categories, select one of them V mask As the query vector, the similar object vectors are retrieved from the same pool according to the Euclidean distance between the vectors, denoted as R mask , and can be sorted from large to small according to the similarity score, and then the final representation vector R of similar objects is extracted through a 4-layer image converter obj = Transformer(R mask )[4th_hidden_state].

[0128] Step 1015: construct a first-stage training prompt word prompt and train a first-stage model. The role of prompt is to enable the model to effectively learn the sorting relationship of similar objects.

[0129] Exemplarily, the second training stage may include the following steps:

[0130] Step 1021: Use the weights trained in the first stage to initialize the image encoder, object encoder and LLM of the second stage, that is, the second stage continues to be trained on the basis of the first stage.

[0131] Step 1022: prepare the second-stage image samples and construct image-text question-answer pairs and comparative learning sample pairs. The comparative learning sample pairs are automatically constructed using the click behavior of Internet users, and a large number of samples can be collected for training.

[0132] Step 1023: Calculate the last hidden layer vector and the middle hidden layer vector for subsequent cross-layer fusion.

[0133] Step 1024: Use sigmoid gate to fuse cross-layer vectors so that both sets of vectors pay extra attention to image information, thereby improving image and text retrieval effects.

[0134] Step 1025: Calculate the final loss value loss. The final loss includes contrastive learning loss and cross entropy loss (predicting next-token). Contrastive learning loss is used to train the retrieval ability of the model, and cross entropy loss is used to train the image-text dialogue ability of the model.

[0135] Exemplarily, the above prediction stage may include the following steps:

[0136] Step 103: Use the model trained in the second training stage to recall image and text samples similar to the user's interest samples.

[0137] Through the above-mentioned trained model, a sample vector pool can be constructed, and all image and text samples are executed through the forward calculation process of the model to obtain a vector pool of all samples. The image and text samples viewed by the user also obtain sample vectors in the same way and serve as query vectors. Based on the distance between the vectors, several samples with the smallest distance are found from the full sample vector pool as similar image and text samples to be recalled. In addition, users can also ask questions about the image and text samples they have viewed based on the multimodal large model, such as asking what the main content of this sample is. The large model can automatically help summarize and improve the user experience. In this way, the large model can simultaneously realize the functions of image and text question and answer and image and text sample similarity retrieval.

[0138] Step 104: recommend similar image and text samples to target users.

[0139] With the similar image and text samples recalled in step 103, the recommendation system will further recommend these similar samples to the user in the subsequent list. Since the content is of interest to the user, the user will continue to watch it, thereby improving user stickiness and experience.

[0140] In this way, the first stage of training above trains the image encoder and object encoder based on feature mask, so that the model pays more attention to local object information. The second stage of training above trains a model that can support both text-to-picture dialogue and retrieval based on text-to-picture question-answering and cross-layer multi-branch fusion comparative learning, which not only makes the representation vector output by the model more able to express implicit associations, but also does not forget the text-to-picture dialogue capability. In addition, user interaction behavior is used during model comparative training, which can construct a large number of training samples and save manpower. The trained model can be used in text-to-picture dialogue scenarios and similarity recall scenarios, providing strong technical support for recommendation recall and machine dialogue.

[0141] The above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or, under the premise that there is no contradiction, can also be executed in combination with each other. The specific implementation can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0142] In some embodiments of the present application, the model training method provided by the model training device may be executed by the model training device. In some embodiments of the present application, the model training device provided by some embodiments of the present application is described by taking the model training method executed by the model training device as an example.

[0143] like Fig. 9 As shown, some embodiments of the present application provide a model training device 90 , which may include: a training module 91 , a determination module 92 , and a processing module 93 .

[0144] Among them, the training module 91 can be used to train the first model based on the first feature vector and the second feature vector to obtain the second model, wherein the first feature vector is the feature vector corresponding to the object area in the first sample image, and the second feature vector is the feature vector corresponding to the image included in the first sample set, and the Euclidean distance with the first feature vector meets the threshold requirement, and the first sample set includes the first sample image. The determination module 92 can be used to determine the first layer feature vector and the second layer feature vector and the third feature vector corresponding to the second model based on the second sample image, wherein the second sample image includes an image and text, and the third feature vector is the feature vector corresponding to the image area in the second sample image. The processing module 93 can be used to perform vector fusion processing on the first layer feature vector and the third feature vector determined by the determination module 92 to obtain the fourth feature vector, and perform vector fusion processing on the second layer feature vector and the third feature vector determined by the determination module 92 to obtain the fifth feature vector. The training module 91 can also be used to train the second model based on the fourth feature vector and the fifth feature vector obtained by the processing module 93 to obtain the third model.

[0145] In some embodiments of the present application, the training module 91 can be specifically used to convert the above-mentioned first feature vector into a first representation vector, and convert the above-mentioned second feature vector into a second representation vector; and based on the first representation vector and the second representation vector, determine the training prompt word; and based on the training prompt word, train the above-mentioned first model to obtain the above-mentioned second model.

[0146] In some embodiments of the present application, the processing module 93 can be specifically used to perform vector fusion processing on the above-mentioned first-layer feature vector and the above-mentioned third feature vector through a fusion function to obtain the above-mentioned fourth feature vector, and to perform vector fusion processing on the above-mentioned second-layer feature vector and the third feature vector to obtain the above-mentioned fifth feature vector.

[0147] In some embodiments of the present application, the first-layer feature vector is a feature vector of the first layer of the second model, and the second-layer feature vector is a feature vector of the second layer of the second model. The training module 91 can be specifically used to determine the contrastive learning loss value of the first layer based on the fourth feature vector; and to determine the contrastive learning loss value of the second layer based on the fifth feature vector; and to train the second model based on the contrastive learning loss value of the first layer and the contrastive learning loss value of the second layer to obtain the third model; wherein the contrastive learning loss value is used to train the retrieval ability of the model.

[0148] In some embodiments of the present application, the processing module 93 can also be used to obtain the image feature vector of the first sample image and the mask information of the object area before the training module 91 trains the first model based on the first feature vector and the second feature vector to obtain the second model; and based on the mask information of the object area, perform a binary operation on the image feature vector to obtain the first feature vector; and determine the feature vector whose Euclidean distance with the first feature vector among all feature vectors corresponding to the first sample set is less than or equal to the Euclidean distance threshold as the second feature vector.

[0149] In the model training device provided in some embodiments of the present application, on the one hand, since the model can be trained based on the feature vector of the object area in the image, the trained model can pay more attention to the local object information in the image, and can focus on fine-grained object information when performing feature retrieval, thereby achieving fine-grained feature retrieval. On the other hand, since the trained model can be subjected to vector fusion processing, and the trained model can continue to be trained based on the feature vector obtained after the vector fusion processing, the representation ability of the feature vector can be improved, so that the final trained model can support both text-image dialogue and feature vector retrieval. In this way, the effect of feature retrieval through the model can be improved.

[0150] The model training device in some embodiments of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobilepersonal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiments of the present application are not specifically limited.

[0151] The model training device in some embodiments of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

[0152] The model training device provided in some embodiments of the present application can implement the various processes implemented in the above-mentioned method embodiments and achieve the same technical effects. To avoid repetition, it will not be described here.

[0153] like Fig.10 As shown, some embodiments of the present application also provide an electronic device 100, including a processor 101 and a memory 102, wherein the memory 102 stores programs or instructions that can be executed on the processor 101, and when the program or instructions are executed by the processor 101, the various steps of the above-mentioned model training method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0154] It should be noted that the electronic devices in some embodiments of the present application include mobile electronic devices and non-mobile electronic devices.

[0155] Fig.11 A schematic diagram of the hardware structure of an electronic device for implementing some embodiments of the present application.

[0156] like Fig.11As shown, the electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0157] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.11 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0158] Among them, the processor 1010 can be used to train the first model based on the first feature vector and the second feature vector to obtain the second model, the first feature vector is the feature vector corresponding to the object area in the first sample image, and the second feature vector is the feature vector whose Euclidean distance with the first feature vector meets the threshold requirement among all feature vectors corresponding to the first sample set, and the first sample set includes the first sample image; and based on the second sample image, determine the first-layer feature vector and the second-layer feature vector and the third feature vector corresponding to the second model, the second sample image includes an image and text, and the third feature vector is the feature vector corresponding to the image area in the second sample image; and perform vector fusion processing on the first-layer feature vector and the third feature vector to obtain a fourth feature vector, and perform vector fusion processing on the second-layer feature vector and the third feature vector to obtain a fifth feature vector; and based on the fourth feature vector and the fifth feature vector, train the second model to obtain the third model.

[0159] In some embodiments of the present application, the processor 1010 can be specifically used to convert the above-mentioned first feature vector into a first representation vector, and convert the above-mentioned second feature vector into a second representation vector; and based on the first representation vector and the second representation vector, determine the training prompt word; and based on the training prompt word, train the above-mentioned first model to obtain the above-mentioned second model.

[0160] In some embodiments of the present application, the processor 1010 can be specifically used to perform vector fusion processing on the above-mentioned first-layer feature vector and the above-mentioned third feature vector through a fusion function to obtain the above-mentioned fourth feature vector, and perform vector fusion processing on the above-mentioned second-layer feature vector and the third feature vector to obtain the above-mentioned fifth feature vector.

[0161] In some embodiments of the present application, the first-layer feature vector is a feature vector of the first layer of the second model, and the second-layer feature vector is a feature vector of the second layer of the second model. The processor 1010 can be specifically used to determine the contrastive learning loss value of the first layer based on the fourth feature vector; and determine the contrastive learning loss value of the second layer based on the fifth feature vector; and train the second model based on the contrastive learning loss value of the first layer and the contrastive learning loss value of the second layer to obtain the third model; wherein the contrastive learning loss value is used to train the retrieval ability of the model.

[0162] In some embodiments of the present application, the processor 1010 may also be used to obtain the image feature vector of the first sample image and the mask information of the object area before training the first model based on the first feature vector and the second feature vector to obtain the second model; and based on the mask information of the object area, perform a binary operation on the image feature vector to obtain the first feature vector; and determine the feature vector whose Euclidean distance with the first feature vector among all feature vectors corresponding to the first sample set is less than or equal to the Euclidean distance threshold as the second feature vector.

[0163] In the electronic devices provided in some embodiments of the present application, on the one hand, since the model can be trained based on the feature vector of the object area in the image, the trained model can pay more attention to the local object information in the image, and can focus on fine-grained object information when performing feature retrieval, thereby achieving fine-grained feature retrieval. On the other hand, since the trained model can be subjected to vector fusion processing, and the trained model can continue to be trained based on the feature vector obtained after the vector fusion processing, the representation ability of the feature vector can be improved, so that the model finally obtained by training can support both text-image dialogue and feature vector retrieval. In this way, the effect of feature retrieval through the model can be improved.

[0164] It should be understood that in some embodiments of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0165] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in some embodiments of the present application includes but is not limited to these and any other suitable types of memory.

[0166] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0167] Some embodiments of the present application also provide a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned model training method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0168] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0169] Some embodiments of the present application further provide a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned model training method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0170] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0171] Some embodiments of the present application provide a computer program, which is stored in a storage medium and is executed by at least one processor to implement the various processes of the above-mentioned model training method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0172] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0174] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Based on a first feature vector and a second feature vector, the first model is trained to obtain a second model, wherein the first feature vector is a feature vector corresponding to an object region in a first sample image, and the second feature vector is a feature vector corresponding to an image included in a first sample set, and the Euclidean distance with the first feature vector meets a threshold requirement, and the first sample set includes the first sample image; Determine a first-layer feature vector and a second-layer feature vector, as well as a third feature vector corresponding to the second model based on a second sample image, wherein the second sample image includes an image and text, and the third feature vector is a feature vector corresponding to an image region in the second sample image; Performing vector fusion processing on the first layer feature vector and the third feature vector to obtain a fourth feature vector, and performing vector fusion processing on the second layer feature vector and the third feature vector to obtain a fifth feature vector; The second model is trained based on the fourth eigenvector and the fifth eigenvector to obtain a third model.

2. The method according to claim 1, characterized in that The step of training the first model based on the first feature vector and the second feature vector to obtain the second model includes: Converting the first feature vector into a first representation vector, and converting the second feature vector into a second representation vector; Determine a training prompt word based on the first representation vector and the second representation vector; Based on the training prompt words, the first model is trained to obtain the second model.

3. The method according to claim 1, characterized in that The performing vector fusion processing on the first layer feature vector and the third feature vector to obtain a fourth feature vector, and performing vector fusion processing on the second layer feature vector and the third feature vector to obtain a fifth feature vector, comprises: Through the fusion function, the first layer feature vector and the third feature vector are subjected to vector fusion processing to obtain the fourth feature vector, and the second layer feature vector and the third feature vector are subjected to vector fusion processing to obtain the fifth feature vector.

4. The method according to claim 1, characterized in that: The first layer feature vector is a feature vector of the first layer of the second model, and the second layer feature vector is a feature vector of the second layer of the second model; The step of training the second model based on the fourth eigenvector and the fifth eigenvector to obtain a third model comprises: Determining a contrastive learning loss value of the first layer based on the fourth eigenvector; Based on the fifth eigenvector, determining a contrastive learning loss value of the second layer; Based on the contrastive learning loss value of the first layer and the contrastive learning loss value of the second layer, training the second model to obtain the third model; Among them, the contrastive learning loss value is used to train the retrieval ability of the model.

5. The method according to any one of claims 1 to 4, characterized in that Before training the first model based on the first eigenvector and the second eigenvector to obtain the second model, the method further includes: Acquire an image feature vector of the first sample image and mask information of the object area; Based on the mask information of the object area, performing a binary operation on the image feature vector to obtain the first feature vector; Among the feature vectors corresponding to the images included in the first sample set, a feature vector whose Euclidean distance with the first feature vector meets a threshold requirement is determined as the second feature vector.

6. A model training device, characterized in that: The device comprises: a training module, a determination module and a processing module; The training module is used to train the first model based on a first feature vector and a second feature vector to obtain a second model, wherein the first feature vector is a feature vector corresponding to an object area in a first sample image, and the second feature vector is a feature vector corresponding to an image included in a first sample set, and the Euclidean distance with the first feature vector meets a threshold requirement, and the first sample set includes the first sample image; The determination module is used to determine the first layer feature vector and the second layer feature vector and the third feature vector corresponding to the second model based on the second sample image, wherein the second sample image includes an image and text, and the third feature vector is a feature vector corresponding to the image area in the second sample image; The processing module is used to perform vector fusion processing on the first layer feature vector determined by the determination module and the third feature vector to obtain a fourth feature vector, and perform vector fusion processing on the second layer feature vector determined by the determination module and the third feature vector to obtain a fifth feature vector; The training module is further used to train the second model based on the fourth eigenvector and the fifth eigenvector obtained by the processing module to obtain a third model.

7. The device according to claim 6, characterized in that The training module is specifically used to convert the first feature vector into a first representation vector and convert the second feature vector into a second representation vector; and determine a training prompt word based on the first representation vector and the second representation vector; and train the first model based on the training prompt word to obtain the second model.

8. The device according to claim 6, characterized in that The processing module is specifically used to perform vector fusion processing on the first layer feature vector and the third feature vector through a fusion function to obtain the fourth feature vector, and to perform vector fusion processing on the second layer feature vector and the third feature vector to obtain the fifth feature vector.

9. The device according to claim 6, characterized in that The first layer feature vector is a feature vector of the first layer of the second model, and the second layer feature vector is a feature vector of the second layer of the second model; The training module is specifically used to determine the contrastive learning loss value of the first layer based on the fourth eigenvector; and determine the contrastive learning loss value of the second layer based on the fifth eigenvector; and train the second model based on the contrastive learning loss value of the first layer and the contrastive learning loss value of the second layer to obtain the third model; Among them, the contrastive learning loss value is used to train the retrieval ability of the model.

10. The device according to any one of claims 5 to 9, characterized in that The processing module is further used to train the first model based on the first feature vector and the second feature vector in the training module, and before obtaining the second model, obtain the image feature vector of the first sample image and the mask information of the object area; and based on the mask information of the object area, perform a binary operation on the image feature vector to obtain the first feature vector; and determine the feature vector corresponding to the image included in the first sample set, whose Euclidean distance with the first feature vector meets the threshold requirement, as the second feature vector.

11. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the model training method as described in any one of claims 1 to 5 are implemented.

12. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the steps of the model training method described in any one of claims 1 to 5 are implemented.

Images