Vector retrieval model training, vector recall method and electronic device

Abstract

The embodiments of the present application disclose vector retrieval model training, vector recall method and electronic device, wherein the training method includes: obtaining a training data set, wherein the training data set includes multiple data objects, multiple query conditions and corresponding user behavior sequences; using a pre-trained representation model to obtain the query conditions and the multimodal fusion representation vectors of the data objects; generating a semantically enhanced multimodal fusion representation vector corresponding to the data object; by inputting the query conditions and the multimodal fusion representation vectors of the data objects into the user tower in the vector retrieval model, and introducing the interaction between the multimodal fusion representation vectors of the query conditions and the multimodal fusion representation vectors of the data objects through the user behavior sequence, so as to generate a semantically enhanced multimodal fusion representation vector corresponding to the user. Through the embodiments of the present application, the model training efficiency can be improved, and the efficiency and relevance of personalized vector retrieval recall products can be enhanced.

Description

Technical Field

[0001] The present application relates to the field of personalized vector recall technology, and in particular to vector retrieval model training, vector recall methods and electronic equipment. Background Art

[0002] Personalized vector recall is a technology used in recommendation systems. Its core is to use deep learning models to express users and search objects in vectors. Specifically, the system maps users and search objects to a low-dimensional dense vector space. By calculating the similarity between user vectors and search object vectors, the system can find and recall the search objects that best match the user's interests. The advantage of this method is that the vectors on the search object side can be calculated offline, and matching search objects can be quickly found online through efficient vector retrieval methods. At the same time, more user features and contextual information can be incorporated to provide more personalized recommendations.

[0003] Personalized vector recall technology also plays an important role in product search and recall in product information service systems (also known as "e-commerce platforms", etc.), which can provide differentiated and efficient candidate products based on users' personalized preferences. Initially, personalized vector recall mainly expressed vectors based on the text features of users and products. However, many recent studies have shown that adding multimodal representation to personalized vector recall can improve the accuracy and semantic relevance of its recalled products. Therefore, personalized vector recall retrieval based on multimodal representation is a very worthy research topic.

[0004] There is a personalized vector recall method based on multimodal representation in the prior art. Specifically, a multimodal feature extraction module is added to the personalized vector model to perform text representation and image representation of the product, and obtain the product text vector and product image vector respectively. The two are merged to obtain the multimodal representation vector of the product, and then the similarity is calculated with the user's text representation vector to determine the matching products.

[0005] Although the above scheme can realize personalized vector recall based on multimodal representation to a certain extent, on the one hand, it can only realize multimodal representation of commodities, but not multimodal representation of user features; since the personalized vector recall model is a "dual-tower structure", namely "commodity tower" and "user tower", which are used to represent commodities and users respectively, if multimodal representation of users is required, it is necessary to add a multimodal feature extraction module to the "user tower", which will cause a surge in the number of parameters of the personalized vector recall model and seriously affect the model performance. In addition, since the "commodity tower" and the "user tower" are independent of each other, even if multimodal modules are added to the "commodity tower" and the "user tower", the interaction between the multimodal representation of commodities and users cannot be realized, and it is difficult to effectively enhance the efficiency and relevance of personalized vector retrieval recall products. Summary of the invention

[0006] The present application provides vector retrieval model training, vector recall methods and electronic devices, which can improve model training efficiency and enhance the efficiency and relevance of personalized vector retrieval recall products.

[0007] This application provides the following solutions:

[0008] A vector retrieval model training method, comprising:

[0009] Acquire a training data set, wherein the training data set includes a plurality of data objects, a plurality of query conditions and corresponding user behavior sequences, wherein the user behavior sequence includes at least one data object information associated with the query condition and associated with a user operation behavior;

[0010] Using the pre-trained representation model to obtain the multimodal fusion representation vector of the query condition and the multimodal fusion representation vector of the data object;

[0011] By inputting the multimodal fusion representation vector of the data object into the data object tower in the vector retrieval model of the double-tower structure, a semantically enhanced multimodal fusion representation vector corresponding to the data object is generated;

[0012] By inputting the query condition and the multimodal fusion representation vector of the data object into the user tower in the vector retrieval model, and introducing the interaction between the multimodal fusion representation vector of the query condition and the multimodal fusion representation vector of the data object through the user behavior sequence, a semantically enhanced multimodal fusion representation vector corresponding to the user is generated;

[0013] The similarity between the semantically enhanced multimodal fusion representation vector corresponding to the data object and the semantically enhanced multimodal fusion representation vector corresponding to the user is calculated, and multiple rounds of iterative calculations are performed according to a preset loss function to complete the training of the vector retrieval model.

[0014] The method of using a pre-trained representation model to obtain a multimodal fusion representation vector of a query condition and a multimodal fusion representation vector of a data object includes:

[0015] Using a pre-trained text representation model to obtain text representation vectors corresponding to the query condition and the data object, and using a pre-trained image representation model to obtain image representation vectors corresponding to the query condition and the data object, respectively;

[0016] The text representation vector of the query condition is fused with the image representation vector to obtain a multimodal fusion representation vector of the query condition, and the text representation vector of the data object is fused with the image representation vector to obtain a multimodal fusion representation vector of the data object.

[0017] The step of generating a semantically enhanced multimodal fusion representation vector corresponding to the data object includes:

[0018] The multimodal fusion representation vector of the data object is input into the data object tower in the vector retrieval model so as to fuse the multimodal fusion representation vector of the data object with the basic representation vector of the data object modeled by the vector retrieval model to obtain a semantically enhanced multimodal fusion representation vector corresponding to the data object.

[0019] The step of generating a semantically enhanced multimodal fusion representation vector corresponding to the user includes:

[0020] Obtaining a user multimodal behavior sequence according to the multimodal fusion representation vectors respectively corresponding to the data objects included in the user behavior sequence;

[0021] By using an attention mechanism, the multimodal fusion representation vector of the query condition corresponding to the user behavior sequence is fused with the multimodal fusion representation vector of the data object in the user multimodal behavior sequence to obtain a multimodal fusion representation vector of the user;

[0022] The multimodal fusion representation vector of the query condition, the multimodal fusion representation vector corresponding to the user, and the basic representation vector of the user modeled by the vector retrieval model are fused to obtain a semantically enhanced multimodal fusion representation vector corresponding to the user.

[0023] Among them, when fusing different representation vectors, it includes:

[0024] Different representation vectors are connected, and the multi-layer perceptron structure is used to reduce the dimension of the connected high-dimensional representation vector to obtain a low-dimensional fused representation vector.

[0025] The performing of multiple rounds of iterative calculations according to a preset loss function to complete the training of the vector retrieval model includes:

[0026] The result of the similarity calculation is used as an estimated value of the click rate that can be obtained by the corresponding data object when the user inputs the query condition, and the contrastive learning loss function is used to maximize the distance between the estimated values ​​of the click rates of the positive and negative samples to train the vector retrieval model;

[0027] The positive and negative samples are determined based on the click conditions of users on the data objects in the training data set.

[0028] A personalized vector recall method, comprising:

[0029] After receiving a search request submitted by a user, determining a query condition associated with the search request and a user behavior sequence associated with the query condition, wherein the user behavior sequence includes at least one data object information associated with the query condition and associated with the user operation behavior;

[0030] Obtaining a multimodal fusion representation vector of the query condition and a multimodal representation vector of a plurality of data objects in a database; the multimodal fusion representation vector of the query condition and the multimodal representation vector of the data object are generated in advance according to a pre-trained representation model;

[0031] By inputting the multimodal fusion representation vector of the data object into the data object tower in the vector retrieval model of the double-tower structure, a semantically enhanced multimodal fusion representation vector corresponding to the data object is generated;

[0032] By inputting the query condition and the multimodal fusion representation vector of the data object into the user tower in the vector retrieval model, and introducing the interaction between the multimodal fusion representation vector of the query condition and the multimodal fusion representation vector of the data object through the user behavior sequence, a semantically enhanced multimodal fusion representation vector corresponding to the user is generated;

[0033] A similarity calculation is performed on the semantically enhanced multimodal fusion representation vector corresponding to the data object and the semantically enhanced multimodal fusion representation vector corresponding to the user, and a personalized vector recall result is determined according to the similarity calculation result.

[0034] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the methods described above.

[0035] An electronic device, comprising:

[0036] one or more processors; and

[0037] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of any of the methods described above.

[0038] A computer program product comprises a computer program / computer executable instructions, wherein the computer program / computer executable instructions implement the steps of any of the aforementioned methods when executed by a processor in an electronic device.

[0039] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0040] Through the embodiment of the present application, the pre-trained representation model can be used to realize the multimodal fusion representation of the query conditions and the data objects, and then the pre-obtained multimodal fusion representation can be used as the input information of the vector retrieval model to generate semantically enhanced multimodal fusion representation vectors for the data objects and users respectively. Among them, when the semantically enhanced multimodal fusion representation vector is generated for the user on the user tower side, the interaction between the multimodal fusion representation vector of the query condition and the multimodal fusion representation vector of the data object can also be introduced through the user behavior sequence. In this way, since the pre-trained representation model is used to generate the multimodal representation vector, it is not necessary to add a multimodal representation module to the vector retrieval model, so it can avoid causing the number of parameters of the vector retrieval model to be too large, and the rich prior knowledge of the pre-trained representation model itself can be fully utilized. In addition, the interaction between the multimodal fusion representation vector of the query condition and the multimodal fusion representation vector of the data object can also be realized. The similarity between the semantically enhanced multimodal fusion representation vectors generated for the data object and the user can more accurately express the user's estimated score for the click rate of the specific data object, thereby improving the training efficiency of the model.

[0041] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 It is a schematic diagram of the system architecture provided by the embodiment of the present application;

[0044] Figure 2 is a flow chart of the model training method provided in an embodiment of the present application;

[0045] Figure 3 is a flow chart of a personalized vector recall method provided in an embodiment of the present application;

[0046] Figure 4 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0048] In order to facilitate understanding of the specific implementation scheme provided by the embodiment of the present application, the concept of the "double tower" structure of the vector retrieval model is first briefly introduced below. The "tower" here does not refer to a tower in the physical sense, but borrows architectural terms to vividly describe the model structure. The so-called double tower structure refers to the use of two independent neural networks to encode users and data objects (such as commodities, which can be called Items) respectively, which is suitable for processing sparse vectors (such as high-dimensional feature vectors with most elements being zero) retrieval tasks, and has high retrieval efficiency and good scalability. Among them, the user tower (User Tower) is responsible for encoding the features on the user side and generating user vector representations. These features may include user attributes, historical behaviors, etc. The data object tower (Item Tower) is responsible for encoding the features on the data object side and generating data object vector representations. These features may include data object attributes, text descriptions, labels, etc.

[0049] During the training phase, the dual-tower model learns effective user and data object representations by maximizing the inner product (or similarity measure) between the user vector and the corresponding positive sample data object vector, while minimizing the inner product between the user vector and the corresponding negative sample data object vector. In this way, users and data objects are mapped to the same latent feature space.

[0050] In the reasoning stage (also known as the retrieval stage), given a user, its vector representation can be obtained through the user tower, and then the similarity is calculated with all data object vectors (pre-calculated and stored by the data object tower), so as to quickly find the data objects most relevant to the user and complete the retrieval task.

[0051] Since the user tower and the data object tower are independent of each other, and in practical applications, only one pre-calculation of the data object tower is often required, this structure is conducive to efficient retrieval under large-scale data.

[0052] In an embodiment of the present application, an improvement is made to the implementation scheme of adding multimodal representation to the personalized vector retrieval model based on the personalized vector retrieval model of the above-mentioned dual-tower structure. In the improved scheme, firstly, the multimodal fusion representation vectors of Query (query conditions such as keywords input by the user) and Item (that is, the object to be queried, for example, in the commodity information service system, it can be a commodity, and in other systems it can be other types of data objects, such as articles, etc.) can be obtained respectively through the pre-trained representation model, and then, the multimodal fusion representation vectors of Query and Item can be used as input information of the vector retrieval model to realize the introduction of multimodal representation.

[0053] That is to say, in the embodiment of the present application, the multimodal module is not directly added to the vector retrieval model, but the multimodal fusion representation vectors of the query and item are obtained in advance through the pre-trained representation model, and then input into the vector retrieval model as known information. In this way, it can avoid causing the number of parameters of the vector retrieval model to be too large, and can also use the rich prior knowledge of the pre-trained representation model itself to achieve a more comprehensive and accurate multimodal representation vector for the query and item.

[0054] Among them, Figure 1 As shown, the above-mentioned pre-trained representation model can be divided into a pre-trained text representation model and a pre-trained image representation model, which are respectively used to extract the text representation vectors of the query and the item, and the image representation vectors of the query and the item. Then, the text representation vector of the query can be fused with the image representation vector to obtain a multimodal fusion representation vector of the query, and the text representation vector of the item can be fused with the image representation vector to obtain a multimodal fusion representation vector of the item.

[0055] In addition, specifically in the process of introducing the above-mentioned multimodal fusion representation vectors of Query and Item into the vector retrieval model, on the data object tower (I tower) side, the multimodal fusion representation vector of Item can be input into the I tower of the vector retrieval model to generate a semantically enhanced multimodal fusion representation vector corresponding to Item. On the user tower (U tower) side, both the multimodal fusion representation vectors of Query and Item can be input into the user tower, and the interaction between the multimodal fusion representation vector of Query and the multimodal fusion representation vector of Item can be introduced through the user behavior sequence (which may include at least one Item information associated with the user operation behavior and related to the Query), thereby generating a semantically enhanced multimodal fusion representation vector corresponding to the user.

[0056] Finally, the semantically enhanced multimodal fusion representation vector corresponding to the Item and the semantically enhanced multimodal fusion representation vector corresponding to the user can be similarly calculated, and the personalized vector recall result can be determined based on the similarity calculation result. Alternatively, in the process of training the vector retrieval model, multiple rounds of iterative calculations can be performed according to a preset loss function to complete the training of the vector retrieval model.

[0057] Through the above method, it is possible to avoid making the number of parameters of the vector retrieval model too large, realize the interaction between the multimodal fusion representation vector of the query and the multimodal fusion representation vector of the item, and make full use of the rich prior knowledge of the pre-trained representation model itself to obtain a better multimodal fusion effect.

[0058] The specific implementation scheme provided in the embodiments of the present application is described in detail below.

[0059] Embodiment 1

[0060] First, this embodiment provides a vector retrieval model training method, see Figure 2 , the method may include:

[0061] S201: Acquire a training data set, wherein the training data set includes a plurality of data objects, a plurality of query conditions and corresponding user behavior sequences, wherein the user behavior sequence includes at least one data object information related to the query condition and associated with a user operation behavior.

[0062] Among them, the data objects (Item) in the training data set can be all the data objects in the database, for example, all the commodities in the commodity information service system, etc. The query conditions (Query) and user behavior sequences can be obtained from the historical search records, where the Query can be a large number of queries used by users in the historical search process, including input keywords, etc. The user behavior sequence can include data objects related to the specific Query that the user has operated. For example, a user enters "down jacket" as a query to initiate a search at a certain moment. In the period before this, he clicked to view some "down jacket" products, and added some "down jacket" products to favorites or shopping carts, etc.; in addition, after initiating the search, he also clicked to view the details of a certain product in the search results, or added a certain search result to the shopping cart, and even purchased a certain product, etc., then among the products associated with the operation behaviors before and after the search, the products related to "down jacket" can constitute the user behavior sequence corresponding to the query "down jacket". For example, a user behavior sequence may be (product 1, product 2, product 3...). Of course, in specific implementation, specific behavior type information may also be associated with specific products, for example, click, add to cart, purchase, and so on.

[0063] Among them, the operation behaviors before the search can be divided into the operation behaviors generated on the day of the search and the historical operation behaviors generated in the recent period (for example, 30 days, etc.). Of course, different users may enter the same query condition to initiate a search, but the products associated with the operation behaviors of different users before and after the search are usually different. Therefore, the same query can correspond to the user behavior sequences generated by multiple different users.

[0064] S202: Using a pre-trained representation model to obtain a multimodal fusion representation vector of a query condition and a multimodal fusion representation vector of a data object.

[0065] After obtaining the training data set, you can use the pre-trained representation model to obtain the multimodal fusion representation vector of the query and the multimodal fusion representation vector of the item. The so-called multimodal fusion representation vector is a vector formed by fusing the representations of multiple modalities such as text and image for the same representation object.

[0066] Among them, the so-called pre-trained representation model is a model dedicated to performing functions such as feature extraction. Pre-trained models play an important role in natural language processing and image processing. They can better understand and process large amounts of text and image data. Pre-trained models can help text and image data to extract features and vector representations, thereby improving the accuracy of downstream tasks. In specific implementation, the pre-trained representation model may include a pre-trained text representation model, a pre-trained image representation model, etc., which are used to extract text features and image features, respectively. Therefore, in a more specific implementation scheme of an embodiment of the present application, the pre-trained text representation model can be used to first obtain the text representation vectors corresponding to the Query and Item, respectively, and the pre-trained image representation model can be used to obtain the image representation vectors corresponding to the Query and Item, respectively; then the text representation vector of the Query can be fused with the image representation vector to obtain a multimodal fusion representation vector of the Query, and the text representation vector of the Item can be fused with the image representation vector to obtain a multimodal fusion representation vector of the Item.

[0067] Among them, typical pre-trained text representation models include BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer); pre-trained image representation models include BLIP (Bootstrapping Language-Image Pre-training), CLIP (Contrastive Language-Image Pre-training), etc. The embodiment of the present application can use the above pre-trained representation models to complete the generation of text representation vectors and image representation vectors.

[0068] Specifically, for the pre-trained text representation model, the title of the query and item can be used as input, and the mean of the token representation vector of the last layer of the text representation of the pre-trained model can be used as output to obtain the text vector representation of the query and item. For the pre-trained image representation model, assuming that the BLIP model is used, the picture of the item (such as a product) can be input into the image encoder of BLIP to obtain the image representation vector of the item; for the query, since the query itself may be pure text content, the query can be input into BLIP's Q-Former (a lightweight Transformer model used to achieve efficient alignment between vision and language, using a learnable query vector to extract the most useful visual features for text generation from the frozen image encoder) to obtain the image representation vector of the query.

[0069] After obtaining the image representation vector and text representation vector of Query and Item, the image representation vector and the text representation vector can be fused to obtain a multimodal fusion representation vector. Among them, in order to fully fuse the image representation vector and the text representation vector and reduce the memory usage during online prediction, the embodiment of the present application also provides a multimodal representation fusion model. The model includes two multimodal fusion dimensionality reduction structures of Query and Item. For the Query multimodal fusion dimensionality reduction structure, the text representation vector and the image representation vector of the aforementioned extracted Query can be first connected (for example, the "concat()" method can be used, which is used to connect two or more arrays, does not change the existing array, but only returns a copy of the connected array). Since the vector obtained after direct connection may be very long, for example, the text representation vector may be 768-dimensional, and the image representation vector may be 256-dimensional, the two connected together will obtain a vector of more than 1000 dimensions. Therefore, after the connection, structures such as MLP (Multilayer Perceptron) can also be used for dimensionality reduction processing, and finally a low-dimensional multimodal fusion vector representation is obtained, for example, it can be 128-dimensional, etc., to reduce the occupancy of memory space. Similarly, for Item, the model can also connect the text representation vector of the Item extracted above with the image representation vector, and then use MLP to perform fusion and dimensionality reduction, and finally obtain a low-dimensional Item multimodal fusion representation vector.

[0070] That is to say, in the embodiment of the present application, the multimodal representation module is not directly added to the vector retrieval model, but the text features and image features of the Query and Item are extracted in advance through pre-trained text representation models, image representation models, etc., and then further fused into the multimodal fusion representation vector of the Query and the multimodal fusion representation vector of the Item.

[0071] It should be noted that, in specific implementation, the multimodal fusion representation vectors of Query and Item can also be obtained by other means. For example, some pre-trained representation models themselves have the ability to understand cross-modal interactions. Therefore, the pre-trained multimodal model can also be directly used to extract multimodal features of Query and Item.

[0072] After obtaining the multimodal fusion representation vector of the query and the multimodal fusion representation vector of the item, this information can be saved for subsequent vector retrieval calculations.

[0073] S203: Generate a semantically enhanced multimodal fusion representation vector corresponding to the data object by inputting the multimodal fusion representation vector of the data object into the data object tower in the vector retrieval model of the double-tower structure.

[0074] The above steps obtain the multimodal fusion representation vector of the query and the multimodal fusion representation vector of the item, which can be saved. Afterwards, this multimodal fusion representation vector can be used as input information of the vector retrieval model and participate in subsequent vector retrieval calculations.

[0075] Specifically, the personalized vector retrieval model is a recall model that can convert users, query conditions, and data objects into representation vectors, and then use these vectors to quickly calculate<U,Q, I> The vector inner product of . For the specified<U, Q> , the personalized vector retrieval model will take the top K items with the highest internal score ranking as the final retrieval results. At present, the personalized vector retrieval model has been widely used in search systems, and it can return different search results for different users. However, the existing personalized vector retrieval model is mainly oriented to efficiency modeling, and the model lacks modeling of semantic information such as text and images. Therefore, the recalled products are mainly some high-efficiency and high-explosive products (product definitions, performance, quality or prices are significantly different from existing products, greatly exceeding user expectations and triggering word-of-mouth spread). However, for the search system, due to the limitation of the query words, the user's purchase intention is relatively certain. Therefore, only considering the efficiency of the product and ignoring the correlation between the recalled product and the user's search query will seriously affect the user experience and be unfavorable to the user's transaction and conversion.

[0076] In order to improve the efficiency and relevance of the personalized vector retrieval model, the embodiment of the present application proposes a personalized vector retrieval model based on multimodal fusion representation. The model can adopt a dual-tower structure, that is, it can be composed of a User tower and an Item tower, which calculates the inner product of the semantically enhanced multimodal user representation vector output by the User tower and the semantically enhanced multimodal Item representation vector output by the Item tower, and then uses the obtained inner integral as the estimated score of CTR (Click-Through Rate).

[0077] In the Item tower of the personalized vector retrieval model based on multimodal fusion representation provided in the embodiment of the present application, the multimodal fusion representation vector of the Item obtained above and the basic vector modeled by the personalized vector retrieval model for the Item can be further fused to obtain a semantically enhanced multimodal representation vector of the Item. In other words, since the personalized vector retrieval model itself also has the modeling capability and can model the Item to obtain a representation vector, the representation vector can be used as a basic representation vector and further fused with the multimodal fusion representation vector of the Item obtained above to obtain richer information. Specifically, when performing the fusion, the multimodal fusion representation vector of the Item can also be first connected with the aforementioned basic representation vector (the concat() method can also be used), and then the connection result can be reduced in dimensionality through the MLP algorithm structure, thereby generating a low-dimensional (for example, 128-dimensional) semantically enhanced multimodal representation vector for the Item.

[0078] S204: By inputting the query condition and the multimodal fusion representation vector of the data object into the user tower in the vector retrieval model, and introducing the interaction between the multimodal fusion representation vector of the query condition and the multimodal fusion representation vector of the data object through the user behavior sequence, a semantically enhanced multimodal fusion representation vector corresponding to the user is generated.

[0079] On the User tower side, the multimodal fusion representation vectors of both Query and Item can be used as input information. Moreover, the user behavior sequence can be used as a bridge to introduce the interaction between the multimodal fusion representation vector of Query and the multimodal fusion representation vector of Item, and on this basis, a semantically enhanced multimodal fusion representation vector corresponding to the user can be generated.

[0080] In a specific implementation, the aforementioned Item multimodal fusion representation vector can be first combined with the user behavior sequence, and for each Item in the user behavior sequence, its Item multimodal fusion representation vector is obtained, thereby obtaining the user multimodal representation behavior sequence. In other words, assuming that a user behavior sequence is (item 1, item 2, item 3), the user multimodal representation behavior sequence can be (multimodal fusion representation vector of item 1, multimodal fusion representation vector of item 2, multimodal fusion representation vector of item 3).

[0081] Then, since the user behavior sequence is also related to the Query, the Query associated with the user behavior sequence can be determined, and the Query multimodal fusion representation output by the previous step and the user multimodal representation behavior sequence are fused based on the multimodal attention mechanism to express the semantic correlation between the Query and the specific Item in the sequence. The user multimodal behavior representation sequence is weighted and fused according to the semantic correlation to obtain the user multimodal vector.

[0082] Then, the query multimodal fusion representation vector, the user multimodal vector, and the basic user vector modeled by the personalized vector retrieval model can be further fused to generate an enhanced multimodal fusion representation vector for the user. Specifically, the aforementioned query multimodal fusion representation vector, the user multimodal vector, and the basic user vector can be connected, and then the connection result can be reduced in dimension through a structure such as MLP to obtain a low-dimensional (for example, 128-dimensional) enhanced multimodal fusion representation vector.

[0083] S205: Calculate the similarity between the semantically enhanced multimodal fusion representation vector corresponding to the data object and the semantically enhanced multimodal fusion representation vector corresponding to the user, and perform multiple rounds of iterative calculations according to a preset loss function to complete the training of the vector retrieval model.

[0084] Through the aforementioned steps S203 and S204, the semantically enhanced multimodal fusion representation vector corresponding to the item and the semantically enhanced multimodal fusion representation vector corresponding to the user are obtained. After that, the similarity between the two can be calculated. For example, an inner product calculation can be performed. The result obtained can be used as the click-through rate estimate of the corresponding item when the user enters the corresponding query. The training of the vector retrieval model is completed by optimizing the accuracy of the click-through rate estimate. For example, it can be used<U, Q, I> All click samples (when a user enters a query to search, if he clicks on an item, then the<U, Q, I> The vector retrieval model is trained by taking the current click sample (the current click sample) as the positive sample and performing negative sampling in the Batch (a batch can be a subset of the training data set), taking the samples other than the current click sample in the Batch as negative samples, and then using the contrastive learning loss function (such as Hinge Loss) to maximize the distance between the positive and negative samples.

[0085] In summary, through the embodiment of the present application, the pre-trained representation model can be used to realize the multimodal fusion representation of the query conditions and the data objects, and then the pre-obtained multimodal fusion representation can be used as the input information of the vector retrieval model to generate semantically enhanced multimodal fusion representation vectors for the data objects and users respectively. Among them, when the semantically enhanced multimodal fusion representation vector is generated for the user on the user tower side, the interaction between the multimodal fusion representation vector of the query condition and the multimodal fusion representation vector of the data object can also be introduced through the user behavior sequence. In this way, since the pre-trained representation model is used to generate the multimodal representation vector, it is not necessary to add a multimodal representation module to the vector retrieval model. Therefore, it is possible to avoid causing the number of parameters of the vector retrieval model to be too large, and the rich prior knowledge of the pre-trained representation model itself can be fully utilized. In addition, the interaction between the multimodal fusion representation vector of the query condition and the multimodal fusion representation vector of the data object can also be realized. The similarity between the semantically enhanced multimodal fusion representation vectors generated for the data object and the user can more accurately express the user's estimated score for the click rate of the specific data object, thereby improving the training efficiency of the model.

[0086] Embodiment 2

[0087] The above-mentioned embodiment 1 mainly introduces the training method of the vector retrieval model. In this embodiment 2, when the vector retrieval model is used for vector recall, the pre-trained representation model can also be used to obtain the query conditions and the multimodal fusion representation vector of the data object in advance and save them. When the user inputs a specific Query to initiate a retrieval, the recall result can be obtained by inputting the above-mentioned multimodal fusion representation vector into the vector retrieval model. Since the rich prior knowledge of the pre-trained representation model itself can be fully utilized, and the interaction between the multimodal fusion representation vector of the query condition and the multimodal fusion representation vector of the data object can be realized, the correlation between the recall result and the query condition can be improved to a certain extent.

[0088] Specifically, this second embodiment provides a personalized vector recall method, see Figure 3 , the method may specifically include:

[0089] S301: After receiving a search request submitted by a user, determine a query condition associated with the search request and a user behavior sequence associated with the query condition, wherein the user behavior sequence includes at least one data object information associated with the query condition and associated with the user operation behavior.

[0090] This second embodiment introduces the specific process of the reasoning stage. The so-called reasoning result is the process of using the vector retrieval model to perform specific vector recall after completing the training of the vector retrieval model. In this process, the user will enter a specific query to initiate a search request. Therefore, in a specific request, the specific user, query, and corresponding user behavior sequence can be fixed. Among them, the user behavior sequence can mainly be a sequence of items related to the current query in the items related to the historical operation behaviors or current operation behaviors (mainly some behaviors of the current search behavior on the day of the current search behavior) performed by the user before initiating the search request.

[0091] S302: Obtain a multimodal fusion representation vector of the query condition and multimodal representation vectors of multiple data objects in a database; the multimodal fusion representation vector of the query condition and the multimodal representation vector of the data object are generated in advance based on a pre-trained representation model.

[0092] Among them, the multimodal fusion representation vector of the query can also be generated in advance through a pre-trained representation model and saved in the database. In this way, after the user initiates a specific search request, the multimodal fusion representation vector of the currently input query can be obtained by querying the database. After receiving the specific query input by the user, it can be first determined whether the query exists in the database. If so, the multimodal fusion representation vector corresponding to the query can be directly retrieved; if the query does not exist in the database, multiple queries similar to the query can be retrieved from the database, and the average of the multimodal fusion representation vectors of these similar queries can be taken as the multimodal fusion representation vector of the query, and so on.

[0093] S303: Generate a semantically enhanced multimodal fusion representation vector corresponding to the data object by inputting the multimodal fusion representation vector of the data object into the data object tower in the vector retrieval model of the double-tower structure.

[0094] S304: By inputting the query condition and the multimodal fusion representation vector of the data object into the user tower in the vector retrieval model, and introducing the interaction between the multimodal fusion representation vector of the query condition and the multimodal fusion representation vector of the data object through the user behavior sequence, a semantically enhanced multimodal fusion representation vector corresponding to the user is generated.

[0095] Steps S303 and S304 are basically the same as steps S203 and S204 and will not be described in detail here.

[0096] S305: Calculate the similarity between the semantically enhanced multimodal fusion representation vector corresponding to the data object and the semantically enhanced multimodal fusion representation vector corresponding to the user, and determine a personalized vector recall result according to the similarity calculation result.

[0097] After obtaining the semantically enhanced multimodal fusion representation vectors corresponding to the item and the user, the similarity calculation can be performed between the two. Of course, there will be multiple items, and the semantically enhanced multimodal fusion representation vector corresponding to the user can be used to perform similarity calculations with the semantically enhanced multimodal fusion representation vectors corresponding to each item. The similarity calculation results obtained can be used to represent the probability of the user clicking on the corresponding item under the current query state predicted by the model, and then vector recall can be performed based on the calculation results. For example, the items with similarity calculation results higher than a certain threshold or ranked in the top N can be returned to the user as recall results, and so on.

[0098] In this way, since we can make full use of the rich prior knowledge of the pre-trained representation model itself, and realize the interaction between the multimodal fusion representation vector of the query condition and the multimodal fusion representation vector of the data object, we can improve the relevance of the recall results to a certain extent, improve the quality of the recall results, and thus improve the user experience.

[0099] For the parts not described in detail in the second embodiment, please refer to the description of the first embodiment and other parts of this specification, which will not be repeated here.

[0100] It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

[0101] Corresponding to the first embodiment, the present application also provides a vector retrieval model training device, which may include:

[0102] A training data set acquisition unit, used to acquire a training data set, wherein the training data set includes a plurality of data objects, a plurality of query conditions and corresponding user behavior sequences, wherein the user behavior sequence includes at least one data object information associated with the query condition and associated with the user operation behavior;

[0103] A multimodal representation unit, used to obtain a multimodal fusion representation vector of a query condition and a multimodal fusion representation vector of a data object using a pre-trained representation model;

[0104] A first semantic enhancement unit, configured to generate a semantically enhanced multimodal fusion representation vector corresponding to the data object by inputting the multimodal fusion representation vector of the data object into a data object tower in a vector retrieval model of a double-tower structure;

[0105] A second semantic enhancement unit is used to generate a semantically enhanced multimodal fusion representation vector corresponding to the user by inputting the query condition and the multimodal fusion representation vector of the data object into the user tower in the vector retrieval model and introducing the interaction between the multimodal fusion representation vector of the query condition and the multimodal fusion representation vector of the data object through the user behavior sequence;

[0106] The similarity calculation unit is used to calculate the similarity between the semantically enhanced multimodal fusion representation vector corresponding to the data object and the semantically enhanced multimodal fusion representation vector corresponding to the user, and perform multiple rounds of iterative calculations according to a preset loss function to complete the training of the vector retrieval model.

[0107] The multimodal representation unit can be specifically used for:

[0108] Using a pre-trained text representation model to obtain text representation vectors corresponding to the query condition and the data object, and using a pre-trained image representation model to obtain image representation vectors corresponding to the query condition and the data object, respectively;

[0109] The text representation vector of the query condition is fused with the image representation vector to obtain a multimodal fusion representation vector of the query condition, and the text representation vector of the data object is fused with the image representation vector to obtain a multimodal fusion representation vector of the data object.

[0110] The first semantic enhancement unit can be specifically used to: input the multimodal fusion representation vector of the data object into the data object tower in the vector retrieval model, so as to fuse the multimodal fusion representation vector of the data object with the basic representation vector of the data object modeled by the vector retrieval model, and obtain a semantically enhanced multimodal fusion representation vector corresponding to the data object.

[0111] The second semantic enhancement unit may be specifically configured to: obtain a user multimodal behavior sequence according to the multimodal fusion representation vectors respectively corresponding to the data objects included in the user behavior sequence;

[0112] By using an attention mechanism, the multimodal fusion representation vector of the query condition corresponding to the user behavior sequence is fused with the multimodal fusion representation vector of the data object in the user multimodal behavior sequence to obtain a multimodal fusion representation vector of the user;

[0113] The multimodal fusion representation vector of the query condition, the multimodal fusion representation vector corresponding to the user, and the basic representation vector of the user modeled by the vector retrieval model are fused to obtain a semantically enhanced multimodal fusion representation vector corresponding to the user.

[0114] When fusing different representation vectors, the different representation vectors can be connected, and the connected high-dimensional representation vectors can be reduced in dimensionality using a multi-layer perceptron structure to obtain a low-dimensional fused representation vector.

[0115] Specifically, during training, the result of the similarity calculation can be used as an estimated value of the click rate that can be obtained by the corresponding data object when the user inputs the query condition, and the vector retrieval model is trained using a contrastive learning loss function with the goal of maximizing the distance between the estimated values ​​of the click rates of the positive and negative samples;

[0116] The positive and negative samples are determined based on the click conditions of users on the data objects in the training data set.

[0117] Corresponding to the second embodiment, the embodiment of the present application further provides a personalized vector recall device, which may include:

[0118] A request receiving unit, configured to, after receiving a search request submitted by a user, determine a query condition associated with the search request and a user behavior sequence associated with the query condition, wherein the user behavior sequence includes at least one data object information associated with the query condition and associated with the user operation behavior;

[0119] A multimodal vector acquisition unit, used to acquire a multimodal fusion representation vector of the query condition and multimodal representation vectors of multiple data objects in a database; the multimodal fusion representation vector of the query condition and the multimodal representation vector of the data object are generated in advance according to a pre-trained representation model;

[0120] A first semantic enhancement unit, configured to generate a semantically enhanced multimodal fusion representation vector corresponding to the data object by inputting the multimodal fusion representation vector of the data object into a data object tower in a vector retrieval model of a double-tower structure;

[0121] A second semantic enhancement unit is used to generate a semantically enhanced multimodal fusion representation vector corresponding to the user by inputting the query condition and the multimodal fusion representation vector of the data object into the user tower in the vector retrieval model and introducing the interaction between the multimodal fusion representation vector of the query condition and the multimodal fusion representation vector of the data object through the user behavior sequence;

[0122] The similarity calculation unit is used to calculate the similarity between the semantically enhanced multimodal fusion representation vector corresponding to the data object and the semantically enhanced multimodal fusion representation vector corresponding to the user, and determine the personalized vector recall result according to the similarity calculation result.

[0123] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0124] And an electronic device, comprising:

[0125] one or more processors; and

[0126] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0127] A computer program product includes a computer program / computer executable instructions, which, when executed by a processor in an electronic device, implement the steps of the method described in the aforementioned method embodiment.

[0128] in, Figure 4 The architecture of the electronic device is shown as an example, which may include a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, and a memory 420. The processor 410, the video display adapter 411, the disk drive 412, the input / output interface 413, the network interface 414, and the memory 420 may be communicatively connected via a communication bus 430.

[0129] Among them, the processor 410 can be implemented by a general-purpose CPU (Central Processing Unit, processor), a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solution provided in this application.

[0130] The memory 420 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 420 can store an operating system 421 for controlling the operation of the electronic device 400, and a basic input and output system (BIOS) for controlling the low-level operation of the electronic device 400. In addition, a web browser 423, a data storage management system 424, and a vector retrieval processing system 425, etc. can also be stored. The above-mentioned vector retrieval processing system 425 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided in the present application is implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0131] The input / output interface 413 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0132] The network interface 414 is used to connect to a communication module (not shown) to achieve communication interaction between the device and other devices. The communication module can achieve communication through a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WIFI, Bluetooth, etc.).

[0133] The bus 430 comprises a pathway for transmitting information between the various components of the device (eg, the processor 410 , the video display adapter 411 , the disk drive 412 , the input / output interface 413 , the network interface 414 , and the memory 420 ).

[0134] It should be noted that, although the above device only shows a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, a memory 420, a bus 430, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0135] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0136] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0137] The above is a detailed introduction to the vector retrieval model training, vector recall method and electronic device provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A vector retrieval model training method, characterized in that: include: Acquire a training data set, wherein the training data set includes a plurality of data objects, a plurality of query conditions and corresponding user behavior sequences, wherein the user behavior sequence includes at least one data object information associated with the query condition and associated with a user operation behavior; Using the pre-trained representation model to obtain the multimodal fusion representation vector of the query condition and the multimodal fusion representation vector of the data object; By inputting the multimodal fusion representation vector of the data object into the data object tower in the vector retrieval model of the double-tower structure, a semantically enhanced multimodal fusion representation vector corresponding to the data object is generated; The multimodal fusion representation vector of the query condition and the data object is input into the user tower in the vector retrieval model, and the multimodal fusion representation vectors corresponding to the data objects included in the user behavior sequence are respectively obtained, and the multimodal fusion representation vector of the query condition corresponding to the user behavior sequence is fused with the multimodal fusion representation vector of the data object in the user multimodal behavior sequence through an attention mechanism to obtain the multimodal fusion representation vector of the user; The multimodal fusion representation vector of the query condition, the multimodal fusion representation vector corresponding to the user, and the basic representation vector of the user modeled by the vector retrieval model are fused to obtain a semantically enhanced multimodal fusion representation vector corresponding to the user; The similarity between the semantically enhanced multimodal fusion representation vector corresponding to the data object and the semantically enhanced multimodal fusion representation vector corresponding to the user is calculated, and multiple rounds of iterative calculations are performed according to a preset loss function to complete the training of the vector retrieval model.

2. The method according to claim 1, characterized in that The method of using the pre-trained representation model to obtain the multimodal fusion representation vector of the query condition and the multimodal fusion representation vector of the data object includes: Using a pre-trained text representation model to obtain text representation vectors corresponding to the query condition and the data object, and using a pre-trained image representation model to obtain image representation vectors corresponding to the query condition and the data object, respectively; The text representation vector of the query condition is fused with the image representation vector to obtain a multimodal fusion representation vector of the query condition, and the text representation vector of the data object is fused with the image representation vector to obtain a multimodal fusion representation vector of the data object.

3. The method according to claim 1, characterized in that The generating of the semantically enhanced multimodal fusion representation vector corresponding to the data object includes: The multimodal fusion representation vector of the data object is input into the data object tower in the vector retrieval model so as to fuse the multimodal fusion representation vector of the data object with the basic representation vector of the data object modeled by the vector retrieval model to obtain a semantically enhanced multimodal fusion representation vector corresponding to the data object.

4. The method according to claim 2 or 3, characterized in that: When fusing different representation vectors, it includes: Different representation vectors are connected, and the multi-layer perceptron structure is used to reduce the dimension of the connected high-dimensional representation vector to obtain a low-dimensional fused representation vector.

5. The method according to claim 2 or 3, characterized in that: The performing multiple rounds of iterative calculations according to a preset loss function to complete the training of the vector retrieval model includes: The result of the similarity calculation is used as an estimated value of the click rate that can be obtained by the corresponding data object when the user inputs the query condition, and the contrastive learning loss function is used to maximize the distance between the estimated values ​​of the click rates of the positive and negative samples to train the vector retrieval model; The positive and negative samples are determined based on the click conditions of users on the data objects in the training data set.

6. A personalized vector recall method, characterized in that: include: After receiving a search request submitted by a user, determining a query condition associated with the search request and a user behavior sequence associated with the query condition, wherein the user behavior sequence includes at least one data object information associated with the query condition and associated with the user operation behavior; Obtaining a multimodal fusion representation vector of the query condition and a multimodal representation vector of a plurality of data objects in a database; the multimodal fusion representation vector of the query condition and the multimodal representation vector of the data object are generated in advance according to a pre-trained representation model; By inputting the multimodal fusion representation vector of the data object into the data object tower in the vector retrieval model of the double-tower structure, a semantically enhanced multimodal fusion representation vector corresponding to the data object is generated; The multimodal fusion representation vector of the query condition and the data object is input into the user tower in the vector retrieval model, and the multimodal fusion representation vectors corresponding to the data objects included in the user behavior sequence are respectively obtained, and the multimodal fusion representation vector of the query condition corresponding to the user behavior sequence is fused with the multimodal fusion representation vector of the data object in the user multimodal behavior sequence through an attention mechanism to obtain the multimodal fusion representation vector of the user; The multimodal fusion representation vector of the query condition, the multimodal fusion representation vector corresponding to the user, and the basic representation vector of the user modeled by the vector retrieval model are fused to obtain a semantically enhanced multimodal fusion representation vector corresponding to the user; A similarity calculation is performed on the semantically enhanced multimodal fusion representation vector corresponding to the data object and the semantically enhanced multimodal fusion representation vector corresponding to the user, and a personalized vector recall result is determined according to the similarity calculation result.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

8. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 6.

9. A computer program product comprising a computer program / computer executable instructions, characterized in that: When the computer program / computer executable instructions are executed by a processor in an electronic device, the steps of the method according to any one of claims 1 to 6 are implemented.

Images