Product mounting, retrieval, recommendation, training processing method, device and electronic equipment

Through the feature extraction model trained by comparative learning and adversarial learning technology, the problem of semantic gap between multimedia data and product data is solved, cross-domain alignment is achieved, and the matching effect of content retrieval and recommendation is improved.

CN113420166BActive Publication Date: 2024-11-26ALIBABA INNOVATION PRIVATE LIMITED
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110328304.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-03-26
Publication Date
2024-11-26
Estimated Expiration
2041-03-26

AI Technical Summary

Technical Problem

The prior art is difficult to match and integrate multimedia data and product data, resulting in poor cross-domain content recommendation and product mount effects, mainly due to the huge semantic gap between the two.

Method used

The feature extraction model is trained using contrast learning and adversarial learning techniques. By obtaining sample combinations of multiple content domains, feature extraction and contrast learning training are performed, and feature vectors with cross-domain alignment are generated to achieve matching and recommendations of multimedia data and product data. .

Benefits of technology

Effectively narrow the distance of the internal feature vectors of cross-domain sample combinations, amplify the distance of the feature vectors between cross-domain sample combinations, realize cross-domain alignment of multimedia data and product data, and improve the matching effect of content retrieval and recommendation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113420166B_ABST
    Figure CN113420166B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention provides a method, device and electronic device for product mounting, retrieval, recommendation and training processing, wherein the training method includes: obtaining samples associated with content in multiple content domains to form multiple cross-domain sample combinations; using the trained feature extraction model to extract features from the original data of the samples in the cross-domain sample combination to generate feature vectors corresponding to the samples; performing contrastive learning training on the feature extraction model to reduce the distance between the feature vectors corresponding to the samples within the cross-domain sample combination and to enlarge the distance between the feature vectors corresponding to the samples between the cross-domain sample combinations as the training target. The embodiment of the present invention trains the feature extraction model through contrastive learning to achieve feature extraction of data from different content domains. The feature vector has the property of cross-domain alignment, and can be further processed by feature comparison, effectively improving the matching effect of cross-domain content retrieval and recommendation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to a method, device and electronic equipment for product mounting, retrieval, recommendation and training, and belongs to the field of computer technology. Background Art

[0002] With the development of Internet technology, multimedia content has become more and more popular in the e-commerce field. For example, related products can be linked through live broadcasts and short videos, which has a good publicity effect. From the user's perspective, rich multimedia content can also help users better understand the relevant knowledge and application scenarios of products, thus providing users with rich information for selection.

[0003] However, since multimedia data and product data belong to different content domains, they have different organizational forms. The display form of multimedia data is freer and less constrained, and can contain a variety of creative elements, such as complex visual backgrounds, complex natural text titles, and diversified performance themes. The organizational form of product data generally follows e-commerce constraints and is more structured. For example, the product background requires simplicity and standardization, and the product title is keyword stacked for easy retrieval. Therefore, there is a huge semantic gap between multimedia data and product data. Due to the existence of this semantic gap, it is difficult to match and merge multimedia data and product data in the existing technology, so as to provide users with better cross-domain content recommendations or product mounting. Summary of the invention

[0004] The embodiment of the present invention provides a method, device and electronic device for product mounting, retrieval, recommendation and training processing to realize feature comparison processing across content domains.

[0005] In order to achieve the above object, an embodiment of the present invention provides a method for training a feature extraction model, comprising:

[0006] Acquire samples associated with contents in multiple content domains to form multiple cross-domain sample combinations;

[0007] Using the trained feature extraction model, extract features from the original data of the samples in the cross-domain sample combination to generate feature vectors corresponding to the samples;

[0008] The feature extraction model is trained by contrastive learning to reduce the distance between feature vectors corresponding to samples within the cross-domain sample combination and to enlarge the distance between feature vectors corresponding to samples between the cross-domain sample combinations as training targets.

[0009] An embodiment of the present invention provides a method for processing commodity mounting, including:

[0010] Acquire multimedia data, perform cross-domain feature extraction on the multimedia data, and generate a multimedia feature vector corresponding to the multimedia data, wherein the cross-domain feature extraction enables the generated feature vector to be aligned across domains;

[0011] Acquire multiple candidate product data, perform cross-domain feature extraction on the candidate product data, and generate multiple product feature vectors;

[0012] The commodity data to be mounted is determined from the plurality of candidate commodity data according to the correlation between the plurality of commodity feature vectors and the multimedia feature vector.

[0013] An embodiment of the present invention provides a method for processing commodity mounting, including:

[0014] Acquire multimedia data, perform cross-domain feature extraction on the multimedia data, and generate a multimedia feature vector corresponding to the multimedia data, wherein the cross-domain feature extraction enables the generated feature vector to be aligned across domains;

[0015] Retrieving product data according to the multimedia feature vector, and obtaining product data corresponding to a product feature vector whose correlation with the multimedia feature vector is greater than a preset threshold;

[0016] The commodity data is mounted on the multimedia data.

[0017] The embodiment of the present invention provides a recommendation processing method, including:

[0018] Obtain historical multimedia data and / or historical product data of a user's historical visits;

[0019] Performing cross-domain feature extraction on the historical multimedia data and / or historical commodity data to generate corresponding historical multimedia feature vectors and / or historical commodity feature vectors, wherein the cross-domain feature extraction enables the generated feature vectors to be aligned across domains;

[0020] According to the historical multimedia feature vectors and / or historical product feature vectors, multimedia data and / or product data are retrieved, and multimedia data and / or product data whose correlation with the multimedia feature vectors and / or product feature vectors is greater than a preset threshold are obtained as recommendation data.

[0021] An embodiment of the present invention provides a retrieval processing method, including:

[0022] Generate a query vector based on the search information input by the user;

[0023] According to the query vector, a query is performed in a product feature vector database and / or a multimedia feature vector database to obtain a product feature vector and / or a multimedia feature vector having a first correlation with the query vector greater than a preset first threshold, wherein the product feature vector and / or the multimedia feature vector in the product feature vector database and / or the multimedia feature vector database is obtained based on cross-domain feature extraction, and the cross-domain feature extraction enables the generated feature vector to be aligned across domains;

[0024] According to the acquired product feature vector and / or multimedia feature vector, the corresponding product data and / or multimedia data are returned to the user as a search result.

[0025] An embodiment of the present invention provides a training device for a feature extraction model, comprising:

[0026] A sample acquisition module, used to acquire samples associated with contents in multiple content domains to form multiple cross-domain sample combinations;

[0027] A feature vector generation module, used to use the trained feature extraction model to extract features from the original data of the samples in the cross-domain sample combination, and generate a feature vector corresponding to the sample;

[0028] The training module is used to perform comparative learning training on the feature extraction model to reduce the distance between feature vectors corresponding to samples within the cross-domain sample combination and to enlarge the distance between feature vectors corresponding to samples between the cross-domain sample combinations as training targets.

[0029] An embodiment of the present invention provides a device for processing commodity mounting, including:

[0030] A multimedia feature extraction module, used to obtain multimedia data, perform cross-domain feature extraction on the multimedia data, and generate a multimedia feature vector corresponding to the multimedia data, wherein the cross-domain feature extraction enables the generated feature vector to be aligned across domains;

[0031] A product feature extraction module is used to obtain multiple candidate product data, perform cross-domain feature extraction on the candidate product data, and generate multiple product feature vectors;

[0032] The relevance processing module is used to determine the commodity data to be mounted from the plurality of candidate commodity data according to the relevance between the plurality of commodity feature vectors and the multimedia feature vector.

[0033] An embodiment of the present invention provides a processing device for associated commodity mounting, including:

[0034] A multimedia feature extraction module, used to obtain multimedia data, perform cross-domain feature extraction on the multimedia data, and generate a multimedia feature vector corresponding to the multimedia data, wherein the cross-domain feature extraction enables the generated feature vector to be aligned across domains;

[0035] A product retrieval processing module, used to retrieve product data according to the multimedia feature vector, and obtain product data corresponding to a product feature vector whose correlation with the multimedia feature vector is greater than a preset threshold;

[0036] The mounting processing module is used to mount the commodity data onto the multimedia data.

[0037] An embodiment of the present invention provides a recommendation processing device, including:

[0038] A historical data acquisition module is used to acquire historical multimedia data and / or historical commodity data that the user has visited in the past;

[0039] A historical data feature extraction module, used to perform cross-domain feature extraction on the historical multimedia data and / or historical commodity data to generate corresponding historical multimedia feature vectors and / or historical commodity feature vectors, wherein the cross-domain feature extraction enables the generated feature vectors to be aligned across domains;

[0040] A data retrieval processing module is used to retrieve multimedia data and / or commodity data based on the historical multimedia feature vectors and / or historical commodity feature vectors, and obtain multimedia data and / or commodity data whose correlation with the multimedia feature vectors and / or commodity feature vectors is greater than a preset threshold as recommendation data.

[0041] An embodiment of the present invention provides a retrieval processing device, including:

[0042] A query vector generation module, used to generate a query vector according to the search information input by the user;

[0043] a query vector retrieval module, configured to query a product feature vector database and / or a multimedia feature vector database according to the query vector, and obtain a product feature vector and / or a multimedia feature vector having a first correlation with the query vector greater than a preset first threshold, wherein the product feature vector and / or the multimedia feature vector in the product feature vector database and / or the multimedia feature vector database is obtained based on cross-domain feature extraction, and the cross-domain feature extraction enables the generated feature vector to be aligned across domains;

[0044] The search result feedback module is used to return corresponding product data and / or multimedia data to the user as the search result based on the acquired product feature vector and / or multimedia feature vector.

[0045] An embodiment of the present invention provides a method for processing commodity mounting, including:

[0046] Get multimedia data uploaded by users;

[0047] According to the content features obtained by performing cross-domain feature extraction on the multimedia data, selecting matching product data in a product database and recommending the matching product data to the user;

[0048] In response to the user's selection of the commodity data, the selected commodity data is mounted into the multimedia data.

[0049] An embodiment of the present invention provides a method for processing commodity mounting, including:

[0050] Obtain multimedia data uploaded by users and multiple commodity data to be mounted;

[0051] Obtaining a correlation between the commodity data and the multimedia data according to content features obtained by performing cross-domain feature extraction on the multimedia data and commodity features obtained by performing cross-domain feature extraction on the commodity data;

[0052] According to the relevance, commodity data is recommended from a plurality of commodity data to be mounted.

[0053] An embodiment of the present invention provides an electronic device, including:

[0054] Memory, used to store programs;

[0055] A processor is used to run the program stored in the memory to execute the aforementioned feature extraction model training method, and / or the aforementioned commodity mounting processing method, and / or the aforementioned recommendation processing method, and / or the aforementioned retrieval processing method.

[0056] The method, device and electronic device provided in the embodiments of the present invention train a cross-content feature extraction model through a comparative learning training method, thereby realizing feature extraction of data from different content domains. The extracted feature vector has the property of cross-domain alignment, so that further feature comparison and other processing can be performed, effectively improving the matching effect of cross-domain content retrieval and recommendation.

[0057] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1A processing system for training a feature extraction model according to an embodiment of the present invention;

[0059] Figure 2 A schematic diagram of an application scenario of a feature extraction model according to an embodiment of the present invention;

[0060] Figure 3 This is a flow chart of a method for processing commodity mounting according to an embodiment of the present invention;

[0061] Figure 4 A schematic diagram of an application scenario of a method for processing commodity mounting according to an embodiment of the present invention;

[0062] Figure 5 The second flowchart of the method for processing commodity mounting according to the embodiment of the present invention;

[0063] Figure 6 A flowchart of a processing method recommended by an embodiment of the present invention;

[0064] Figure 7 A schematic diagram of an application scenario of a recommended processing method according to an embodiment of the present invention;

[0065] Figure 8 A schematic diagram of a flow chart of a retrieval processing method according to an embodiment of the present invention;

[0066] Fig. 9 A schematic diagram of an application scenario of the retrieval processing method according to an embodiment of the present invention;

[0067] Fig.10 A flowchart of a method for training a feature extraction model according to an embodiment of the present invention;

[0068] Fig.11 This is one of the structural schematic diagrams of a commodity mounting processing device according to an embodiment of the present invention;

[0069] Fig.12 This is a second structural schematic diagram of a processing device for mounting goods according to an embodiment of the present invention;

[0070] Fig.13 A schematic diagram of the structure of a processing device recommended by an embodiment of the present invention;

[0071] Fig.14 A schematic diagram of the structure of a retrieval processing device according to an embodiment of the present invention;

[0072] Fig.15 A schematic diagram of the structure of a training device for a feature extraction model according to an embodiment of the present invention;

[0073] Fig.16 Schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0075] The technical solution of the present invention is further illustrated by some specific embodiments below.

[0076] In order to achieve matching and fusion between cross-domain data, an embodiment of the present invention provides a method for training a feature extraction model. The method trains the feature extraction model based on contrastive learning and adversarial learning techniques. The trained model can achieve cross-domain multimodal data matching and fusion, thereby aligning the semantic differences between multimedia data and commodity data, and realizing cross-domain data comparison and processing.

[0077] like Figure 1 As shown, it is a processing system for training a feature extraction model according to an embodiment of the present invention. The feature extraction model at the bottom is the trained model, and the training sample is input into the feature extraction model to generate a feature vector. The contrastive learning training module and the adversarial learning training module at the top perform training processing based on the training objective function, and the feature extraction model is updated according to the training feedback results. Through multiple rounds of training iteration processing, the feature vector output by the feature extraction model gradually approaches the training objectives of contrastive learning and adversarial learning.

[0078] In the embodiment of the present invention, multimedia domain and commodity domain are used as examples to illustrate how to solve the cross-domain multimodal fusion and matching problem. The trained feature extraction model is also used to perform feature extraction processing on the data in these two fields, and correspondingly, the training samples also come from the data in these two fields.

[0079] Regardless of the multimedia domain or the commodity domain, the data can be in a multimodal form. In this embodiment, the multimodal data includes image data and text data. In the multimedia domain, image data can include, for example, images of short videos uploaded by users, live broadcast images, photos in social media such as Weibo, etc., and text data can be subtitles in videos and text content on social media. In addition, in the embodiment of the invention, the audio data is converted into text and processed as text data. Similarly, in the commodity domain, image data can include, for example, videos and pictures introducing commodities, and text data can include titles of commodities, content introducing commodities, etc.

[0080] Although both the product domain and the multimedia domain can contain image data and text data, their organizational forms are quite different. The display form of multimedia data is freer and unconstrained, while the organizational form of product data generally follows the constraints of e-commerce and is more structured. For example, its short videos are taken from users' daily life, with rich visual elements and strong scene-based, while the introduction videos in the product domain generally have a simple background and need to highlight the product itself. The images are more intuitive and the key points are highlighted, and they must also comply with the regulations of the e-commerce platform, such as time length, background requirements, etc. In terms of text data, the text data in the multimedia domain generally conforms to the habits of natural language and will describe many users' feelings, etc., while the text data in the product domain will contain some structured non-natural language, such as product titles that stack many keywords in order to hit the search, and the text of the product description is also relatively simple and direct to describe the product itself. Due to the organizational forms of products in the two fields, there will be a large semantic gap. It is difficult to compare the semantic features of the two fields through simple feature extraction of images or texts, and it is also impossible to compare the semantics.

[0081] In the embodiment of the present invention, the purpose of training the feature extraction model is to bridge the semantic gap between different fields and align the extracted features. The input of the feature extraction model is the original data of the commodity domain or the multimedia domain. After being extracted by the feature extraction model, a feature vector corresponding to the original data is generated. The feature vector has the characteristic of cross-domain alignment, that is, no matter whether the original data comes from the commodity domain or the multimedia domain, the feature vector extracted by the model can be in the same vector space, and various correlation comparisons or vector operations can be performed. Among them, the original data can include two channels of data, image data and text data. After being processed by the feature extraction model, an image feature vector and a text feature vector are generated, and the feature vectors of these two modes also have alignment characteristics, and can perform cross-modal comparisons or vector operations, etc. As an optional implementation method, the image feature vector and the text feature vector can also be fused into a feature vector representing the original data.

[0082] For the content structure of the feature extraction model, the model structure for feature extraction and fusion processing of image data and text data is mainly adopted. For image data, for example, a visual feature extraction model can be used, such as a pre-trained ResNet (Residual neural network) model for feature extraction. Among them, a visual feature can be extracted for each image, and both multimedia data and product data may include multiple images, such as a video may include multiple frames of images, or a product may be displayed through multiple pictures. For such multiple visual features, max pooling processing can be performed on these visual features to obtain a unique feature representation of all image data for a certain multimedia data or product data, that is, fused into an image feature vector. On the other hand, for text data, similar to image data, a text feature extraction model can be used for feature extraction, such as using a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model, or an end-to-end machine learning model can be used to correspond each word in the text to a globally unique ID, and then perform an embedding operation on each ID. Finally, all word embeddings involved in the input text are normalized to obtain a unique feature representation, that is, a text feature vector. The normalization operation can be performed by, for example, max pooling processing or pooling processing based on the attention mechanism.

[0083] exist Figure 1 In the example shown, product data is represented by p, multimedia data is represented by s, and after feature extraction by the feature extraction model, the image feature vector corresponding to the product data is represented by v p , the text feature vector corresponding to the commodity data is represented as t p , the image feature vector corresponding to the multimedia data is represented as v s , the text feature vector corresponding to the multimedia data is represented as t s .

[0084] The model training process can be divided into two parts: contrastive learning and adversarial learning. In contrastive learning, two sample pairs are selected from the training sample set as contrastive learning groups. Each sample pair contains multimedia data and product data with strong content relevance, and the content relevance between the two sample pairs in the contrastive learning group is weak.

[0085] Each of the above sample pairs can be generated by the commodity data mounted by the existing multimedia data (the correlation between the commodity and the multimedia has been confirmed). For example, if a commodity is mounted in a short video, then the multimedia data corresponding to the short video and the commodity data corresponding to the commodity form a sample pair. The two sample pairs in the above comparative learning group can be obtained by randomly selecting from the training sample set. It should be noted that multiple commodities mounted on a video can form sample pairs with the video respectively, but since these sample pairs are related to the video, their contents have a strong correlation and cannot form a comparative learning group. Therefore, in the above process of forming a comparative learning group, the two selected sample pairs should exclude the situation of being mounted on the same video. Of course, the above sample pairs and comparative learning groups can be generated by manual or computer screening, as long as the basic principle of strong correlation of the content within the sample pairs and weak correlation of the content between the sample pairs is met.

[0086] The following is an introduction to the specific contrastive learning process. For any given positive sample pair consisting of a multimedia data sample and a commodity data sample that are related in content, {s + , p +}, randomly select another negative sample pair consisting of a multimedia data sample and a product data sample that are related in content from the entire content pool as the training set {s _ , p _}, forming a comparative learning group. It should be noted that, as a training strategy, the purpose of random selection is to ensure that the selected sample pairs have a high probability of weak correlation. The "+" and "-" here are mainly to indicate that the content correlation between the two sample pairs is weak, forming a comparative relationship.

[0087] For the selected comparative learning group, the feature extraction process is performed by using the trained feature extraction model to generate two sets of multimodal feature vectors. and in, They represent the image feature vector corresponding to the multimedia data extracted from the positive sample pair, the text feature vector corresponding to the multimedia data, the image feature vector corresponding to the commodity data, and the text feature vector corresponding to the commodity data. Similarly, They respectively represent the image feature vector corresponding to the multimedia data, the text feature vector corresponding to the multimedia data, the image feature vector corresponding to the product data, and the text feature vector corresponding to the product data extracted from the positive sample pair.

[0088] Based on the above feature vectors, the learning process of contrast training is carried out. The learning goal is to minimize the distance between multimodal vectors across domains in positive sample pairs, and maximize the distance between multimodal vectors across domains in negative sample pairs. The distance between vectors can be measured using the Euclidean distance, that is, d(x, y) = ||xy|| 2 , where x and y represent any two vectors and can be any eigenvectors in the above vector group.

[0089] For the visual modality, that is, for the image feature vectors in the above positive sample pairs and negative sample pairs, the following dual domain feature constraints can be used as the loss function in terms of image features.

[0090]

[0091] Among them, the function f h Defined as f h (x1, x2) = max(0, μ + x1 - x2), μ is the boundary distance, which is a positive number or 0. The specific value can be set according to actual needs. It is mainly used to adjust the expected difference between x1 and x2 so that the model can have a certain error tolerance. As shown in formula (1), it contains four f h The summation operation is performed at each f h The function contains the distance calculation of two sets of vectors, where the first set of vectors is the cross-domain vector distance within the sample pair, and the second set of vectors is the cross-domain vector distance between sample pairs. For example, The distance between the image feature vectors of the multimedia data and the product data of the positive sample pair is calculated, and The distance between the image feature vector of the multimedia data in the positive sample pair and the image feature vector of the commodity data in the negative sample pair is calculated. From the training objective, we hope to minimize The value of The value of , that is, to reduce the cross-domain vector distance within the sample pair and expand the cross-domain vector distance between sample pairs. For image features, its training goal is to minimize the loss function The ideal value is 0. Similarly, the following dual domain feature constraints can be determined as the loss function for text features:

[0092]

[0093] The definitions of the functions in formula (2) can refer to formula (1), except that the feature vectors are all replaced by text feature vectors. For text features, the training goal is to minimize the loss function The ideal value is 0.

[0094] The dual domain constraints of text features and image features are combined into the overall feature constraints for multimodal features. The overall loss function can be:

[0095]

[0096] The overall goal of contrastive learning is to adjust the model parameters in the feature extraction model through continuous iterative processing so that the feature vector generated by the feature extraction model can minimize the loss function of formula (3). The final training result is as follows: Figure 1 As shown in the contrastive learning training module, in forming the text feature vector space and the image feature vector space, the distance between the cross-domain feature vectors in the same sample pair is relatively small, while the distance between the cross-domain feature vectors in different sample pairs is relatively large.

[0097] The above introduces the training process of contrastive learning, and the following introduces the training process of adversarial learning. In an embodiment of the present invention, adversarial learning is used as a supplement to contrastive learning training, with the purpose of better overcoming the cross-domain multimodal semantic gap. In the process of adversarial learning, a discriminator is set to identify the content domain to which the feature vector belongs, and the training goal of the adversarial learning training module for the feature extraction model is to hope that the discriminator cannot accurately identify the content domain to which the feature vector belongs. In other words, the training of the adversarial learning part actually adds a certain difficulty to contrastive learning, so as to improve the training effect of contrastive learning, so that the feature vectors extracted from different content domains can be better aligned.

[0098] Specifically, the adversarial learning training module can be composed of two discriminators D v and D t , where D v It is used to determine whether the input image feature vector comes from the multimedia domain or the commodity domain. Similarly, D t It is used to determine whether the input text feature vector comes from the multimedia domain or the product domain. Figure 1 In the example of , the training goal of the adversarial learning training module is to make the discriminator unable to accurately distinguish whether the feature vector comes from the multimedia domain or the product domain. v and D t A model structure combining MLP (Multi-Layer Perceptron) and GRL (gradient reversal layer) can be used.

[0099] exist Figure 1 In the example shown, z is used to represent the sample label value, which takes the value of 0 or 1, where 0 represents that it is from the multimedia domain and 1 represents that it is from the product domain. The discriminant value represents the actual output of the discriminator, and its value range is between 0 and 1, which is equivalent to giving the probability that the discriminated feature vector belongs to the multimedia domain or the product domain.

[0100] The loss function of adversarial learning can be defined as follows:

[0101]

[0102] The model parameters in the feature extraction model are denoted as θ E , the model parameters in the discriminator are recorded as θ C , based on contrastive learning and integrated with adversarial learning, the training process can be expressed as the following min-max game:

[0103]

[0104] in, and represents the model parameters obtained through training, where the right side of the equation and represents the model parameters fixed by the previous round of training, and the left side of the equation and Indicates the model parameters obtained in this round of training. d (θ E )and represents the loss function when the feature extraction model uses the model parameters in brackets, that is, when the model parameters in brackets are used, the feature vector is extracted and substituted into formula (3) for calculation. Similarly, L adv (θC) and Represents the loss function when the discriminator uses the model parameters in brackets, that is, when the model parameters in brackets are adopted, determine and z values, and substitute them into formula (4) for calculation.

[0105] In the training process shown in formula (5), the formula in the first row is used to train the model parameters of the feature extraction model. The training goal is to make The value is the smallest, where the right side of the equation The value does not change in this round of training and is considered a fixed value. During the training process, the model parameters of the feature extraction model are adjusted to achieve the training goal. does not change during the training process, as long as L d (θ E ) is smaller, Therefore, the training process of this part is mainly based on contrastive learning, that is, to improve the cross-domain feature extraction capability of the feature extraction model and realize the alignment of cross-domain feature vectors, so that the discriminator cannot accurately identify the field corresponding to the feature vector.

[0106] The second line of formula (5) is used to train the model parameters of the discriminator. The training goal is to make The value is the largest, where the right side of the equation The value does not change in this round of training and is considered a fixed value. During the training process, the model parameters of the discriminator are adjusted to achieve the training goal. There is no change during the training process, as long as The smaller, Therefore, the training process of this part is mainly aimed at the training of the discriminator, that is, to improve the discriminator's ability to identify which field the feature vector comes from, thereby increasing the difficulty of contrastive learning.

[0107] In the model training process based on formula (5), the training processes corresponding to the first and second rows are performed alternately, and the model parameters of the feature extraction model and the discriminator are continuously changed, thereby realizing the fusion of contrastive learning and adversarial learning.

[0108] The feature extraction model trained based on the aforementioned contrastive learning or contrastive learning combined with adversarial learning can be applied to product mounting in multimedia, user product recommendation, and product retrieval. Figure 2As shown, it is a schematic diagram of the application scenario of the feature extraction model of the embodiment of the present invention. The feature extraction model of the embodiment of the present invention can be applied to a platform with content providing services, and the e-commerce platform is used as an example in the figure for illustration. On the e-commerce platform, a database storing massive commodity data and multimedia data, a retrieval / recommendation engine for providing retrieval recommendation services, a module for live broadcast processing for providing live broadcast services, and a user information interaction processing module for interacting with the user terminal are configured. In the embodiment of the present invention, the feature extraction model can be configured in the retrieval / recommendation engine, so that in the process of retrieving and recommending commodity data or multimedia data, feature extraction is performed to achieve cross-domain feature processing. In addition, the e-commerce platform can also pre-process the database storing commodity data and multimedia data in advance to form a vector database corresponding to the commodity data and multimedia data, so as to facilitate the subsequent comparison processing in the retrieval and recommendation process. In addition, in the live broadcast processing, the live video can also be feature extracted based on the feature extraction model provided in the embodiment of the present invention, and the matching commodity data obtained based on the retrieval engine is mounted. In addition, the feature vector extracted by the feature extraction model can also be used to examine whether the commodity data mounted by the user matches the live video. The following is a detailed introduction to the application of the above feature extraction model in live broadcast services and retrieval recommendation services.

[0109] like Figure 3 As shown, it is one of the flow charts of the method for processing commodity mounting according to an embodiment of the present invention, such as Figure 4 As shown, it is a schematic diagram of an application scenario of a method for processing commodity mounting according to an embodiment of the present invention. The method can be applied to e-commerce platforms, social platforms with multimedia publishing functions, and live broadcast platforms, etc., and is used to mount commodity data with certain relevance into multimedia data. Specifically, the method may include:

[0110] S101: Acquire multimedia data, perform cross-domain feature extraction on the multimedia data, and generate a multimedia feature vector corresponding to the multimedia data. Among them, cross-domain feature extraction enables the generated feature vector to be aligned across domains. The multimedia data in the embodiment of the present invention may include videos published by video websites, short videos published by social platforms, live videos of live broadcast platforms, etc. The application scenario of this embodiment can be to find suitable mounted products for the video when the video is released, or to match suitable mounted products for the already released video. The above-mentioned cross-domain feature extraction can be implemented using a feature extraction model obtained based on the training method introduced above. In this embodiment, cross-domain mainly refers to cross-multimedia domain and commodity domain. As introduced above, the content organization forms of these two fields are quite different. Through the cross-domain feature extraction processing provided by the embodiment of the present invention, the feature vectors extracted for the two content domains can be in the same vector space, and can be compared or fused based on semantic relevance, that is, it has the characteristics of cross-domain alignment.

[0111] S102: Acquire multiple candidate product data, perform cross-domain feature extraction on the candidate product data, and generate multiple product feature vectors. The candidate product data can be determined by the user's selection operation in the product database. For example, after the user shoots a short video, it is uploaded to a social platform or an e-commerce platform, and some products that he wants to mount are selected. In this case, the product selected by the user may not necessarily match the current video. The user may just randomly mount the product to increase the click-through rate. Therefore, the platform side needs to conduct a certain review to avoid a large number of mounted products that do not match the video. In addition, the candidate product data can also come from recommendations from some merchants or recommendations based on platform advertising strategies, and then further screened using the method of this embodiment. The cross-domain feature extraction mentioned in this step can also be implemented using a feature extraction model obtained based on the training method introduced above. The extracted product feature vector can be aligned with the multimedia feature vector extracted in the previous step, and the correlation calculation between vectors can be performed.

[0112] S103: Determine the product data to be mounted from the plurality of candidate product data according to the correlation between the plurality of product feature vectors and the multimedia feature vectors. After the product feature vectors and the multimedia feature vectors are extracted, the correlation can be determined by calculating the distance between the vectors. The correlation between the vectors also represents the correlation between the candidate product data and the multimedia data, so that the product data with the highest correlation ranking can be selected from the candidate product data for mounting.

[0113] Combination Figure 4The example interface of , on the user terminal, may include an interactive interface located at the top for users to upload videos or broadcast live, and a functional area located at the bottom for selecting mounted products. Users can select from the product items below as candidate products and add them to the video screen. For example, they can be added to the lower left area of ​​the video screen. When the audience watches the video or live broadcast, they can access the product data by clicking on the product thumbnail presented in the lower left area. The function control area in the upper interactive interface can be used to adjust the video screen and the thumbnail display position of the mounted products. The product items in the mounted product selection area can come from the user's own selection, or from some recommendations from platform merchants, etc. After the user selects certain products as candidate product items, they need to be processed by the above method before they can be mounted on the video screen, that is, they need to be screened from these candidate products according to relevance before they can be mounted.

[0114] The above-mentioned commodity feature vector and / or multimedia feature vector may include an image feature vector extracted based on image data and / or a text feature vector extracted based on text data. Specifically, the above-mentioned candidate commodity data and multimedia data may further include data of two modalities, image data and text data, wherein image data is such as pictures, videos, etc., and text data may include text content, titles, subtitles, etc. In addition, audio data may be pre-converted into text data. The data of the above two modalities may be processed for cross-domain feature extraction respectively to generate image feature vectors and text feature vectors. The image feature vectors and text feature vectors corresponding to the commodity data and multimedia data may be respectively calculated for the correlation of feature vectors, or may be cross-correlated. As described above, the feature extraction model of an embodiment of the present invention may be based on a pre-trained feature extraction model with cross-modal feature extraction capability. Therefore, the extracted image feature vector and the text data may also be calculated for correlation, thereby realizing cross-domain feature comparison based on cross-modality.

[0115] The product mounting processing method of the embodiment of the present invention generates a product feature vector corresponding to the product domain and a multimedia feature vector corresponding to the multimedia domain based on cross-domain feature extraction processing, and performs cross-domain feature comparison to screen candidate product data, and mounts product data that has a better match with multimedia data on the media data, so that users can obtain more valuable product information when browsing multimedia content such as videos or live broadcasts.

[0116] like Figure 5As shown, it is a second flow chart of a method for processing product mounting according to an embodiment of the present invention. The method can be applied to e-commerce platforms, social platforms with multimedia publishing functions, and live broadcast platforms, etc., and is used to mount product data with certain relevance into multimedia data. Specifically, the method may include:

[0117] S201: Acquire multimedia data, perform cross-domain feature extraction on the multimedia data, and generate a multimedia feature vector corresponding to the multimedia data, wherein the cross-domain feature extraction enables the generated feature vector to be aligned across domains. The processing of this step is consistent with step S101 in the previous embodiment.

[0118] S202: Retrieve product data based on the multimedia feature vector, and obtain product data corresponding to the product feature vector whose correlation with the multimedia feature vector is greater than a preset threshold. This embodiment can be applied to scenarios where e-commerce platforms or social platforms select products for mounting for videos uploaded by users or live broadcasts in progress. The platform side can pre-build a product feature vector database, that is, use the feature extraction model provided by the embodiment of the present invention to extract product feature vectors with cross-domain alignment properties from the existing product data in the e-commerce platform in advance, and form a product feature vector database. Accordingly, this step can specifically include: searching in the product feature vector database based on the multimedia feature vector, obtaining the product feature vector whose correlation with the multimedia feature vector is greater than a preset threshold, and then obtaining the corresponding product data based on the product feature vector.

[0119] S203: Mounting the product data on the multimedia data. The retrieved product data can be directly mounted on the multimedia data. In addition, the retrieved product data can also be first recommended to the user as candidate product data, and then, according to the user's selection of the candidate product data, the product data mounting process is performed, thereby giving the user a certain degree of autonomy and selectivity.

[0120] Similar to the above-mentioned embodiment, the above-mentioned commodity feature vector and / or multimedia feature vector may include an image feature vector extracted based on image data and / or a text feature vector extracted based on text data. The image feature vector and text feature vector corresponding to the commodity data and multimedia data may be respectively subjected to feature vector correlation calculation, or may be cross-correlation calculation.

[0121] The method for processing product mounting in an embodiment of the present invention generates a product feature vector corresponding to the product domain and a multimedia feature vector corresponding to the multimedia domain based on cross-domain feature extraction processing, and performs cross-domain feature vector-based retrieval to obtain product data that has a good match with the multimedia data and mount it on the media data, so that users can obtain more valuable product information when browsing multimedia content such as videos or live broadcasts.

[0122] like Figure 6 As shown, it is a flowchart of a processing method recommended by an embodiment of the present invention, such as Figure 7 As shown, it is a schematic diagram of an application scenario of the recommendation processing method of an embodiment of the present invention. The method can be applied to e-commerce platforms, social platforms with multimedia publishing functions, and live broadcast platforms, etc., for actively recommending content to users. Specifically, the method may include:

[0123] S301: Obtain historical multimedia data and / or historical product data that the user has visited in the past. In the application scenario of this embodiment, active recommendations can be made based on the user's historical access behavior. For example, when a user opens a page of an e-commerce APP, some relevant content recommendations can be displayed to the user on the homepage without entering any keywords. The above-mentioned historical multimedia data and / or historical product data can be recorded through the user's APP, with the user's explicit permission, and reported to the e-commerce platform to provide users with a better content recommendation experience.

[0124] S302: Perform cross-domain feature extraction on historical multimedia data and / or historical commodity data to generate corresponding historical multimedia feature vectors and / or historical commodity feature vectors, wherein the cross-domain feature extraction enables the generated feature vectors to be aligned across domains. In this step, the cross-domain feature extraction process is the same as the feature extraction process on multimedia data and commodity data mentioned in the previous embodiment, and can be implemented using a feature extraction model obtained based on the training method introduced above. Furthermore, the above-mentioned historical multimedia data and / or historical commodity data may include image data and / or text data, and accordingly, the above-mentioned commodity feature vectors and / or multimedia feature vectors may include image feature vectors extracted based on image data and / or text feature vectors extracted based on text data.

[0125] S303: Retrieve multimedia data and / or commodity data based on historical multimedia feature vectors and / or historical commodity feature vectors, and obtain multimedia data and / or commodity data whose correlation with the multimedia feature vector and / or commodity feature vector is greater than a preset threshold as recommended data. After extracting the commodity feature vector and multimedia feature vector, the correlation can be determined by calculating the distance between the vectors. The correlation between the vectors also represents the correlation between the commodity data and the multimedia data, so that the commodity data and multimedia data with high correlation with the user's historical behavior can be retrieved, and reasonable content recommendations can be made for the user.

[0126] like Figure 7As shown in FIG. 1 , it shows a recommendation interface of an online shopping APP on the user terminal side. The user can generate various recommendation data on the recommendation page through the above recommendation processing method without entering any information. Figure 7 On the application interface shown, some product items are recommended to users, as well as short videos uploaded by other users and live broadcasts being performed by merchants. Related product items are mounted inside the short videos and live broadcasts as further recommendations.

[0127] In the above-mentioned process of retrieving multimedia data and / or product data, the search can be performed in a pre-generated database storing multimedia feature vectors and / or a database of product feature vectors. The platform can pre-extract feature vectors from the multimedia data and / or product data it stores using the feature extraction model provided by the embodiment of the present invention to form a feature vector database. On this basis, the database can be searched based on the user's historical multimedia feature vectors and / or historical product feature vectors to obtain multimedia feature vectors and / or product feature vectors whose correlation with the historical multimedia feature vectors and / or historical product feature vectors is greater than a preset threshold; then, based on the obtained multimedia feature vectors and / or product feature vectors, the corresponding multimedia data and / or product data are determined as recommended data.

[0128] In the user's historical access behavior, there is generally a large amount of historical multimedia data and / or historical product data. Accordingly, there are also multiple historical multimedia feature vectors and / or historical product feature vectors extracted from the historical multimedia data and / or historical product data. In an embodiment of the present invention, retrieval can be performed based on each multimedia feature vector and / or historical product feature vector, and the multimedia data and / or product data with a large number of hits can be used as recommended data. In addition, multiple historical multimedia feature vectors and / or historical product feature vectors can also be fused to generate a fused feature vector, which is equivalent to a semantic abstraction of the user's overall historical behavior. Then, multimedia data and / or product data are retrieved based on the fused feature vector to obtain multimedia data and / or product data whose correlation with the fused feature vector is greater than a preset threshold.

[0129] The recommendation processing method of an embodiment of the present invention performs cross-domain feature extraction on historical multimedia data and / or historical product data involved in the user's historical access behavior based on cross-domain feature extraction processing, and retrieves multimedia data and / or product data with better matching based on the extracted feature vectors and recommends them to the user, thereby allowing the user to obtain a better content recommendation experience.

[0130] like Figure 8As shown, it is a flowchart of the retrieval processing method according to an embodiment of the present invention. Fig. 9 As shown, it is a schematic diagram of an application scenario of the search processing method of an embodiment of the present invention. The method can be applied to an e-commerce platform to perform content search based on keywords input by a user. Specifically, the method may include:

[0131] S401: Generate a query vector according to the search information input by the user. The application scenario of this embodiment can be a search initiated by a user inputting search information through a search page of an e-commerce platform. Among them, the feature extraction of the search information can be performed using the feature extraction model provided by the above embodiment of the present invention.

[0132] S402: According to the query vector, query is performed in the product feature vector database and / or the multimedia feature vector database to obtain the product feature vector and / or the multimedia feature vector whose first correlation with the query vector is greater than the preset first threshold. Among them, the product feature vector and / or the multimedia feature vector in the product feature vector database and / or the multimedia feature vector database is obtained based on cross-domain feature extraction, which can be specifically performed using the feature extraction model provided in the aforementioned embodiment of the present invention. The generated feature vector can be aligned across domains through cross-domain feature extraction. As previously introduced, the e-commerce platform can extract feature vectors from multimedia data and product data in advance using the feature extraction model provided in the embodiment of the present invention to form a feature vector database. On this basis, after the user's input search information is converted into a query vector through step S401, it can be compared with the vector in the vector database.

[0133] S403: According to the obtained product feature vector and / or multimedia feature vector, the corresponding product data and / or multimedia data are returned to the user as the search result. The search results returned to the user may include multimedia data such as short videos and live broadcasts, and may also include product data published on the e-commerce platform. In addition, the multimedia data may further include product data, such as product links associated with the short video.

[0134] like Fig. 9 As shown, it shows the page where the user searches on the online shopping APP of the terminal. The user enters keywords in the search box on the left interface. After the APP interacts with the server of the e-commerce platform, it returns to the page on the right. The returned search results include product items and short videos or merchant live broadcasts that are attached with product items.

[0135] For the product data mounted in the multimedia data, the e-commerce platform can determine the correlation between the mounted product data and the multimedia data when returning the search results to the user, and eliminate some irrelevant products. Specifically, the above method can also include: obtaining a product feature vector corresponding to the product data mounted in the multimedia data; obtaining a second correlation between the multimedia feature vector corresponding to the multimedia data and the product feature vector, if the second correlation is greater than or equal to a preset second threshold, retaining the mounted product data in the search results returned to the user, otherwise, deleting the mounted product data.

[0136] In addition, for multimedia data that has no product data or less product data in the search results, relevant product data can be further obtained for mounting and recommended to the user. Specifically, when returning multimedia data to the user, it can also include: querying in a product feature vector database according to the multimedia feature vector corresponding to the multimedia data, obtaining a product feature vector whose correlation with the multimedia feature vector is greater than a preset third threshold; obtaining corresponding product data according to the product feature vector, and mounting it in the multimedia data as a search result for the user.

[0137] The retrieval processing method of an embodiment of the present invention, based on a feature vector database formed by cross-domain feature extraction processing, can perform cross-domain feature vector matching queries based on feature vectors formed by retrieval information input by a user, and obtain multimedia data and / or product data that meets the user's retrieval requirements, thereby being able to provide users with rich retrieval results from different content domains.

[0138] In the above embodiment, commodity data and multimedia data are used as two content domains as examples to introduce the application of the feature extraction model and the model training principle of the embodiment of the present invention. In fact, the feature extraction model and model training method of the embodiment of the present invention can realize feature extraction of data in multiple content domains.

[0139] like Fig.10 As shown, it is a flow chart of the training method of the feature extraction model of the embodiment of the present invention. The training method of the embodiment of the present invention can be applied to e-commerce platforms, search engines, and data service platforms that provide cloud services, and is used to train feature extraction models in processing systems such as search engines and content recommendations, so as to realize feature extraction of data in different content domains, and enable the extracted feature vectors to be aligned, so as to facilitate subsequent feature comparison or fusion between various content domains. Further processing, the method may include:

[0140] S501: Obtain content-related samples from multiple content domains to form multiple cross-domain sample combinations. In order to train the feature extraction model, a sample set can be pre-built. Among them, the sample can come from data in any content domain, such as a video from the multimedia domain or a product data from the product domain. If there is a known correlation between these samples from different content domains, then these samples can be combined into a cross-domain sample combination, and the user performs model training. In the previous example, the sample pair composed of product data mounted in multimedia data is used as an example of a cross-domain sample combination. In actual applications, since there may be multiple content domains, the number of samples in the cross-domain sample combination may also be multiple.

[0141] S502: Use the trained feature extraction model to extract features from the original data of the samples in the cross-domain sample combination, and generate feature vectors corresponding to the samples. Wherein, the sample data of any content domain may include data of multiple modalities, where the modality refers to the form in which the data itself exists, and the modality generally may include: image, audio, text, etc. Therefore, the feature extraction process in this step may include: extracting features from the original data of each modality of the samples in the cross-domain sample combination, and generating feature vectors of multiple modalities corresponding to the samples. In an embodiment of the present invention, image data and text data may be selected as the main modalities, and audio data may be converted into text data in advance, that is, the original data of the above-mentioned samples may include image data and / or text data, and accordingly, the feature extraction process in this step may include: extracting features from the image data and / or text data of the samples in the cross-domain sample combination, and generating image feature vectors and / or text feature vectors corresponding to the samples.

[0142] It should be noted here that the contrastive learning and the subsequent adversarial learning in the embodiments of the present invention are mainly training for cross-domain feature extraction capabilities. When there is a need for cross-modal feature extraction, the above-mentioned trained feature extraction model can adopt a pre-trained feature extraction model with cross-modal feature extraction capabilities. For example, in model selection, the pre-trained BERT model can be used for the text feature extraction part, or an end-to-end machine learning model can be adopted. For the image feature extraction part, a pre-trained ResNet model can be used.

[0143] S503: Perform contrastive learning training on the feature extraction model to reduce the distance between the feature vectors corresponding to the samples within the cross-domain sample combination and to enlarge the distance between the feature vectors corresponding to the samples between the cross-domain sample combinations as the training target. In the previous text, commodity data and multimedia data have been used as examples to illustrate the corresponding training process and loss function calculation. For the case of more than two content domains, that is, the case where there are multiple samples within the sample combination, the training and loss function can also be constructed in the same way as in the above example. In the loss function, the distance calculation can still use the method of calculating the distance between two vectors, except that the number of distance calculations required will be more than the case of two content domains, but the overall principle is the same, that is, it is hoped that the distance between the feature vectors of different domains within the same sample combination is as small as possible, and the distance between the feature vectors of different domains of different sample combinations is as large as possible. The training process of contrastive learning can refer to formulas (2) and (3) mentioned in the above embodiment for training.

[0144] In addition, in order to improve the training effect of contrastive learning, the embodiment of the present invention can introduce adversarial learning for auxiliary training. Therefore, the above method can also include: performing adversarial learning training on the feature extraction model to reduce the accuracy of the discriminator of adversarial learning for the content domain to which the feature vector belongs as a training goal. In this embodiment, the role of adversarial learning is to increase the difficulty of contrastive learning and serve as a discriminator to test the effect of contrastive learning. That is, when the discriminator of adversarial learning cannot accurately determine which content domain the feature vector extracted by the feature extraction model comes from, it is equivalent to that contrastive learning has well aligned the feature vectors of different content domains, achieving a better cross-domain feature extraction effect.

[0145] Specifically, during the training process of adversarial learning, the following training processes may be performed alternately:

[0146] 1) Fix the model parameters of the discriminator and train the feature extraction model to reduce the discriminator's accuracy in distinguishing the content domain to which the feature vector belongs. The training process of this part is to train the feature extraction model while fixing the model parameters of the discriminator and combining the loss function of the aforementioned contrastive learning to improve the cross-domain alignment of the feature vectors extracted by the feature extraction model.

[0147] 2) The model parameters of the feature extraction model are fixed, and the discriminator is trained to improve the accuracy of the discriminator in distinguishing the content domain to which the feature vector belongs. In this part of the training process, the discriminator's discrimination ability is improved while the model parameters of the feature extraction model are fixed, thereby increasing the difficulty of the training process in 1) above.

[0148] The above-mentioned adversarial learning training process can be trained using the functions of formula (4) and formula (5) in the above-mentioned embodiment.

[0149] The method, device and electronic device provided in the embodiments of the present invention train a cross-content feature extraction model through a comparative learning training method, thereby realizing feature extraction of data from different content domains. The extracted feature vector has the property of cross-domain alignment, so that further feature comparison and other processing can be performed, effectively improving the matching effect of cross-domain content retrieval and recommendation.

[0150] The model training method provided in the embodiment of the present invention realizes feature extraction of data from different content domains through contrastive learning or contrastive learning combined with adversarial learning. The extracted feature vectors have the property of cross-domain alignment, so that further feature comparison and other processing can be performed, effectively improving the matching effect of cross-domain content retrieval and recommendation.

[0151] The embodiment of the present invention further provides a method for processing product mounting, which can be applied to an e-commerce platform or a social platform, and can be specifically applied to a software product that provides multimedia data publishing for users who are general consumers, to help users recommend and select product mounting. Specifically, the method includes:

[0152] S601: Acquire multimedia data uploaded by the user. In this embodiment, the user may be a general consumer, and the multimedia data uploaded by the user may be a self-made video, such as life dynamics or product usage content. With the user's permission, some related products may be recommended to the user and mounted in the video as an advertisement recommendation, and the user may also obtain a certain amount of compensation from it.

[0153] S602: According to the content features obtained by performing cross-domain feature extraction on the multimedia data, select matching product data in the product database and recommend them to the user. The above-mentioned extraction process of the content features of the multimedia data may use the feature extraction model introduced in the above-mentioned embodiment.

[0154] S603: In response to the user's selection of commodity data, the selected commodity data is mounted in the multimedia data. The platform can recommend multiple commodity data to the user, and the user can finally decide the commodity data to be mounted in the multimedia data according to his own preferences or ideas.

[0155] In addition, after the multimedia data is released, a large amount of comment data may be received. In the embodiment of the present invention, these comment data can also be obtained, and according to the characteristics of the comment data, product data matching the comment data is selected in the product database and recommended to the user, so as to update the product data mounted in the multimedia data. This process can be performed regularly as the comment data changes, thereby helping users to continuously update the product data mounted in the multimedia data.

[0156] For example, after a user uploads a video of his travel life, the initial products that are posted are some travel clothing and other products. Due to the good quality of the video, it has received a lot of attention and involved a large number of comments about photography. The platform can match some photography equipment or books based on the new comment data, recommend them to users, and update the products, thereby helping users and merchants achieve a win-win effect.

[0157] In addition, the above-mentioned review data may also include some review data of products. As the product review data changes, it may affect the matching degree with the multimedia data, so that the product data recommended to the user may also be updated according to the product review data.

[0158] The embodiment of the present invention further provides a method for processing product mounting, which can be applied to an e-commerce platform or a social platform, and can be specifically applied to a software product that provides multimedia data publishing for merchant users, for example, to help merchant users select or check product mounting. Specifically, the method includes:

[0159] S701: Acquire multimedia data uploaded by a user and multiple commodity data to be mounted. For merchant users, the uploading of multimedia data has a strong purpose, that is, to promote their own commodities. However, merchant users may be deficient in the matching judgment between commodity data and multimedia data. The method of the embodiment of the present invention can help users to make matching judgments. Therefore, merchant users upload commodity data to be mounted while uploading multimedia data.

[0160] S702: Obtain the relevance between the product data and the multimedia data based on the content features obtained by performing cross-domain feature extraction on the multimedia data and the product features obtained by performing cross-domain feature extraction on the product data. The above-mentioned extraction process of various features may use the feature extraction model introduced in the above-mentioned embodiment to form cross-domain feature content capable of calculating relevance.

[0161] S703: Recommending commodity data from a plurality of commodity data to be mounted according to the relevance. Specifically, the relevance of the plurality of commodity data to the multimedia data may be ranked, and the mountable commodity data may be recommended to the user based on the ranking. For example, a certain number of commodity data ranked high may be selected as the mountable commodity data.

[0162] In addition, after the multimedia data is released, a large amount of comment data may be received. In the embodiment of the present invention, these comment data can also be obtained, and the product data recommended to the user can be updated based on the features obtained by cross-domain feature extraction of the comment data. For example, the user can be advised whether to replace some mounted product data or re-produce multimedia data. In addition, the above comment data can also include some comment data of the product. As the product comment data changes, it can affect the matching degree with the multimedia data, so the product data recommendation to the user can also be updated based on the product comment data.

[0163] In the process of recommending product data, the correlation between multimedia data features and product features is mainly considered. In addition, the user's geographical location information can also be introduced to recommend product data. For example, for some agricultural products, it is often the case that merchant users or ordinary consumer users are concentrated in one area. In this case, based on the characteristics of the region, some product data related to the region can be recommended to be mounted in the multimedia data, thereby promoting the sales of products in the region.

[0164] like Fig.11 As shown, it is one of the structural schematic diagrams of the processing device for mounting goods according to an embodiment of the present invention. The device can be applied to e-commerce platforms, social platforms with multimedia publishing functions, and live broadcast platforms, etc., and is used to mount commodity data with certain relevance into multimedia data. Specifically, the device may include:

[0165] The multimedia feature extraction module 11 is used to obtain multimedia data, perform cross-domain feature extraction on the multimedia data, and generate a multimedia feature vector corresponding to the multimedia data, wherein the cross-domain feature extraction enables the generated feature vector to be aligned across domains. The multimedia data in the embodiment of the present invention may include videos published by video websites, short videos published by social platforms, live videos of live broadcast platforms, etc. The application scenario of this embodiment can be to find suitable mounted products for the video when the video is published, or to match suitable mounted products for the published video. The above-mentioned cross-domain feature extraction can be implemented using a feature extraction model obtained based on the training method introduced above. In this embodiment, cross-domain mainly refers to cross-multimedia domain and commodity domain. As introduced above, the content organization forms of these two fields are quite different. Through the cross-domain feature extraction processing provided by the embodiment of the present invention, the feature vectors extracted for the two content domains can be in the same vector space, and can be compared or fused based on semantic relevance, that is, it has the characteristics of cross-domain alignment.

[0166] The product feature extraction module 12 is used to obtain multiple candidate product data, perform cross-domain feature extraction on the candidate product data, and generate multiple product feature vectors. The candidate product data can be determined by the user's selection operation in the product database. In addition, the candidate product data can also come from the recommendations of some merchants or recommendations based on platform advertising strategies. Then the device of this embodiment is used to further screen the candidate products. The cross-domain feature extraction mentioned in the above process can also be implemented using a feature extraction model obtained based on the training method introduced above. The extracted product feature vector can be aligned with the multimedia feature vector, and the correlation calculation between vectors and other processing can be performed.

[0167] The correlation processing module 13 is used to determine the commodity data to be mounted from the plurality of candidate commodity data according to the correlation between the plurality of commodity feature vectors and the multimedia feature vectors. After the commodity feature vectors and the multimedia feature vectors are extracted, the correlation can be determined by calculating the distance between the vectors. The correlation between the vectors also represents the correlation between the candidate commodity data and the multimedia data, so that the commodity data with the highest correlation ranking can be selected from the candidate commodity data for mounting.

[0168] The above-mentioned commodity feature vectors and / or multimedia feature vectors may include image feature vectors extracted based on image data and / or text feature vectors extracted based on text data. The image feature vectors and text feature vectors corresponding to the commodity data and multimedia data may be respectively subjected to feature vector correlation calculations, or cross-correlation calculations may be performed. As previously described, the feature extraction model of an embodiment of the present invention may be based on a pre-trained feature extraction model with cross-modal feature extraction capabilities. Therefore, correlation calculations may also be performed between the extracted image feature vectors and the text-based data, thereby achieving cross-domain feature comparison based on cross-modality.

[0169] The product mounting processing device of the embodiment of the present invention generates a product feature vector corresponding to the product domain and a multimedia feature vector corresponding to the multimedia domain based on cross-domain feature extraction processing, and performs cross-domain feature comparison to screen candidate product data, and mounts product data that has a better match with the multimedia data on the media data, so that users can obtain more valuable product information when browsing multimedia content such as videos or live broadcasts.

[0170] like Fig.12 As shown, it is a second structural schematic diagram of a processing device for mounting goods according to an embodiment of the present invention. The device can be applied to e-commerce platforms, social platforms with multimedia publishing functions, and live broadcast platforms, etc., and is used to mount commodity data with certain relevance into multimedia data. Specifically, the device may include:

[0171] The multimedia feature extraction module 21 is used to obtain multimedia data, perform cross-domain feature extraction on the multimedia data, and generate a multimedia feature vector corresponding to the multimedia data, wherein the cross-domain feature extraction enables the generated feature vector to be aligned across domains. The processing of this step is consistent with the multimedia feature extraction module 11 in the previous embodiment.

[0172] The product retrieval processing module 22 is used to retrieve product data based on the multimedia feature vector, and obtain product data corresponding to the product feature vector whose correlation with the multimedia feature vector is greater than a preset threshold. This embodiment can be applied to scenarios where e-commerce platforms or social platforms select products for mounting for videos uploaded by users or live broadcasts in progress. The platform side can pre-build a product feature vector database, that is, the existing product data in the e-commerce platform is used using the feature extraction model provided by the embodiment of the present invention to pre-extract product feature vectors with cross-domain alignment properties, and form a product feature vector database. Accordingly, the processing of this module can specifically include: searching in the product feature vector database according to the multimedia feature vector, obtaining the product feature vector whose correlation with the multimedia feature vector is greater than the preset threshold, and then obtaining the corresponding product data according to the product feature vector.

[0173] The mounting processing module 23 is used to mount the commodity data on the multimedia data. In addition, the retrieved commodity data can be recommended to the user as candidate commodity data first, and then, according to the user's selection of the candidate commodity data, the mounting processing of the commodity data is performed, thereby giving the user a certain degree of autonomy and selectivity.

[0174] Similar to the above-mentioned embodiment, the above-mentioned commodity feature vector and / or multimedia feature vector may include an image feature vector extracted based on image data and / or a text feature vector extracted based on text data. The image feature vector and text feature vector corresponding to the commodity data and multimedia data may be respectively subjected to feature vector correlation calculation, or may be cross-correlation calculation.

[0175] The commodity mounting processing device of the embodiment of the present invention generates a commodity feature vector corresponding to the commodity domain and a multimedia feature vector corresponding to the multimedia domain based on cross-domain feature extraction processing, and performs cross-domain feature vector-based retrieval, thereby obtaining commodity data that has a better match with the multimedia data and mounting it on the media data, so that users can obtain more valuable commodity information when browsing multimedia content such as videos or live broadcasts.

[0176] like Fig.13 As shown, it is a schematic diagram of the structure of a recommendation processing device according to an embodiment of the present invention. The device can be applied to an e-commerce platform, a social platform with a multimedia publishing function, a live broadcast platform, etc., for actively recommending content to users. Specifically, the device may include:

[0177] The historical data acquisition module 31 is used to acquire the historical multimedia data and / or historical commodity data of the user's historical access. In the application scenario of this embodiment, active recommendations can be made based on the user's historical access behavior. For example, when a user opens the page of an e-commerce APP, some relevant content recommendations can be displayed to the user on the homepage without entering any keywords. The above-mentioned historical multimedia data and / or historical commodity data can be recorded through the user's APP, with the user's explicit permission, and reported to the e-commerce platform to provide users with a better content recommendation experience.

[0178] The historical data feature extraction module 32 is used to perform cross-domain feature extraction on historical multimedia data and / or historical commodity data to generate corresponding historical multimedia feature vectors and / or historical commodity feature vectors, wherein the cross-domain feature extraction enables the generated feature vectors to be aligned across domains.

[0179] The cross-domain feature extraction process is the same as the feature extraction process for multimedia data and commodity data mentioned in the previous embodiment, and can be implemented using a feature extraction model obtained based on the training method introduced above. In addition, the above-mentioned historical multimedia data and / or historical commodity data may include image data and / or text data, and accordingly, the above-mentioned commodity feature vector and / or multimedia feature vector may include an image feature vector extracted based on image data and / or a text feature vector extracted based on text data.

[0180] The data retrieval processing module 33 is used to retrieve multimedia data and / or commodity data according to the historical multimedia feature vectors and / or historical commodity feature vectors, and obtain multimedia data and / or commodity data whose correlation with the multimedia feature vectors and / or commodity feature vectors is greater than a preset threshold as recommended data. After the commodity feature vectors and multimedia feature vectors are extracted, the correlation can be determined by calculating the distance between the vectors. The correlation between the vectors also represents the correlation between the commodity data and the multimedia data, so that the commodity data and multimedia data with high correlation with the user's historical behavior can be retrieved, and reasonable content recommendations can be made for the user.

[0181] In the above-mentioned process of retrieving multimedia data and / or product data, the search can be performed in a pre-generated database storing multimedia feature vectors and / or a database of product feature vectors. The platform can pre-extract feature vectors from the multimedia data and / or product data it stores using the feature extraction model provided by the embodiment of the present invention to form a feature vector database. On this basis, the database can be searched based on the user's historical multimedia feature vectors and / or historical product feature vectors to obtain multimedia feature vectors and / or product feature vectors whose correlation with the historical multimedia feature vectors and / or historical product feature vectors is greater than a preset threshold; then, based on the obtained multimedia feature vectors and / or product feature vectors, the corresponding multimedia data and / or product data are determined as recommended data.

[0182] The recommendation processing device of the embodiment of the present invention performs cross-domain feature extraction on the historical multimedia data and / or historical product data involved in the user's historical access behavior based on cross-domain feature extraction processing, and retrieves multimedia data and / or product data with better matching based on the extracted feature vectors and recommends them to the user, thereby allowing the user to obtain a better content recommendation experience.

[0183] like Fig.14 As shown, it is a schematic diagram of the structure of a search processing device according to an embodiment of the present invention. The device can be applied to an e-commerce platform to perform content search according to keywords input by a user. Specifically, the device may include:

[0184] The query vector generation module 41 is used to generate a query vector according to the search information input by the user. The application scenario of this embodiment can be a search initiated by a user inputting search information through a search page of an e-commerce platform. Among them, the feature extraction of the search information can be performed using the feature extraction model provided in the above embodiment of the present invention.

[0185] The query vector retrieval module 42 is used to query the product feature vector database and / or the multimedia feature vector database according to the query vector, and obtain the product feature vector and / or the multimedia feature vector whose first correlation with the query vector is greater than the preset first threshold value, wherein the product feature vector and / or the multimedia feature vector in the product feature vector database and / or the multimedia feature vector database is obtained based on cross-domain feature extraction, and the cross-domain feature extraction enables the generated feature vector to be aligned across domains. As previously introduced, the e-commerce platform can extract feature vectors from multimedia data and product data in advance using the feature extraction model provided by the embodiment of the present invention to form a feature vector database. On this basis, after the search information input by the user is converted into a query vector, it can be compared with the vector in the vector database.

[0186] The search result feedback module 43 is used to return the corresponding product data and / or multimedia data to the user as the search result according to the obtained product feature vector and / or multimedia feature vector. The search results returned to the user may include multimedia data such as short videos and live broadcasts, and may also include product data published on the e-commerce platform. In addition, the multimedia data may further include product data, such as product links associated with the short video.

[0187] For the product data mounted in the multimedia data, the e-commerce platform can determine the relevance between the mounted product data and the multimedia data when returning the search results to the user, and eliminate some irrelevant products. In addition, for multimedia data that has no mounted product data or less mounted product data in the search results, the relevant product data can be further obtained for mounting and recommended to the user.

[0188] The retrieval processing device of an embodiment of the present invention, based on a feature vector database formed by cross-domain feature extraction processing, can perform cross-domain feature vector matching queries according to feature vectors formed by retrieval information input by a user, and obtain multimedia data and / or product data that meets the user's retrieval requirements, thereby being able to provide users with rich retrieval results from different content domains.

[0189] like Fig.15As shown, it is a schematic diagram of the structure of the training device of the feature extraction model of the embodiment of the present invention. The training device of the embodiment of the present invention can be applied to e-commerce platforms, search engines, and data service platforms that provide cloud services, and is used to train feature extraction models in processing systems such as search engines and content recommendations, so as to realize feature extraction of data in different content domains, and enable the extracted feature vectors to be aligned, so as to facilitate subsequent feature comparison or fusion between various content domains. Further processing, the device may include:

[0190] The sample acquisition module 51 is used to obtain content-related samples in multiple content domains to form multiple cross-domain sample combinations. In order to train the feature extraction model, a sample set can be pre-constructed. Among them, the sample can come from data in any content domain, such as a video from a multimedia domain or a commodity data from a commodity domain. If there is a known correlation between these samples from different content domains, then these samples can be combined into a cross-domain sample combination, and the user performs model training. In the previous example, the sample pair composed of commodity data mounted in multimedia data is used as an example of a cross-domain sample combination. In actual applications, since there may be multiple content domains, the number of samples in the cross-domain sample combination may also be multiple.

[0191] The feature vector generation module 52 is used to use the trained feature extraction model to perform feature extraction on the original data of the samples in the cross-domain sample combination, and generate a feature vector corresponding to the sample. Wherein, no matter which content domain the sample data is from, it can include data of multiple modalities. The modality mentioned here refers to the form in which the data itself exists. The modality generally includes: image, audio, text, etc. Therefore, the feature extraction process may include: performing feature extraction on the original data of each modality of the sample in the cross-domain sample combination, and generating feature vectors of multiple modalities corresponding to the sample. In an embodiment of the present invention, image data and text data can be selected as the main modalities, and audio data can be converted into text data in advance, that is, the original data of the above-mentioned sample can include image data and / or text data. Accordingly, the feature extraction process may include: performing feature extraction on the image data and / or text data of the sample in the cross-domain sample combination, and generating an image feature vector and / or text feature vector corresponding to the sample.

[0192] It should be noted here that the contrastive learning and the subsequent adversarial learning in the embodiments of the present invention are mainly training for cross-domain feature extraction capabilities. When there is a need for cross-modal feature extraction, the above-mentioned trained feature extraction model can adopt a pre-trained feature extraction model with cross-modal feature extraction capabilities. For example, in model selection, the pre-trained BERT model can be used for the text feature extraction part, or an end-to-end machine learning model can be adopted. For the image feature extraction part, a pre-trained ResNet model can be used.

[0193] The training module 53 is used to perform comparative learning training on the feature extraction model to reduce the distance between feature vectors corresponding to samples within the cross-domain sample combination and to enlarge the distance between feature vectors corresponding to samples between the cross-domain sample combinations as training targets.

[0194] In addition, in order to improve the training effect of contrastive learning, the embodiment of the present invention can introduce adversarial learning for auxiliary training. Therefore, the above-mentioned training module 53 can also be used for: performing adversarial learning training on the feature extraction model to reduce the accuracy of the discriminator of adversarial learning for the content domain to which the feature vector belongs as a training goal. In this embodiment, the role of adversarial learning is to increase the difficulty of contrastive learning and serve as a discriminator to test the effect of contrastive learning. That is, when the discriminator of adversarial learning cannot accurately determine which content domain the feature vector extracted by the feature extraction model comes from, it is equivalent to that contrastive learning has well aligned the feature vectors of different content domains, achieving a better cross-domain feature extraction effect.

[0195] Specific examples of contrastive learning and adversarial learning have been introduced in the previous training method embodiments and will not be repeated here.

[0196] The model training device provided in the embodiment of the present invention can align the feature vectors extracted from different content domains by the trained feature extraction model through contrastive learning or contrastive learning combined with adversarial learning, thereby realizing cross-domain feature comparison and feature fusion, and effectively improving the matching effect of cross-domain content retrieval and recommendation.

[0197] The above embodiments describe the process of model training method, commodity mounting processing method, retrieval method, recommendation method and corresponding device structure. The functions of the above methods and devices can be realized by means of an electronic device, such as Fig.16 As shown, it is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention, which specifically includes: a memory 110 and a processor 120.

[0198] The memory 110 is used to store programs.

[0199] In addition to the above-mentioned programs, the memory 110 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0200] The memory 110 may be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0201] The processor 120 is coupled to the memory 110 and is used to execute the program in the memory 110 to perform the operation steps of any one method or multiple methods described in the above embodiments.

[0202] In addition, the processor 120 may also include the various modules described in the aforementioned embodiments to perform processing of any one or more methods described in the aforementioned embodiments, and the memory 110 may be used, for example, to store data required for these modules to perform operations and / or output data.

[0203] The specific description of the above-mentioned processing process, the detailed description of the technical principle and the detailed analysis of the technical effect have been described in detail in the previous embodiments and will not be repeated here.

[0204] Furthermore, as shown in the figure, the electronic device may also include other components such as a communication component 130, a power component 140, an audio component 150, and a display 160. The figure only schematically shows some components, which does not mean that the electronic device only includes the components shown in the figure.

[0205] The communication component 130 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component 130 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 130 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0206] The power supply component 140 provides power to various components of the electronic device. The power supply component 140 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0207] The audio component 150 is configured to output and / or input audio signals. For example, the audio component 150 includes a microphone (MIC), and when the electronic device is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 110 or sent via the communication component 130. In some embodiments, the audio component 150 also includes a speaker for outputting audio signals.

[0208] The display 160 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0209] In addition, an embodiment of the present invention further provides a computer program product, including a computer program or instructions. When the computer program or instructions are executed by a processor, the processor is caused to implement the aforementioned method.

[0210] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0211] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a feature extraction model, comprising: Acquire samples associated with contents in multiple content domains to form multiple cross-domain sample combinations; Using the trained feature extraction model, extracting features from raw data of samples in the cross-domain sample combination to generate feature vectors corresponding to the samples, wherein the samples are multimodal data; Performing comparative learning training on the feature extraction model to reduce the distance between feature vectors corresponding to samples within the cross-domain sample combination and to enlarge the distance between feature vectors corresponding to samples between the cross-domain sample combinations as a training target; Performing adversarial learning training on the feature extraction model, with the aim of reducing the accuracy of the adversarial learning discriminator in distinguishing the content domain to which the feature vector belongs as a training target; In the adversarial learning training, the training process is performed alternately, including fixing the model parameters of the discriminator and training the feature extraction model to reduce the discriminator's accuracy in distinguishing the content domain to which the feature vector belongs; fixing the model parameters of the feature extraction model and training the discriminator to improve the discriminator's accuracy in distinguishing the content domain to which the feature vector belongs.

2. The method according to claim 1, wherein: The original data includes data of multiple modalities, and the step of extracting features from the original data of the samples in the cross-domain sample combination to generate feature vectors corresponding to the samples includes: Feature extraction is performed on the original data of each modality of the samples in the cross-domain sample combination to generate feature vectors of multiple modalities corresponding to the samples.

3. The method according to claim 1, wherein: The original data of the sample includes image data and / or text data. The extracting features from the original data of the samples in the cross-domain sample combination to generate feature vectors corresponding to the samples includes: Feature extraction is performed on the image data and / or text data of the samples in the cross-domain sample combination to generate image feature vectors and / or text feature vectors corresponding to the samples.

4. The method according to claim 2, wherein: The trained feature extraction model adopts a pre-trained feature extraction model with cross-modal feature extraction capability.

5. A method for processing commodity mounting, comprising: Acquire multimedia data, perform cross-domain feature extraction on the multimedia data, and generate a multimedia feature vector corresponding to the multimedia data, wherein the cross-domain feature extraction enables the generated feature vector to be aligned across domains, and the cross-domain feature extraction is implemented by a feature extraction model, and the training process of the feature extraction model includes contrastive learning training and adversarial learning training, and the adversarial learning training takes reducing the discrimination accuracy of the adversarial learning discriminator for the content domain to which the feature vector belongs as a training goal; in the adversarial learning training, alternately perform training processing, including fixing the model parameters of the discriminator and training the feature extraction model to reduce the discrimination accuracy of the discriminator for the content domain to which the feature vector belongs; fixing the model parameters of the feature extraction model and training the discriminator to improve the discrimination accuracy of the discriminator for the content domain to which the feature vector belongs; Acquire multiple candidate product data, perform cross-domain feature extraction on the candidate product data, and generate multiple product feature vectors, wherein the multimedia data and the candidate product data are both multimodal data; The commodity data to be mounted is determined from the plurality of candidate commodity data according to the correlation between the plurality of commodity feature vectors and the multimedia feature vector.

6. The method according to claim 5, wherein: Acquiring multiple candidate product data includes: In response to a user selecting an operation in a product database, the plurality of candidate product data are determined.

7. The method according to claim 5, wherein: The commodity feature vector and / or multimedia feature vector includes an image feature vector extracted based on image data and / or a text feature vector extracted based on text data.

8. A method for processing commodity mounting, comprising: Acquire multimedia data, perform cross-domain feature extraction on the multimedia data, and generate a multimedia feature vector corresponding to the multimedia data, wherein the cross-domain feature extraction enables the generated feature vector to be aligned across domains, and the cross-domain feature extraction is implemented by a feature extraction model, and the training process of the feature extraction model includes contrastive learning training and adversarial learning training, and the adversarial learning training takes reducing the discrimination accuracy of the adversarial learning discriminator for the content domain to which the feature vector belongs as a training goal; in the adversarial learning training, alternately perform training processing, including fixing the model parameters of the discriminator and training the feature extraction model to reduce the discrimination accuracy of the discriminator for the content domain to which the feature vector belongs; fixing the model parameters of the feature extraction model and training the discriminator to improve the discrimination accuracy of the discriminator for the content domain to which the feature vector belongs; Retrieve product data according to the multimedia feature vector, and obtain product data corresponding to the product feature vector whose correlation with the multimedia feature vector is greater than a preset threshold, wherein the multimedia data and the product data are respectively multimodal data; The commodity data is mounted on the multimedia data.

9. The method according to claim 8, wherein: Retrieving product data according to the multimedia feature vector, and obtaining product data corresponding to a product feature vector whose correlation with the multimedia feature vector is greater than a preset threshold comprises: According to the multimedia feature vector, searching in a product feature vector database to obtain a product feature vector whose correlation with the multimedia feature vector is greater than a preset threshold; The corresponding product data is obtained according to the product feature vector.

10. A recommended processing method, comprising: Obtain historical multimedia data and / or historical product data of a user's historical visits; Performing cross-domain feature extraction on the historical multimedia data and / or historical commodity data to generate corresponding historical multimedia feature vectors and / or historical commodity feature vectors, wherein the cross-domain feature extraction enables the generated feature vectors to be aligned across domains, the historical multimedia data and the historical commodity data are both multimodal data, the cross-domain feature extraction is implemented by a feature extraction model, the training process of the feature extraction model includes contrastive learning training and adversarial learning training, and the adversarial learning training takes reducing the discriminant accuracy of the adversarial learning discriminator for the content domain to which the feature vector belongs as a training goal; In the adversarial learning training, training processing is performed alternately, including fixing the model parameters of the discriminator and training the feature extraction model to reduce the discriminant accuracy of the discriminator for the content domain to which the feature vector belongs; fixing the model parameters of the feature extraction model and training the discriminator to improve the discriminant accuracy of the discriminator for the content domain to which the feature vector belongs; According to the historical multimedia feature vectors and / or historical product feature vectors, multimedia data and / or product data are retrieved, and multimedia data and / or product data whose correlation with the multimedia feature vectors and / or product feature vectors is greater than a preset threshold are obtained as recommendation data.

11. The method according to claim 10, wherein: According to the historical multimedia feature vector and / or the historical commodity feature vector, multimedia data and / or commodity data are retrieved, and the multimedia data and / or commodity data whose correlation with the multimedia feature vector and / or the commodity feature vector is greater than a preset threshold are obtained as recommendation data, including: According to the historical multimedia feature vectors and / or historical commodity feature vectors, searching in a database storing multimedia feature vectors and / or a database storing commodity feature vectors, obtaining multimedia feature vectors and / or commodity feature vectors whose correlation with the historical multimedia feature vectors and / or historical commodity feature vectors is greater than a preset threshold; According to the acquired multimedia feature vector and / or commodity feature vector, corresponding multimedia data and / or commodity data are determined as the recommendation data.

12. The method according to claim 10, wherein: The historical multimedia data and / or historical commodity data are multiple, the historical multimedia feature vectors and / or the historical commodity feature vectors are also multiple, and the method further includes: fusing the multiple historical multimedia feature vectors and / or the historical commodity feature vectors to generate a fused feature vector, The retrieving multimedia data and / or commodity data according to the historical multimedia feature vector and / or the historical commodity feature vector to obtain multimedia data and / or commodity data having a correlation with the multimedia feature vector and / or the commodity feature vector greater than a preset threshold comprises: Multimedia data and / or commodity data are retrieved according to the fused feature vector, and multimedia data and / or commodity data having a correlation with the fused feature vector greater than a preset threshold are obtained.

13. A search processing method, comprising: Generate a query vector based on the search information input by the user; According to the query vector, a query is performed in a product feature vector database and / or a multimedia feature vector database to obtain a product feature vector and / or a multimedia feature vector having a first correlation with the query vector greater than a preset first threshold, wherein the product feature vector and / or the multimedia feature vector in the product feature vector database and / or the multimedia feature vector database is obtained based on cross-domain feature extraction, the cross-domain feature extraction enables the generated feature vector to be aligned across domains, the cross-domain feature extraction is implemented by a feature extraction model, the training process of the feature extraction model includes contrastive learning training and adversarial learning training, and the adversarial learning training takes reducing the discrimination accuracy of the adversarial learning discriminator for the content domain to which the feature vector belongs as a training goal; In the adversarial learning training, training processing is performed alternately, including fixing the model parameters of the discriminator and training the feature extraction model to reduce the discriminant accuracy of the discriminator for the content domain to which the feature vector belongs; fixing the model parameters of the feature extraction model and training the discriminator to improve the discriminant accuracy of the discriminator for the content domain to which the feature vector belongs, wherein the product feature vector and the multimedia feature vector both correspond to vectors of multimodal data; According to the acquired product feature vector and / or multimedia feature vector, the corresponding product data and / or multimedia data are returned to the user as a search result.

14. The method according to claim 13, wherein: In the case of returning multimedia data to the user, it also includes: Obtaining a commodity feature vector corresponding to the commodity data mounted in the multimedia data; A second correlation between the multimedia feature vector corresponding to the multimedia data and the product feature vector is obtained. If the second correlation is greater than or equal to a preset second threshold, the mounted product data is retained in the search results returned to the user; otherwise, the mounted product data is deleted.

15. The method according to claim 13, wherein: In the case of returning multimedia data to the user, it also includes: According to the multimedia feature vector corresponding to the multimedia data, query is performed in a product feature vector database to obtain a product feature vector whose correlation with the multimedia feature vector is greater than a preset third threshold value; The corresponding product data is obtained according to the product feature vector, and is mounted into the multimedia data and provided to the user as a search result.

16. A training device for a feature extraction model, comprising: A sample acquisition module, used to acquire samples associated with contents in multiple content domains to form multiple cross-domain sample combinations; A feature vector generation module, used to extract features from the original data of the samples in the cross-domain sample combination using the trained feature extraction model, and generate feature vectors corresponding to the samples, wherein the samples are multimodal data; A training module, used for performing contrastive learning training on the feature extraction model to reduce the distance between feature vectors corresponding to samples within the cross-domain sample combination and to enlarge the distance between feature vectors corresponding to samples between the cross-domain sample combinations as a training target, and performing adversarial learning training on the feature extraction model to reduce the discriminant accuracy of the adversarial learning discriminator for the content domain to which the feature vector belongs as a training target; In the adversarial learning training, the training process is performed alternately, including fixing the model parameters of the discriminator and training the feature extraction model to reduce the discriminator's accuracy in distinguishing the content domain to which the feature vector belongs; fixing the model parameters of the feature extraction model and training the discriminator to improve the discriminator's accuracy in distinguishing the content domain to which the feature vector belongs.

17. A commodity mounting processing device, comprising: A multimedia feature extraction module, used to obtain multimedia data, perform cross-domain feature extraction on the multimedia data, and generate a multimedia feature vector corresponding to the multimedia data, wherein the cross-domain feature extraction enables the generated feature vector to be aligned across domains, and the cross-domain feature extraction is implemented by a feature extraction model, and the training process of the feature extraction model includes contrastive learning training and adversarial learning training, and the adversarial learning training takes reducing the discrimination accuracy of the adversarial learning discriminator for the content domain to which the feature vector belongs as a training goal; in the adversarial learning training, the training process is performed alternately, including fixing the model parameters of the discriminator and training the feature extraction model to reduce the discrimination accuracy of the discriminator for the content domain to which the feature vector belongs; fixing the model parameters of the feature extraction model and training the discriminator to improve the discrimination accuracy of the discriminator for the content domain to which the feature vector belongs; A product feature extraction module is used to obtain multiple candidate product data, perform cross-domain feature extraction on the candidate product data, and generate multiple product feature vectors, wherein the multimedia data and the candidate product data are both multimodal data; The relevance processing module is used to determine the commodity data to be mounted from the plurality of candidate commodity data according to the relevance between the plurality of commodity feature vectors and the multimedia feature vector.

18. A processing device for mounting related commodities, comprising: A multimedia feature extraction module, used to obtain multimedia data, perform cross-domain feature extraction on the multimedia data, and generate a multimedia feature vector corresponding to the multimedia data, wherein the cross-domain feature extraction enables the generated feature vector to be aligned across domains, and the cross-domain feature extraction is implemented by a feature extraction model, and the training process of the feature extraction model includes contrastive learning training and adversarial learning training, and the adversarial learning training takes reducing the discrimination accuracy of the adversarial learning discriminator for the content domain to which the feature vector belongs as a training goal; in the adversarial learning training, the training process is performed alternately, including fixing the model parameters of the discriminator and training the feature extraction model to reduce the discrimination accuracy of the discriminator for the content domain to which the feature vector belongs; fixing the model parameters of the feature extraction model and training the discriminator to improve the discrimination accuracy of the discriminator for the content domain to which the feature vector belongs; A product retrieval processing module, used to retrieve product data according to the multimedia feature vector, and obtain product data corresponding to the product feature vector whose correlation with the multimedia feature vector is greater than a preset threshold, wherein the multimedia data and the product data are respectively multimodal data; The mounting processing module is used to mount the commodity data onto the multimedia data.

19. A recommendation processing device, comprising: A historical data acquisition module is used to acquire historical multimedia data and / or historical commodity data that the user has visited in the past; A historical data feature extraction module, used for performing cross-domain feature extraction on the historical multimedia data and / or historical commodity data to generate corresponding historical multimedia feature vectors and / or historical commodity feature vectors, wherein the cross-domain feature extraction enables the generated feature vectors to be aligned across domains, the historical multimedia data and historical commodity data are both multimodal data, the cross-domain feature extraction is implemented through a feature extraction model, the training process of the feature extraction model includes contrastive learning training and adversarial learning training, the adversarial learning training takes reducing the discrimination accuracy of the adversarial learning discriminator for the content domain to which the feature vector belongs as a training goal; in the adversarial learning training, the training process is performed alternately, including fixing the model parameters of the discriminator and training the feature extraction model to reduce the discrimination accuracy of the discriminator for the content domain to which the feature vector belongs; fixing the model parameters of the feature extraction model and training the discriminator to improve the discrimination accuracy of the discriminator for the content domain to which the feature vector belongs; A data retrieval processing module is used to retrieve multimedia data and / or commodity data based on the historical multimedia feature vectors and / or historical commodity feature vectors, and obtain multimedia data and / or commodity data whose correlation with the multimedia feature vectors and / or commodity feature vectors is greater than a preset threshold as recommendation data.

20. A retrieval processing device, comprising: A query vector generation module, used to generate a query vector according to the search information input by the user; A query vector retrieval module, configured to query a product feature vector database and / or a multimedia feature vector database according to the query vector, and obtain a product feature vector and / or a multimedia feature vector having a first correlation with the query vector greater than a preset first threshold, wherein the product feature vector and / or the multimedia feature vector in the product feature vector database and / or the multimedia feature vector database is obtained based on cross-domain feature extraction, the cross-domain feature extraction enables the generated feature vector to be aligned across domains, the cross-domain feature extraction is implemented by a feature extraction model, the training process of the feature extraction model includes contrastive learning training and adversarial learning training, and the adversarial learning training takes reducing the discriminant accuracy of the adversarial learning discriminator for the content domain to which the feature vector belongs as a training goal; In the adversarial learning training, training processing is performed alternately, including fixing the model parameters of the discriminator and training the feature extraction model to reduce the discriminant accuracy of the discriminator for the content domain to which the feature vector belongs; fixing the model parameters of the feature extraction model and training the discriminator to improve the discriminant accuracy of the discriminator for the content domain to which the feature vector belongs, wherein the product feature vector and the multimedia feature vector both correspond to vectors of multimodal data; The search result feedback module is used to return corresponding product data and / or multimedia data to the user as the search result based on the acquired product feature vector and / or multimedia feature vector.

21. A method for processing commodity mounting, comprising: Get multimedia data uploaded by users; According to the content features obtained by performing cross-domain feature extraction on the multimedia data, matching product data is selected in the product database and recommended to the user, wherein both the multimedia data and the product data are multimodal data, and the cross-domain feature extraction is implemented by a feature extraction model. The training process of the feature extraction model includes contrastive learning training and adversarial learning training. The adversarial learning training takes reducing the discrimination accuracy of the adversarial learning discriminator for the content domain to which the feature vector belongs as a training goal; In the adversarial learning training, training processing is performed alternately, including fixing the model parameters of the discriminator and training the feature extraction model to reduce the discriminant accuracy of the discriminator for the content domain to which the feature vector belongs; fixing the model parameters of the feature extraction model and training the discriminator to improve the discriminant accuracy of the discriminator for the content domain to which the feature vector belongs; In response to the user's selection of the commodity data, the selected commodity data is mounted into the multimedia data.

22. The method according to claim 21, wherein: Also includes: Obtain comment data after multimedia data is released; According to the features obtained by performing cross-domain feature extraction on the comment data, matching products are selected in the product database and recommended to the user, so as to update the products mounted in the multimedia data.

23. A method for processing commodity mounting, comprising: Obtain multimedia data uploaded by users and multiple commodity data to be mounted; According to the content features obtained by performing cross-domain feature extraction on the multimedia data and the product features obtained by performing cross-domain feature extraction on the product data, the correlation between the product data and the multimedia data is obtained, wherein both the multimedia data and the product data are multimodal data, the cross-domain feature extraction is implemented by a feature extraction model, and the training process of the feature extraction model includes contrastive learning training and adversarial learning training, wherein the adversarial learning training takes reducing the discriminant accuracy of the adversarial learning discriminator for the content domain to which the feature vector belongs as a training goal; In the adversarial learning training, training processing is performed alternately, including fixing the model parameters of the discriminator and training the feature extraction model to reduce the discriminant accuracy of the discriminator for the content domain to which the feature vector belongs; fixing the model parameters of the feature extraction model and training the discriminator to improve the discriminant accuracy of the discriminator for the content domain to which the feature vector belongs; According to the relevance, commodity data is recommended from a plurality of commodity data to be mounted.

24. The method according to claim 23, wherein: Recommending product data from a plurality of product data to be mounted according to the relevance includes: The relevance of the plurality of product data to the multimedia data is ranked, and the product data is recommended based on the ranking.

25. The method according to claim 23, wherein: Also includes: Obtain comment data after multimedia data is released; The product recommendation to the user is updated according to the features obtained by performing cross-domain feature extraction on the comment data.

26. An electronic device comprising: Memory, used to store programs; A processor, used to run the program stored in the memory to execute the training method of the feature extraction model described in any one of claims 1 to 4, and / or the commodity mounting processing method described in any one of claims 5 to 9 and 21 to 25, and / or the recommendation processing method described in any one of claims 10 to 12, and / or the retrieval processing method described in any one of claims 13 to 15.