Object recommendation method and device

Abstract

The invention provides an object recommendation method and device, and relates to the technical field of artificial intelligence. The method comprises the steps that a behavior sequence of a user and an image and a description text of a target object are obtained, and the behavior sequence comprises one or more actions; utilizing a multi-modal encoder to extract embedded features of each action and multi-modal features of the image and the description text of the target object; fusing the embedded features of each action based on the multi-modal features to obtain multi-modal behavior sequence features; and determining whether to recommend the target object to the user based on the multi-modal behavior sequence features and the multi-modal features. Through the technical means, the problem that personalized and accurate recommendation service cannot be provided in the related technology is solved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of object recommendation, and in particular to an object recommendation method and device. Background Art

[0002] Traditional recommendation methods still rely primarily on content-based filtering, collaborative filtering, or hybrid recommendations, typically analyzing historical user behavior data and basic object attributes. However, these recommendation mechanisms have limitations in practical applications, particularly in providing personalized and accurate recommendations. Summary of the Invention

[0003] The present disclosure provides an object recommendation method and apparatus, which at least to a certain extent improve the personalization level and accuracy of recommendation services.

[0004] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0005] According to one aspect of the present disclosure, a method for object recommendation is provided, comprising: obtaining a user's behavior sequence and an image and explanatory text of a target object, wherein the behavior sequence includes one or more actions; extracting embedded features of each action and multimodal features of the image and explanatory text of the target object using a multimodal encoder; fusing the embedded features of each action based on the multimodal features to obtain multimodal behavior sequence features; and determining whether to recommend the target object to the user based on the multimodal behavior sequence features and the multimodal features.

[0006] In one embodiment of the present disclosure, before using a multimodal encoder to extract embedded features of each action and multimodal features of the image and explanatory text of the target object, the method also includes: obtaining training data, wherein the training data includes multiple training objects and images and explanatory texts of each training object; using a multimodal encoder to extract multimodal features of the image and explanatory text of each training object, wherein the multimodal features corresponding to each training object include image features of the image of the training object and text features of the explanatory text, and the images and explanatory texts of each training object in the multimodal encoder share a multi-head attention module; based on the image features and text features corresponding to each training object, using a contrast loss function to calculate the contrast loss; and optimizing the parameters of the multimodal encoder based on the contrast loss to fine-tune the multimodal encoder.

[0007] In one embodiment of the present disclosure, before using the multimodal encoder to extract the embedded features of each action and the multimodal features of the image and explanatory text of the target object, the method further includes: obtaining training data, wherein the training data includes multiple training objects and the image and explanatory text of each training object; executing the following loop to fine-tune the multimodal encoder for multiple batches: judging whether m is equal to n, wherein m is the sequence number of the current batch, n is the number of multiple batches, n is a positive integer, m is an integer not greater than n, and the initial value of m is 0; if equal, determining that the fine-tuning of the multimodal encoder for multiple batches is completed and exiting the loop; if less than, using the multimodal encoder to extract each action. The method comprises the following steps: first, determining the multimodal features of an image and a description text of each training object, wherein the multimodal features corresponding to each training object include the image features of the image of the training object and the text features of the description text; using a momentum model to determine the positive and negative samples of each image feature from the multiple text features, and determining the positive and negative samples of each text feature from the multiple image features; using a contrast loss function to calculate the contrast loss based on each image feature and the positive and negative samples of each image feature, each text feature and the positive and negative samples of each text feature; optimizing the parameters of the multimodal encoder according to the contrast loss, optimizing the parameters of the momentum model using the multimodal encoder, and updating m with a value of m plus 1.

[0008] In one embodiment of the present disclosure, the contrast loss L is calculated according to the following contrast loss function: ModalNCE :

[0009]

[0010] Among them, I j is the jth image feature, T j+ is the positive sample of the jth image feature, T i is the feature of the i-th text, except T j+ All text features other than the jth image feature are negative samples, T j is the jth text feature, I j+ is the positive sample of the jth text feature, I i is the i-th image feature, except I j+ All image features other than are negative samples of the jth text feature, K is the number of training objects, K is greater than 0, j and i are both between 0 and K, τ is the temperature coefficient, τ is between 0 and 1.

[0011] In one embodiment of the present disclosure, before using a multimodal encoder to extract multimodal features of images and explanatory texts of each training object, the method also includes: tokenizing the images and texts corresponding to each training object to obtain image tokens and text tokens; performing random masking operations on the image tokens and text tokens; inputting the image tokens and text tokens after the random masking operations into the multimodal encoder, and outputting the prediction results corresponding to the image tokens and the prediction results corresponding to the text tokens; using a cross-entropy loss function to calculate the cross-entropy loss between the image tokens and their corresponding prediction results, as well as the cross-entropy loss between the text tokens and their corresponding prediction results; and optimizing the parameters of the multimodal encoder based on the cross-entropy loss to pre-train the multimodal encoder.

[0012] In one embodiment of the present disclosure, before optimizing the parameters of the multimodal encoder based on the contrast loss, the method further includes: calculating the contrast loss using a contrast loss function based on each image feature and the positive and negative samples of each image feature; or calculating the contrast loss using a contrast loss function based on each text feature and the positive and negative samples of each text feature.

[0013] In one embodiment of the present disclosure, the parameters of the momentum model are optimized according to the following formula, including: θ t =mθ t-1 +(1-m)θ q ; where θ t is the parameter of the current batch of momentum model, θ t-1 is the parameter of the momentum model in the previous batch, θ q is the parameter of the current batch of the multimodal encoder, m is the preset threshold, and m is between 0 and 1.

[0014] In one embodiment of the present disclosure, the embedded features of each action are fused based on the multimodal features to obtain the multimodal behavior sequence feature, including: generating the weight of the embedded features of each action based on the multimodal features; fusing the embedded features of each action according to the weight of the embedded features of each action to obtain the multimodal behavior sequence feature.

[0015] In one embodiment of the present disclosure, the weight of the embedded feature of each action is generated based on the multimodal feature, including: performing a dot product operation on the multimodal feature and the embedded feature of each action to obtain the weight of the embedded feature of each action.

[0016] According to another aspect of the present disclosure, an object recommendation device is provided, comprising: an acquisition module configured to acquire a user's behavior sequence and an image and explanatory text of a target object, wherein the behavior sequence includes one or more actions; an extraction module configured to extract, using a multimodal encoder, embedded features of each action and multimodal features of the image and explanatory text of the target object; a fusion module configured to fuse the embedded features of each action based on the multimodal features to obtain multimodal behavior sequence features; and a recommendation module configured to determine whether to recommend the target object to the user based on the multimodal behavior sequence features and the multimodal features.

[0017] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any of the above methods by executing the executable instructions.

[0018] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any of the above methods is implemented.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including computer instructions stored in a computer-readable storage medium, and the computer instructions implement operating instructions of any of the above methods when executed by a processor.

[0020] In the embodiments of the present disclosure, in order to take into account the potential impact between user behavior and the content of the target object, the embedded features of each action are fused based on multimodal features to obtain multimodal behavior sequence features representing the user behavior sequence, and whether to recommend the target object to the user is determined based on the multimodal behavior sequence features and the multimodal features, thereby solving the problem in related technologies that personalized and accurate recommendation services cannot be provided, and thus improving the personalization level and accuracy of the recommendation service.

[0021] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0024] Figure 1 A schematic diagram of an object recommendation system in an embodiment of the present disclosure is shown.

[0025] Figure 2 A flowchart of an object recommendation method in an embodiment of the present disclosure is shown.

[0026] Figure 3 A flowchart of a multimodal encoder fine-tuning method according to an embodiment of the present disclosure is shown.

[0027] Figure 4 A flowchart of a multimodal encoder pre-training method in an embodiment of the present disclosure is shown.

[0028] Figure 5 A schematic diagram showing another object recommendation system in an embodiment of the present disclosure.

[0029] Figure 6 A flow chart of a feature processing method in an embodiment of the present disclosure is shown.

[0030] Figure 7 A schematic diagram showing an object recommendation device according to an embodiment of the present disclosure.

[0031] Figure 8 A schematic diagram of an electronic device provided in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0032] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0033] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0034] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0035] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0036] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0037] It should be pointed out that, in the absence of conflict, the embodiments of the present disclosure and the technical features therein may be combined with each other.

[0038] For ease of understanding, the terms used in this disclosure are first explained as follows:

[0039] BEiTIII (Bidirectional Encoder Representations from Transformers) is primarily used for image processing and understanding tasks. Inspired by BERT (a model for natural language processing), the BEiT model applies a similar masked language model to images, learning high-level representations of images through pre-training.

[0040] A token is the smallest unit of information processed by a model. Tokenization refers to dividing data or features into the smallest units that a model can recognize and process. Text tokenization divides text (text features) into words (word / subword tokens), while image tokenization divides images (image features) into image blocks or pixel blocks (patch tokens).

[0041] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0042] Figure 1 A schematic diagram of an object recommendation system according to an embodiment of the present disclosure is shown. The object recommendation system includes: a user device 101 and a server 102 .

[0043] Optionally, the application clients installed on different devices 101 are the same, or the application clients installed on two devices 101 are clients of the same type of application on different control system platforms. Based on the different device platforms, the specific form of the application client may also be different. For example, the application client may be a mobile phone client, a PC client, or a World Wide Web (Web) client.

[0044] Those skilled in the art will appreciate that the number of the devices 101 may be greater or less. For example, there may be only one device, or there may be dozens, hundreds, or even more. The embodiments of this application do not limit the number and type of devices.

[0045] Server 102 is composed of one or more servers. An application program may be installed in server 102 to perform the following operations: obtaining a user's behavior sequence and an image and description text of a target object, wherein the behavior sequence includes one or more actions; extracting embedded features of each action and multimodal features of the image and description text of the target object using a multimodal encoder; fusing the embedded features of each action based on the multimodal features to obtain a multimodal behavior sequence feature; and determining whether to recommend the target object to the user based on the multimodal behavior sequence feature and the multimodal features.

[0046] The device 101 is connected to the server 102 via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0047] Optionally, the above-mentioned wireless network or wired network uses standard communication technologies and / or protocols. The network is typically the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged over the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0048] Figure 2The flowchart of an object recommendation method in the embodiment of the present disclosure is shown. The recommended objects in the embodiment of the present disclosure can be news, articles, and products, etc., and the following text takes product recommendation as an example. Figure 2 As shown, the following steps are included:

[0049] S201 , obtaining a user's behavior sequence and an image and description text of a target object, wherein the behavior sequence includes one or more actions.

[0050] In the disclosed embodiments, a behavior sequence is a record of a user's activities related to an object, including browsing, clicking, purchasing, and collecting various objects, as well as the time at which each action occurred. These actions reflect the user's interest and preference for different objects, which are time-dependent (e.g., the closer the action occurs to the current time, the stronger the interest and preference represented). A target object is an object that is determined to be recommended to the user. For example, a target object is a product. The description text of the target object includes textual content such as the product's name, function, and purpose, and the image of the target object is an image of the product.

[0051] S202 , using a multimodal encoder to extract embedded features of each action and multimodal features of the target object's image and description text.

[0052] A multimodal encoder is a tool for extracting features from multiple modal information, such as images and text. In the disclosed embodiments, the multimodal encoder uses a shared multi-head attention module for both images and text (that is, the same multi-head attention module is used to process both images and text), thereby aligning the representational features of the image and the caption text of the object. The multimodal encoder can utilize BEiTIII.

[0053] To facilitate subsequent processing, a multimodal encoder is used to extract embedded features for each action. The multimodal encoder is also used to extract multimodal features from the target object's image and caption text. This allows alignment of the representational features of the user action with the object's image and caption text. The multimodal features corresponding to the target object include the image features of the target object's image and the text features of the caption text.

[0054] S203: Based on the multimodal features, the embedded features of each action are fused to obtain a multimodal behavior sequence feature.

[0055] Multimodal behavior sequence features comprehensively consider the feature representation of the user's actions generated during the interaction with the object. They not only focus on the user's actions themselves, but also combine the content characteristics of the target object. Based on the multimodal features, the embedded features of each action are fused to more comprehensively capture the user's interest preferences and their dynamic changes, thereby providing more personalized and accurate services or recommendations.

[0056] In one embodiment of the present disclosure, the embedded features of each action are fused based on the multimodal features to obtain the multimodal behavior sequence feature, including: generating the weight of the embedded features of each action based on the multimodal features; fusing the embedded features of each action according to the weight of the embedded features of each action to obtain the multimodal behavior sequence feature.

[0057] In an optional embodiment, generating the weight of the embedded feature of each action based on the multimodal feature includes: performing a dot product operation on the multimodal feature and the embedded feature of each action to obtain the weight of the embedded feature of each action.

[0058] In an optional embodiment, the embedded features of each action are fused according to the weights of the embedded features of each action to obtain a multimodal behavior sequence feature, including: weighted summing the embedded features of each action according to the weights of the embedded features of each action to obtain a multimodal behavior sequence feature.

[0059] S204: Determine whether to recommend the target object to the user based on the multimodal behavior sequence feature and the multimodal feature.

[0060] In an optional embodiment, the similarity between the multimodal behavior sequence feature and the multimodal feature is calculated. If the similarity is greater than a preset threshold, the target object is recommended to the user; otherwise, the target object is not recommended to the user.

[0061] According to the technical solution provided by the embodiments of the present disclosure, the same multimodal encoder is used to extract the embedded features of each action and the multimodal features of the target object's image and explanatory text, and the user's behavior is aligned with the object's image and explanatory text in terms of representational features. Because the image and text in the multimodal encoder share a multi-head attention module, the object's image and explanatory text are aligned in terms of representational features. Based on the multimodal features, the embedded features of each action are fused to ultimately obtain the multimodal behavior sequence features corresponding to the user's behavior sequence. This solves the problem of related technologies that cannot provide personalized and accurate recommendation services, thereby improving the personalization level and accuracy of recommendation services.

[0062] For example, on an e-commerce platform, user A frequently browses dresses in the women's clothing category but does not purchase them. When a new dress is added to the store, the image and description of the dress are first obtained, and user A's behavior sequence is collected. Next, a multimodal encoder is used to extract multimodal features from the dress image and description, as well as embedded features of user A's browsing behavior. These features are then fused to generate a multimodal behavior sequence feature. Finally, based on this fused feature, the system determines whether the dress matches user A's interests and decides whether to recommend it.

[0063] In one embodiment of the present disclosure, before using a multimodal encoder to extract embedded features of each action and multimodal features of the image and explanatory text of the target object, the method also includes: obtaining training data, wherein the training data includes multiple training objects and images and explanatory texts of each training object; using a multimodal encoder to extract multimodal features of the image and explanatory text of each training object, wherein the multimodal features corresponding to each training object include image features of the image of the training object and text features of the explanatory text, and the images and explanatory texts of each training object in the multimodal encoder share a multi-head attention module; based on the image features and text features corresponding to each training object, using a contrast loss function to calculate the contrast loss; and optimizing the parameters of the multimodal encoder based on the contrast loss to fine-tune the multimodal encoder.

[0064] The multi-head attention module is a key component in the multimodal encoder, used to parallelize the correlation between features in different representation subspaces. It uses multiple attention heads to interactively model image and text features to enhance the fusion of multimodal information.

[0065] The shared multi-head attention module refers to the use of the same multi-head attention structure parameters for interactive modeling when processing the image features and explanatory text features of the training object, thereby ensuring that the image and text are aligned in a unified semantic space. It should be noted that the multi-head attention modules for processing image features and text features in the prior art are separate. This embodiment connects both the image and text after tokenization and directly inputs them into the same attention module, which allows the two different types of inputs (image and text) to be compared and associated in a unified semantic space, thereby ensuring a more accurate matching relationship between the elements in the image and the text describing them; it can capture the complex connection between the image and the text, thereby generating richer fusion features.

[0066] The contrastive loss function is an objective function used to measure the similarity between features and encourage positive samples to be close and negative samples to be far away. It is often used in contrastive learning tasks.

[0067] In this embodiment, a multimodal encoder is used to extract the image features and text features of each training object, and inside the encoder, the features of the two modalities interact through a shared multi-head attention module to promote their alignment in the representation space. Then, based on these image features and text features, a contrast loss function is used to calculate the contrast loss. The parameters of the multimodal encoder are optimized based on the loss value to complete the fine-tuning of the multimodal encoder. Through the above technical means, the parameter update of the multimodal encoder is guided to make the semantic relationship between the image and the text clearer, and improve the accuracy in understanding the association between user behavior and object features.

[0068] Figure 3 A flowchart of a multimodal encoder fine-tuning method according to an embodiment of the present disclosure is shown. Figure 3 As shown, the following steps are included:

[0069] S301, obtaining training data, wherein the training data includes a plurality of training objects and an image and description text of each training object;

[0070] S302 , executing the following loop to fine-tune the multimodal encoder for multiple batches: determining whether m is equal to n, where m is the sequence number of the current batch, n is the number of batches, n is a positive integer, m is an integer not greater than n, and m is initially 0;

[0071] S303, if equal, it is determined that the fine-tuning of the multimodal encoder for multiple batches is completed, and the loop is exited;

[0072] S304: If it is less than , extracting multimodal features of the image and the explanatory text of each training object using a multimodal encoder, wherein the multimodal features corresponding to each training object include image features of the image of the training object and text features of the explanatory text;

[0073] S305, using a momentum model to determine positive samples and negative samples of each image feature from the multiple text features, and to determine positive samples and negative samples of each text feature from the multiple image features;

[0074] S306, calculating a contrastive loss using a contrastive loss function based on each image feature and its positive and negative samples, each text feature and its positive and negative samples;

[0075] S307 , optimizing the parameters of the multimodal encoder based on the contrast loss, optimizing the parameters of the momentum model using the multimodal encoder, and updating m with a value of m plus 1.

[0076] The momentum model and the multimodal encoder are models of the same architecture. In this embodiment, the multimodal encoder is the model being trained, and the momentum model is an auxiliary model that is slowly updated during the training process to generate stable feature representations and help select positive samples that match various image features or various text features and negative samples that do not match from a large number of samples.

[0077] Each time the momentum model is used to determine the positive and negative samples for each image feature from multiple text features, and each time the momentum model is used to determine the positive and negative samples for each text feature from multiple image features, it can be understood as building a memory bank. The memory bank is used to store the positive and negative samples for each image feature and each text feature.

[0078] Multiple text features and multiple image features are input into the momentum model. The momentum model outputs positive and negative samples for each image feature. Each positive or negative sample of an image feature is a text feature. It also outputs positive and negative samples for each text feature. Each positive or negative sample of a text feature is an image feature. It should be noted that if an image feature has only one positive sample, all text features other than the positive sample are negative samples of the image feature. Similarly, if a text feature has only one positive sample, all image features other than the positive sample are negative samples of the text feature. This technical approach can address the problem of scarce negative samples.

[0079] Positive samples refer to samples that are semantically consistent or highly relevant to the current feature and are used for positive example matching in contrastive learning. Negative samples refer to samples that are semantically irrelevant or significantly different from the current feature and are used for negative example distinction in contrastive learning.

[0080] In this embodiment, a loop is executed to fine-tune the multimodal encoder for multiple batches. In each iteration, it is first determined whether the current batch number m is equal to the total batch number n; if they are equal, the fine-tuning is completed and the loop is exited; otherwise, the multimodal encoder is used to extract the image features and text features of each training object in the current batch. Then, the momentum model is used to select positive samples and negative samples for each image feature from multiple text features, and positive samples and negative samples for each text feature from multiple image features. Based on these samples, the contrast loss function is used to calculate the contrast loss, and the parameters of the multimodal encoder are optimized accordingly, and the parameters of the momentum model are updated at the same time. Finally, m is increased by 1 and the next iteration is entered. Through the above technical means, the multimodal encoder can more accurately align the representation space of images and texts, thereby enhancing the ability to distinguish when processing image objects with high similarity.

[0081] In one embodiment of the present disclosure, the contrast loss L is calculated according to the following contrast loss function: ModalNCE :

[0082]

[0083] Among them, I j is the jth image feature, T j+ is the positive sample of the jth image feature, T i is the feature of the i-th text, except T j+ All text features other than the jth image feature are negative samples, T j is the jth text feature, I j+ is the positive sample of the jth text feature, I i is the i-th image feature, except I j+All image features other than are negative samples of the jth text feature, K is the number of training objects, K is greater than 0, j and i are both between 0 and K, τ is the temperature coefficient, τ is between 0 and 1.

[0084] This contrastive loss function guides the parameter update direction of the multimodal encoder, making the semantic relationship between image and text clearer.

[0085] In one embodiment of the present disclosure, before optimizing the parameters of the multimodal encoder based on the contrast loss, the method further includes: calculating the contrast loss using a contrast loss function based on each image feature and the positive and negative samples of each image feature; or calculating the contrast loss using a contrast loss function based on each text feature and the positive and negative samples of each text feature.

[0086] In one embodiment of the present disclosure, the contrast loss L is calculated according to the following contrast loss function: ModalNCE :

[0087]

[0088] This contrastive loss function guides the parameter update direction of the multimodal encoder, making the semantic relationship between image and text clearer.

[0089] In one embodiment of the present disclosure, the contrast loss L is calculated according to the following contrast loss function: ModalNCE :

[0090]

[0091] This contrastive loss function guides the parameter update direction of the multimodal encoder, making the semantic relationship between image and text clearer.

[0092] In one embodiment of the present disclosure, the parameters of the momentum model are optimized according to the following formula, including:

[0093] θ t =mθ t-1 +(1-m)θ q ;

[0094] Among them, θ t is the parameter of the current batch of momentum model, θ t-1 is the parameter of the momentum model in the previous batch, θ q is the parameter of the current batch of the multimodal encoder, m is the preset threshold, and m is between 0 and 1.

[0095] This contrastive loss function guides the parameter update direction of the multimodal encoder, making the semantic relationship between image and text clearer.

[0096] Figure 4A flowchart of a multimodal encoder pre-training method according to an embodiment of the present disclosure is shown. Figure 4 As shown, the following steps are included:

[0097] S401: Tokenize the image and text corresponding to each training object to obtain image tokens and text tokens;

[0098] S402, performing random masking operations on both the image token and the text token;

[0099] S403: Input the image token and text token after the random mask operation into the multimodal encoder, and output the prediction result corresponding to the image token and the prediction result corresponding to the text token;

[0100] S404, using a cross entropy loss function to calculate the cross entropy loss between the image token and its corresponding prediction result, and the cross entropy loss between the text token and its corresponding prediction result;

[0101] S405 , optimizing parameters of the multimodal encoder according to the cross entropy loss to pre-train the multimodal encoder.

[0102] The cross-entropy loss function is a commonly used objective function for classification tasks, measuring the difference between the model's predicted distribution and the true label distribution. In this embodiment, this function is used to supervise the multimodal encoder to learn the semantic consistency of image and text features.

[0103] Pre-training is a training phase performed before fine-tuning on a specific task. Its purpose is to enable the model to first learn general representation capabilities. In this embodiment, the pre-training phase uses a cross-entropy loss function to optimize the parameters of the multimodal encoder, enabling it to achieve preliminary image-text semantic alignment capabilities.

[0104] In this embodiment, for each training sample, the image and text are tokenized (such as dividing the image into patch tokens and the text into word / subword tokens); the image tokens and text tokens are concatenated into a unified input sequence as the input of the multimodal encoder. In the constructed unified input sequence, some tokens are randomly masked: the text tokens can be replaced with the [MASK] tag; the image tokens can be set to a zero vector or a dedicated mask embedding. The mask ratio can be set to a fixed value (such as 15%), and the mask ratios of the image and text tokens can be kept consistent. The unified input sequence containing the mask is fed into the multimodal encoder; a shared multi-head attention structure is used inside the encoder to achieve cross-modal interaction modeling between image tokens and text tokens; and the context-aware representation of each token is output. For the masked position, the model predicts its original content (for text, it predicts vocabulary; for image, it predicts patch category or pixel distribution); the cross-entropy loss function is used to calculate the cross-entropy loss between the image token and its corresponding prediction result, and the cross-entropy loss between the text token and its corresponding prediction result; the losses of the image and text are weighted and summed to form the total loss. Based on the cross-entropy loss, the parameters of the multimodal encoder are optimized through back propagation; after multiple rounds of iterations, the model gradually improves its ability to understand image and text information. Through the above technical means, the model's initial modeling ability of the semantic relationship between images and text is improved, so that when processing highly similar objects, it can more effectively capture subtle semantic differences and enhance the performance of the multimodal encoder.

[0105] In one optional embodiment, during the pre-training phase, multiple triplets are generated based on the training data, each of which includes a positive image-text pair and a negative image or text pair. A triplet loss function is used to calculate the triplet loss based on the image features and text features of the images within each triplet. The parameters of the multimodal encoder are optimized based on the triplet loss to pre-train the multimodal encoder. This technical approach improves the model's ability to discriminate between objects with similar semantics but different actual categories, enhancing the model's robustness and accuracy.

[0106] In an optional embodiment, training data is obtained, wherein the training data includes multiple training objects and images and explanatory texts of each training object; a multimodal encoder is pre-trained: the multimodal encoder is used to extract multimodal features of the image and explanatory text of each training object, and the cross-entropy loss function is used to calculate the cross-entropy loss based on the image features and text features corresponding to each training object, and the parameters of the multimodal encoder are optimized based on the cross-entropy loss to pre-train the multimodal encoder; the multimodal encoder is fine-tuned: the pre-trained multimodal encoder is used to extract multimodal features of the image and explanatory text of each training object, and the contrast loss function is used to calculate the contrast loss based on the image features and text features corresponding to each training object, and the parameters of the multimodal encoder are optimized based on the contrast loss to fine-tune the multimodal encoder.

[0107] In an optional embodiment, there are multiple target objects. The target objects recommended to the user can be determined through four stages: recall, rough sorting, fine sorting, and re-sorting. Each stage has its own specific goals and processing logic. The following is a detailed description of these four stages:

[0108] The task of the recall phase is to filter out a small set of potentially interesting items from a vast product library for subsequent sorting. Based on the user's multimodal behavior sequence features (i.e., features that combine the user's historical behavior and preferences) and the target object's multimodal features (image and text features), efficient indexing or retrieval mechanisms are used to quickly identify items that match the user's interests. For example, vector similarity search techniques can be used to calculate the similarity score between the user's behavior sequence features and the multimodal features of the items, and the top N items with the highest scores are selected as the candidate set.

[0109] The rough ranking stage aims to preliminarily rank the candidate items obtained in the recall stage, reducing the candidate set size while retaining high-quality items. This stage considers more factors, such as the basic attributes of the items and their real-time nature, but the computational complexity is relatively low. For each candidate item, a dot product or other interactive calculation is performed using its multimodal features and the user's behavioral sequence features to generate a preliminary score. The candidate items are then ranked based on their scores, and the top M items are selected to proceed to the next stage.

[0110] The refined ranking stage is one of the most critical parts of the entire recommendation process. It determines the final list of products presented to the user by conducting a more detailed evaluation of the rough-ranked products. This stage utilizes not only multimodal features and behavioral sequence features, but also additional information such as the user's immediate context (time, location, etc.) and product dynamics (inventory status, promotions, etc.). Specifically, a Deep Interest Network (DIN) is used to extract cross-features between user behavior sequences and the current product's multimodal features. Based on these features, a more complex model is then constructed to predict the user's interest in each product, providing an accurate ranking result.

[0111] The re-ranking phase is responsible for final adjustments to ensure that the recommendation list meets business rules or user experience requirements. For example, to increase diversity and novelty, avoid continuously recommending the same type of products; or re-organize the order of products according to a certain strategy (such as price from low to high). Although this stage is mainly based on the interest scores obtained in the previous stage, some manually set rules are also added for fine-tuning. For example, the position of products can be adjusted by defining a series of constraints to ensure that the recommendation results meet both personalized needs and user experience.

[0112] In an optional embodiment, the recall stage and the rough sorting stage are performed only based on the multimodal features of the target object, and the fine sorting stage and the re-ranking stage are performed based on the multimodal behavior sequence features of the behavior sequence and the multimodal features of the target object.

[0113] Figure 5 A schematic diagram of another object recommendation system according to an embodiment of the present disclosure is shown. Figure 5 As shown, it includes: a multimodal encoder 501, a similarity network 502 and a recommendation network 503.

[0114] The multimodal encoder 501 extracts embedded features of each action. The multimodal encoder extracts multimodal features of the image of the target object and the explanatory text.

[0115] The similarity network 502 fuses the embedded features of each action based on the multimodal features to obtain the multimodal behavior sequence features.

[0116] The recommendation network 503 performs recall, rough sorting, fine sorting, and re-ranking based on the multimodal behavior sequence features and the multimodal features, and ultimately determines whether to recommend the target object to the user.

[0117] Figure 6 A schematic diagram of a feature processing method in an embodiment of the present disclosure is shown. Figure 6 Shown, including:

[0118] embedding 1, embedding 2, embedding 3, and embedding 4 are the embedding features of an action respectively.

[0119] Perform dot product operations on the multimodal features and the embedded features of each action to obtain the weights of the embedded features of each action.

[0120] Perform a dot product operation on the multimodal feature and embedding 1 to obtain the weight of embedding 1, perform a dot product operation on the multimodal feature and embedding 2 to obtain the weight of embedding 2, perform a dot product operation on the multimodal feature and embedding 3 to obtain the weight of embedding 3, and perform a dot product operation on the multimodal feature and embedding 4 to obtain the weight of embedding 4.

[0121] The embedded features of each action are weighted and summed according to the weight of the embedded features of each action to obtain the multimodal behavior sequence features.

[0122] The product of embedding 1 and its weight is added together with the product of embedding 2 and its weight, plus the product of embedding 3 and its weight, plus the product of embedding 4 and its weight.

[0123] The multimodal behavior sequence features can also be subjected to mean pooling processing to update the modal behavior sequence features.

[0124] Figure 7 A schematic diagram of an object recommendation device according to an embodiment of the present disclosure is shown. Figure 7 As shown, the object recommendation device may include:

[0125] An acquisition module 701 is configured to acquire a user's behavior sequence and an image and description text of a target object, wherein the behavior sequence includes one or more actions;

[0126] an extraction module 702 configured to extract embedded features of each action and multimodal features of the target object's image and caption text using a multimodal encoder;

[0127] A fusion module 703 is configured to fuse the embedded features of each action based on the multimodal features to obtain a multimodal behavior sequence feature;

[0128] The recommendation module 704 is configured to determine whether to recommend the target object to the user based on the multimodal behavior sequence feature and the multimodal feature.

[0129] In some embodiments, the extraction module 702 is further configured to obtain training data, wherein the training data includes multiple training objects and images and explanatory texts of each training object; use a multimodal encoder to extract multimodal features of the images and explanatory texts of each training object, wherein the multimodal features corresponding to each training object include image features of the image of the training object and text features of the explanatory text, and the images and explanatory texts of each training object in the multimodal encoder share a multi-head attention module; based on the image features and text features corresponding to each training object, use a contrast loss function to calculate the contrast loss; optimize the parameters of the multimodal encoder based on the contrast loss to fine-tune the multimodal encoder.

[0130] In some embodiments, the extraction module 702 is further configured to obtain training data, wherein the training data includes multiple training objects and images and explanatory texts of each training object; execute the following loop to fine-tune the multimodal encoder for multiple batches: determine whether m is equal to n, where m is the sequence number of the current batch, n is the number of multiple batches, n is a positive integer, m is an integer not greater than n, and the initial value of m is 0; if equal, determine that the fine-tuning of the multimodal encoder for multiple batches is completed and exit the loop; if less than, use the multimodal encoder to extract multimodal features of the images and explanatory texts of each training object, Among them, the multimodal features corresponding to each training object include the image features of the training object and the text features of the explanatory text; the momentum model is used to determine the positive samples and negative samples of each image feature from multiple text features, and the positive samples and negative samples of each text feature are determined from multiple image features; based on each image feature and the positive samples and negative samples of each image feature, each text feature and the positive samples and negative samples of each text feature, the contrast loss is calculated using the contrast loss function; the parameters of the multimodal encoder are optimized according to the contrast loss, the parameters of the momentum model are optimized using the multimodal encoder, and m is updated with a value of m plus 1.

[0131] In some embodiments, the extraction module 702 is further configured to calculate the contrast loss L according to the following contrast loss function: ModalNCE :

[0132]

[0133] Among them, I j is the jth image feature, T j+ is the positive sample of the jth image feature, T i is the feature of the i-th text, except T j+ All text features other than the jth image feature are negative samples, T j is the jth text feature, I j+ is the positive sample of the jth text feature, I iis the i-th image feature, except I j+ All image features other than are negative samples of the jth text feature, K is the number of training objects, K is greater than 0, j and i are both between 0 and K, τ is the temperature coefficient, τ is between 0 and 1.

[0134] In some embodiments, the extraction module 702 is further configured to calculate the contrast loss using a contrast loss function based on each image feature and the positive and negative samples of each image feature; or to calculate the contrast loss using a contrast loss function based on each text feature and the positive and negative samples of each text feature.

[0135] In some embodiments, the extraction module 702 is further configured to calculate the contrast loss L according to the following contrast loss function: ModalNCE :

[0136]

[0137] In some embodiments, the extraction module 702 is further configured to calculate the contrast loss L according to the following contrast loss function: ModalNCE :

[0138]

[0139] In some embodiments, the extraction module 702 is further configured to optimize the parameters of the momentum model according to the following formula, including:

[0140] θ t =mθ t-1 +(1-m)θ q ;

[0141] Among them, θ t is the parameter of the current batch of momentum model, θ t-1 is the parameter of the momentum model in the previous batch, θ q is the parameter of the current batch of the multimodal encoder, m is the preset threshold, and m is between 0 and 1.

[0142] In some embodiments, the extraction module 702 is further configured to tokenize the images and texts corresponding to each training object to obtain image tokens and text tokens; perform random masking operations on the image tokens and text tokens; input the image tokens and text tokens after the random masking operations into the multimodal encoder, and output the prediction results corresponding to the image tokens and the prediction results corresponding to the text tokens; use the cross-entropy loss function to calculate the cross-entropy loss between the image token and its corresponding prediction result, as well as the cross-entropy loss between the text token and its corresponding prediction result; optimize the parameters of the multimodal encoder based on the cross-entropy loss to pre-train the multimodal encoder.

[0143] In some embodiments, the extraction module 702 is further configured to generate multiple triplets based on the training data during the pre-training phase, each triplet comprising a positive sample image-text pair and a negative sample image or text; calculate a triplet loss using a triplet loss function based on the image features of the image and text features of the text within each triplet; and optimize the parameters of the multimodal encoder based on the triplet loss to pre-train the multimodal encoder. Through these technical measures, the model's ability to discriminate between objects with similar semantics but different actual categories is improved, thereby enhancing the model's robustness and accuracy.

[0144] In some embodiments, the extraction module 702 is further configured to obtain training data, wherein the training data includes multiple training objects and images and explanatory texts of each training object; pre-train the multimodal encoder: use the multimodal encoder to extract multimodal features of the images and explanatory texts of each training object, calculate the cross-entropy loss using a cross-entropy loss function based on the image features and text features corresponding to each training object, and optimize the parameters of the multimodal encoder based on the cross-entropy loss to pre-train the multimodal encoder; fine-tune the multimodal encoder: use the pre-trained multimodal encoder to extract multimodal features of the images and explanatory texts of each training object, calculate the contrast loss using a contrast loss function based on the image features and text features corresponding to each training object, and optimize the parameters of the multimodal encoder based on the contrast loss to fine-tune the multimodal encoder.

[0145] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0146] Refer to the following Figure 8 800 according to this embodiment of the present disclosure will be described. Figure 8 The electronic device 800 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0147] like Figure 8 As shown, electronic device 800 is implemented as a general-purpose computing device. Components of electronic device 800 may include, but are not limited to, the aforementioned at least one processing unit 810, the aforementioned at least one storage unit 820, and a bus 830 connecting various system components (including storage unit 820 and processing unit 810).

[0148] The storage unit stores program code, and the program code can be executed by the processing unit 810, so that the processing unit 810 performs the steps described in the "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 810 can perform the following steps of the above method embodiment: obtaining the user's behavior sequence and the image and description text of the target object, wherein the behavior sequence includes one or more actions; using a multimodal encoder to extract the embedded features of each action and the multimodal features of the image and description text of the target object; based on the multimodal features, the embedded features of each action are fused to obtain the multimodal behavior sequence features; and based on the multimodal behavior sequence features and the multimodal features, determining whether to recommend the target object to the user.

[0149] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0150] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0151] Bus 830 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0152] The electronic device 800 may also communicate with one or more external devices 840 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more object recommendations that enable a user to interact with the electronic device 800, and / or any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication may occur via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0153] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0154] In the disclosed exemplary embodiments, a computer-readable storage medium is also provided. The computer-readable storage medium may be a readable signal medium or a readable storage medium.

[0155] In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes program code. When the program product is run on a user device, the program code is used to enable the user device to execute the steps of various exemplary implementations of the present disclosure described in the above "Specific Implementation" section of this specification.

[0156] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0157] In the present disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0158] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0159] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0160] The present disclosure provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the object recommendation method provided in any of the various optional embodiments of the present disclosure.

[0161] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0162] Furthermore, although the steps of the method of the present disclosure are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in this particular order, or that all steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0163] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0164] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope of the present disclosure being indicated by the appended claims.

Claims

1. An object recommendation method, characterized in that: include: Acquire a user's behavior sequence and an image and description text of a target object, wherein the behavior sequence includes one or more actions; extracting embedded features of each action and multimodal features of the image and caption of the target object using a multimodal encoder; Based on the multimodal features, the embedded features of each action are fused to obtain a multimodal behavior sequence feature; Determining whether to recommend the target object to the user is based on the multimodal behavior sequence feature and the multimodal feature.

2. The method according to claim 1, characterized in that Before extracting the embedded features of each action and the multimodal features of the image and the description text of the target object using the multimodal encoder, the method further includes: Acquiring training data, wherein the training data includes a plurality of training objects and an image and description text of each training object; Extracting multimodal features of the image and explanatory text of each training object using the multimodal encoder, wherein the multimodal features corresponding to each training object include image features of the image of the training object and text features of the explanatory text, and the image and explanatory text of each training object in the multimodal encoder share a multi-head attention module; Based on the image features and text features corresponding to each training object, the contrast loss is calculated using the contrast loss function; Parameters of the multimodal encoder are optimized according to the contrastive loss to fine-tune the multimodal encoder.

3. The method according to claim 1, characterized in that Before extracting the embedded features of each action and the multimodal features of the image and the description text of the target object using the multimodal encoder, the method further includes: Acquiring training data, wherein the training data includes a plurality of training objects and an image and description text of each training object; The following loop is executed to fine-tune the multimodal encoder over multiple batches: Determine whether m is equal to n, where m is the serial number of the current batch, n is the number of batches, n is a positive integer, m is an integer not greater than n, and the initial value of m is 0; If yes, it is determined that fine-tuning of the multimodal encoder for multiple batches is completed, and the loop is exited; If it is less than, extracting multimodal features of the image and explanatory text of each training object using the multimodal encoder, wherein the multimodal features corresponding to each training object include image features of the image of the training object and text features of the explanatory text; Using the momentum model to determine the positive and negative samples of each image feature from multiple text features, and to determine the positive and negative samples of each text feature from multiple image features; Based on each image feature and its positive and negative samples, each text feature and its positive and negative samples, a contrast loss function is used to calculate the contrast loss. Optimizing parameters of the multimodal encoder according to the contrastive loss, optimizing parameters of the momentum model using the multimodal encoder, and updating m with a value of m plus 1.

4. The method according to claim 3, characterized in that The contrast loss L is calculated according to the following contrast loss function ModalNCE : Among them, I j is the jth image feature, T j+ is the positive sample of the jth image feature, T i is the feature of the i-th text, except T j+ All text features other than the jth image feature are negative samples, T j is the jth text feature, I j+ is the positive sample of the jth text feature, I i is the i-th image feature, except I j+ All image features other than are negative samples of the jth text feature, K is the number of training objects, K is greater than 0, j and i are both between 0 and K, τ is the temperature coefficient, τ is between 0 and 1.

5. The method according to claim 2, characterized in that Before extracting the multimodal features of the image and the explanatory text of each training object using the multimodal encoder, the method further includes: The images and texts corresponding to each training object are tokenized to obtain image tokens and text tokens; Performing a random masking operation on both the image token and the text token; Inputting the image token and text token after the random mask operation into the multimodal encoder, and outputting the prediction result corresponding to the image token and the prediction result corresponding to the text token; Calculating the cross entropy loss between the image token and its corresponding prediction result, and the cross entropy loss between the text token and its corresponding prediction result using a cross entropy loss function; Parameters of the multimodal encoder are optimized according to the cross entropy loss to pre-train the multimodal encoder.

6. The method according to claim 3, characterized in that Before optimizing the parameters of the multimodal encoder according to the contrast loss, the method further includes: Calculating the contrast loss using the contrast loss function based on each image feature and the positive samples and negative samples of each image feature; or The contrast loss is calculated using the contrast loss function based on each text feature and the positive samples and negative samples of each text feature.

7. The method according to claim 3, characterized in that The parameters of the momentum model are optimized according to the following formula, including: i t =mθ t-1 +(1-m)θ q ; Among them, θ t is the parameter of the current batch of the momentum model, θ t-1 is the parameter of the momentum model in the previous batch, θ q is the parameter of the current batch of the multimodal encoder, m is a preset threshold, and m is between 0 and 1.

8. The method according to claim 1, characterized in that The method of fusing the embedded features of each action based on the multimodal features to obtain the multimodal behavior sequence features includes: Generating weights of embedded features of each action based on the multimodal features; The embedded features of each action are fused according to the weights of the embedded features of each action to obtain the multimodal behavior sequence feature.

9. The method according to claim 8, characterized in that Generating the weight of the embedded features of each action based on the multimodal features includes: A dot product operation is performed on the multimodal features and the embedded features of each action to obtain the weight of the embedded features of each action.

10. An object recommendation device, characterized in that: include: an acquisition module configured to acquire a user's behavior sequence and an image and description text of a target object, wherein the behavior sequence includes one or more actions; an extraction module configured to extract, using a multimodal encoder, embedded features of each action and multimodal features of the image and description text of the target object; a fusion module configured to fuse the embedded features of each action based on the multimodal features to obtain a multimodal behavior sequence feature; A recommendation module is configured to determine whether to recommend the target object to the user based on the multimodal behavior sequence feature and the multimodal feature.

Images