Information processing method and device, information recommendation method and device, electronic equipment, storage medium and program product

Abstract

The embodiment of the application provides an information processing method and device, an information recommendation method and device, an electronic equipment, a storage medium and a program product, relates to the field of artificial intelligence, and comprises the following steps: obtaining a first training sample, wherein the first training sample comprises user historical behavior data, a first candidate object set and a target object index; training a preset model decoder based on the first training sample and model parameters of the preset model decoder, obtaining user behavior prediction information, wherein the user behavior prediction information comprises an index prediction value; the model parameters are preconfigured with a first mapping relationship between an object identifier and a multi-modal vector; adjusting the model parameters based on the user behavior prediction information and the target object index, continuing the training until a first preset convergence condition is reached, and obtaining a second mapping relationship between an adjusted object identifier and a multi-modal vector from the model parameters after the training is completed. The application can improve the logical reasoning capability of a recommendation system and improve the accuracy of information recommendation.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an information processing method and apparatus, an information recommendation method and apparatus, an electronic device, a storage medium, and a program product. Background Technology

[0002] In the recommendation system architecture of related technologies, the first stage typically involves pre-mapping unstructured data such as images and text of objects into a unified vector space to generate multimodal vectors for the objects. The second stage involves retrieving a candidate object set based on the user's query terms and obtaining the multimodal vectors of each object in the candidate set. Based on these multimodal vectors, the objects in the candidate set are ranked, and recommendation information is generated based on the ranked candidate object set.

[0003] However, multimodal vectors lack the ability to perceive user intent. Furthermore, the goal of generating multimodal vectors in the first stage is to maintain semantic consistency between information from different modalities, while the goal of the second stage ranking is to rank candidates according to the probability of the user performing an action on the candidate object. The inconsistent processing goals of the first and second stages lead to low accuracy in ranking based on multimodal vectors. Summary of the Invention

[0004] This application provides an information processing method and apparatus, an information recommendation method and apparatus, an electronic device, a storage medium, and a program product to alleviate or solve one or more technical problems existing in the prior art.

[0005] In a first aspect, embodiments of this application provide an information processing method, including: Obtain a first training sample, which includes user historical behavior data, a first candidate object set, and a target object index; the first candidate object set includes first candidate objects recommended based on the user's historical query terms; the target object index is the index of the target object operated on by the user in the first candidate object set. Based on the first training samples and the model parameters of the preset model decoder, the preset model decoder is trained to obtain user behavior prediction information. The user behavior prediction information includes an index prediction value, which refers to the index of the first candidate object predicted to be operated by the user in the first candidate object set. The model parameters are pre-configured with a first mapping relationship between object identifiers and multimodal vectors. The multimodal vectors are generated by fusing the image and text of the corresponding object. Based on the user behavior prediction information and the target object index, the model parameters are adjusted, and training continues until the first preset convergence condition is met. The second mapping relationship between the adjusted object identifier and the multimodal vector is obtained from the model parameters after training.

[0006] In some embodiments of this application, the user historical behavior data includes the user's spatiotemporal context and a sequence of historical objects; the sequence of historical objects includes objects that the user has previously operated on. The step of training the preset model decoder based on the first training samples and the model parameters of the preset model decoder to obtain user behavior prediction information includes: Based on the spatiotemporal context, the historical object sequence, the first candidate object set, and the first mapping relationship, a fusion prompt vector set is generated; the fusion prompt vector set is used to represent the content features and user behavior features of objects in the vector space. The fused prompt vector set is input into the preset model decoder, and the user behavior prediction information is output.

[0007] In some embodiments of this application, generating a fusion prompt vector set based on the spatiotemporal context, the historical object sequence, the first candidate object set, and the first mapping relationship includes: For each object that the user has operated on in the historical object sequence, the multimodal vector of each object operated on by the user is obtained from the first mapping relationship; For each first candidate object in the first candidate object set, the multimodal vector of each first candidate object is obtained from the first mapping relationship; The spatiotemporal context is encoded into a context vector; The context vector, the multimodal vectors of each object operated by the user, and the multimodal vectors of each first candidate object are combined into the fusion prompt vector set.

[0008] In some embodiments of this application, the step of inputting the fused cue vector set into the preset model decoder and outputting the user behavior prediction information includes: The fused prompt vector set is input into the preset model decoder. The preset model decoder calculates the predicted probability of each first candidate object in the first candidate object set being operated by the user. The predicted probability is calculated based on the hidden state vector and the index embedding matrix. The hidden state vector is used to predict the index prediction value. The index embedding matrix is ​​adjusted during training based on the target object index and the index prediction value. Based on the predicted probability corresponding to each first candidate object, the index of the first candidate object with the highest predicted probability in the first candidate object set is used as the index prediction value. The user behavior prediction information is output by the preset model decoder, and the user behavior prediction information includes the index prediction value.

[0009] In some embodiments of this application, the user behavior prediction information further includes reasoning process information, which is used to explain the reason why the first candidate object corresponding to the index prediction value will be operated by the user based on the user's historical behavior data. Before using the index of the first candidate object with the highest predicted probability in the first candidate object set as the index prediction value based on the predicted probabilities corresponding to each of the first candidate objects, the method further includes: The inference process information is generated by the preset model decoder.

[0010] In some embodiments of this application, adjusting the model parameters based on the user behavior prediction information and the target object index includes: Based on the index prediction value and the target object index, a reward signal is determined for reinforcement learning; the reward signal is used to quantify the degree of consistency between the index prediction value and the target object index. Based on the reward signal, calculate the advantage estimate corresponding to the index prediction value; Based on the advantage estimate, calculate the gradient of the model parameters; The model parameters are adjusted using the gradient.

[0011] In some embodiments of this application, before training the preset model decoder based on the first training samples and the model parameters of the preset model decoder, the method further includes: The first training sample is input into a preset teacher model, and the reasoning reason corresponding to the first training sample is output. The reasoning reason is obtained by reasoning based on the user's historical behavior data and the target object index. The reasoning is added to the first training sample to obtain the updated first training sample.

[0012] In some embodiments of this application, before training the preset model decoder based on the first training samples and the model parameters of the preset model decoder to obtain user behavior prediction information, the method further includes: Retrieve the object identifier, object text, and object image for each object; Based on the object text and object image of each object, a pre-trained image-text alignment model is used to generate multimodal vectors corresponding to each object. The first mapping relationship is obtained by correspondingly storing the object identifiers of each object and the multimodal vectors of each object; Configure the first mapping relationship into the model parameters of the preset model decoder.

[0013] In some embodiments of this application, the training process of the image-text alignment model includes: Obtain a second training sample, which includes positive samples and negative samples. The positive samples include a first query term corresponding to a first sample object, a first object image of the first sample object, and first object text that semantically matches the first object image. The negative samples include multimodal data that does not match the second sample object or the second query term corresponding to the second sample object. Based on the second training sample, the query word vector, object text vector, object image vector and image-text fusion vector corresponding to the second training sample are generated through the image-text alignment model to be trained; Based on the query word vector, the object text vector, the object image vector, and the image-text fusion vector, calculate the contrast loss function value; Based on the contrast loss function value, the model parameters of the image-text alignment model to be trained are adjusted, and training continues until the second preset convergence condition is reached, thus obtaining the trained image-text alignment model.

[0014] In some embodiments of this application, calculating the contrast loss function value based on the query word vector, the object text vector, the object image vector, and the image-text fusion vector includes: Based on the query word vector and the object text vector, calculate the first contrast loss value between the query word and the object text in the second training sample; Based on the query word vector and the object image vector, calculate the second contrast loss value between the query word and the object image in the second training sample; Based on the query word vector and the image-text fusion vector, calculate the third contrast loss value; The first contrast loss value, the second contrast loss value, and the third contrast loss value are weighted and summed to obtain the contrast loss function value.

[0015] In some embodiments of this application, obtaining the positive sample includes: From the query terms used to query the first sample object, obtain the first preset number of query terms that the user uses most frequently; One query term is selected from the first preset number of query terms and used as the first query term corresponding to the first sample object; From the multiple resources corresponding to the first sample object, determine the second preset number of target resources that are operated on the most by the target user, wherein the target user is the user who uses the first query term to query the first sample object; The first image of the first sample object and the second images of the second preset number of target resources are both used as the first object image of the first sample object; The descriptive text corresponding to the first image and the descriptive text corresponding to the second image are both used as the first object text; The positive samples are obtained based on the first query term, the first object image, and the first object text.

[0016] In some embodiments of this application, after obtaining the adjusted second mapping relationship between the object identifier and the multimodal vector from the trained model parameters, the method further includes: Obtain a third training sample, which includes a sample ID interaction vector and a sample multimodal interaction vector of a third sample object; the sample multimodal interaction vector is obtained based on the second mapping relationship. Based on the sample ID interaction vector and the sample multimodal interaction vector, the first sample fusion vector of the third sample object is generated through the concatenation layer in the recommendation model to be trained. Based on the first sample fusion vector, the first probability that the third sample object will be operated by the user is predicted through the recommendation main module in the recommendation model to be trained; Based on the sample ID interaction vector and the sample multimodal interaction vector, the second probability of the third sample object being operated by the user is predicted through the auxiliary multilayer perceptron in the recommendation model to be trained. Calculate the target loss function value based on the first probability and the second probability of each of the third training samples in the current training batch; Based on the target loss function value, the model parameters of the recommendation model to be trained are adjusted, and training continues until the third preset convergence condition is met, thus obtaining the trained recommendation model.

[0017] In some embodiments of this application, the process of obtaining the sample multimodal interaction vector includes: Based on the user identifier of the sample user who has operated the third sample object, obtain the first object identifier of each object that the sample user has operated in the past. Based on each of the first object identifiers, obtain the multimodal vector corresponding to each of the first object identifiers from the second mapping relationship; Based on the second object identifier of the third sample object, the multimodal vector corresponding to the third sample object is obtained from the second mapping relationship; Based on the multimodal vectors corresponding to each of the first object identifiers and the multimodal vectors corresponding to the third sample object, a multimodal interaction vector corresponding to the third sample object is generated by using a preset multimodal feature interaction function.

[0018] In some embodiments of this application, predicting the second probability that the third sample object is operated by the user based on the sample ID interaction vector and the sample multimodal interaction vector, using the auxiliary multilayer perceptron in the recommendation model to be trained, includes: A gradient adjustment operation is performed on the sample ID interaction vector, the gradient adjustment operation being used to control the impact on the parameter updates of the sample ID interaction vector during backpropagation; Based on the sample ID interaction vector, a first fusion weight and a second fusion weight corresponding to the sample ID interaction vector and the sample multimodal interaction vector are determined by a gating network. Based on the first fusion weight and the second fusion weight, the sample ID interaction vector and the sample multimodal interaction vector after performing gradient adjustment operation are weighted and fused to obtain the second sample fusion vector corresponding to the third sample object; The second sample fusion vector is input into the auxiliary multilayer perceptron, which outputs the second probability that the third sample object is operated by the user.

[0019] In some embodiments of this application, calculating the target loss function value based on the first probability and the second probability of each of the third training samples in the current training batch includes: Based on the first probability of each of the third training samples in the current training batch and the ground truth label of each of the third training samples, the main loss value is calculated; the ground truth label is used to characterize whether the corresponding third sample object has been operated by the user. Based on the second probability of each of the third training samples in the current training batch and the ground truth label of each of the third training samples, calculate the auxiliary loss value; The target loss function value is obtained by weighted summation of the main loss value and the auxiliary loss value.

[0020] Secondly, embodiments of this application provide an information recommendation method, including: Based on the current user's current query term, obtain the second candidate object set corresponding to the current query term; Based on the current user's user identifier and the second candidate object set, obtain the ID interaction vector corresponding to each second candidate object in the second candidate object set; Based on the current user's user identifier, the second candidate object set, and the second mapping relationship, the multimodal interaction vectors corresponding to each second candidate object in the second candidate object set are obtained. The second mapping relationship is obtained from the model parameters of the trained preset model decoder. Before the preset model decoder is trained, the model parameters are pre-configured with a first mapping relationship between the object identifier and the multimodal vector. The model parameters are adjusted based on user behavior prediction information and target object index during the training process of the preset model decoder. The user behavior prediction information is output by the preset model decoder based on the user's historical behavior data and the first candidate object set. The target object index is the index of the target object to be operated in the first candidate object set. The user behavior prediction information includes an index prediction value, which refers to the index of the predicted first candidate object to be operated in the first candidate object set. Based on the ID interaction vector and the multimodal interaction vector corresponding to each of the second candidate objects, the probability of each of the second candidate objects being operated by the current user is predicted. Based on the operation probability corresponding to each of the second candidate objects, the second candidate objects are sorted to obtain object recommendation information, and the object recommendation information is sent to the current user.

[0021] In some embodiments of this application, obtaining the ID interaction vector corresponding to each second candidate object in the second candidate object set based on the user identifier of the current user and the second candidate object set includes: Based on the user identifier, obtain the user behavior sequence corresponding to the current user, wherein the user behavior sequence includes the third object identifier of the object that the current user has previously operated on; Obtain the fourth object identifier of the target candidate object, wherein the target candidate object is any second candidate object in the second candidate object set; Based on the user behavior sequence and the fourth object identifier, an ID interaction vector corresponding to the target candidate object is generated through a preset ID feature interaction function.

[0022] In some embodiments of this application, obtaining the multimodal interaction vector corresponding to each second candidate object in the second candidate object set based on the user identifier of the current user, the second candidate object set, and the second mapping relationship includes: Based on the user identifier of the current user, obtain the third object identifier of each object that the current user has previously operated on; Based on each of the third object identifiers, obtain the multimodal vector corresponding to each of the third object identifiers from the second mapping relationship; Based on the fourth object identifier of the target candidate object, the multimodal vector corresponding to the target candidate object is obtained from the second mapping relationship; the target candidate object is any second candidate object in the second candidate object set; Based on the multimodal vectors corresponding to each of the third object identifiers and the multimodal vectors corresponding to the target candidate objects, a multimodal interaction vector corresponding to the target candidate object is generated through a preset multimodal feature interaction function.

[0023] Thirdly, embodiments of this application provide an information recommendation method, including: Send the current user's current query term to the server; The system receives object recommendation information returned by the server based on the current query term, and the object recommendation information is obtained through the method described in the second aspect above.

[0024] Fourthly, embodiments of this application provide an information processing apparatus, including: The first acquisition module is used to acquire a first training sample, which includes user historical behavior data, a first candidate object set, and a target object index; the first candidate object set includes first candidate objects recommended based on the user's historical query terms; the target object index is the index of the target object operated by the user in the first candidate object set. The training module is used to train the preset model decoder based on the first training samples and the model parameters of the preset model decoder to obtain user behavior prediction information. The user behavior prediction information includes an index prediction value, which refers to the index of the first candidate object predicted to be operated by the user in the first candidate object set. The model parameters are pre-configured with a first mapping relationship between object identifiers and multimodal vectors. The multimodal vectors are generated by fusing the image and text of the corresponding object. The parameter adjustment module is used to adjust the model parameters based on the user behavior prediction information and the target object index, continue training until the first preset convergence condition is reached, and obtain the second mapping relationship between the adjusted object identifier and the multimodal vector from the model parameters after training.

[0025] Fifthly, embodiments of this application provide an information recommendation device, comprising: The second acquisition module is used to acquire a second candidate object set corresponding to the current query term based on the current user's current query term; acquire the ID interaction vector corresponding to each second candidate object in the second candidate object set based on the current user's user identifier and the second candidate object set; and acquire the multimodal interaction vector corresponding to each second candidate object in the second candidate object set based on the current user's user identifier, the second candidate object set, and the second mapping relationship. The second mapping relationship is obtained from the model parameters of the pre-trained preset model decoder. Before the preset model decoder is trained, the model parameters are pre-configured with a first mapping relationship between object identifiers and multimodal vectors. The model parameters are adjusted based on user behavior prediction information and target object index during the training process of the preset model decoder. The user behavior prediction information is output by the preset model decoder based on user historical behavior data and the first candidate object set. The target object index is the index of the target object to be operated in the first candidate object set. The user behavior prediction information includes an index prediction value, which refers to the index of the predicted first candidate object to be operated in the first candidate object set. The prediction module is used to predict the probability of the current user operating each of the second candidate objects based on the ID interaction vector and the multimodal interaction vector corresponding to each of the second candidate objects respectively. The sorting module is used to sort each of the second candidate objects based on the operation probability corresponding to each of the second candidate objects, so as to obtain object recommendation information. The sending module is used to send the object recommendation information to the current user.

[0026] In a sixth aspect, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described in any one of the first to third aspects of embodiments of this application.

[0027] In a seventh aspect, embodiments of this application provide a computer-readable storage medium storing a computer program, which is executed by a processor to implement the method described in any one of the first to third aspects of embodiments of this application.

[0028] Eighthly, embodiments of this application provide a computer program product, including a computer program executed by a processor to implement the method described in any one of the first to third aspects of embodiments of this application.

[0029] Based on the information processing method described in the first aspect above, this application has at least the following beneficial effects or advantages: In the embodiments of this application, the first mapping relationship between object identifiers and multimodal vectors is used as part of the model parameters of the preset model decoder. This allows for optimization and adjustment of the multimodal vectors of each object in the first mapping relationship during the training process of the preset model decoder. The first training samples of the preset model decoder include user historical behavior data, which contains a large amount of information about user behavior and / or user intent. The preset model decoder generates user behavior prediction information based on the first training samples including user historical behavior data. Then, based on the user behavior prediction information and the index of target objects actually operated by the user in the first training samples, the multimodal vectors in the first mapping relationship are adjusted. This allows the adjusted multimodal vectors to capture the correlation between different objects in the user historical behavior data caused by user behavior and / or user intent. The adjusted multimodal vectors contain richer user context-aware information, ensuring that the multimodal vectors that have learned user intent are consistent with the goal of subsequent ranking based on user intent. This improves the logical reasoning ability of the recommendation system and enhances the accuracy of information recommendation.

[0030] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0031] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0032] Figure 1 A flowchart illustrating an information processing method provided in an embodiment of this application is shown. Figure 2 A schematic diagram illustrating the principle of the preset model decoder provided in an embodiment of this application is shown; Figure 3 A schematic diagram of the sample triples provided in the embodiments of this application is shown; Figure 4 A schematic diagram illustrating the principle of the image-text alignment model provided in the embodiments of this application is shown; Figure 5 Figure (a) shows the distribution of each modality in text, image, and text + image before the image-text alignment model is trained; Figure 5 Figure (b) shows the distribution of each modality in text, image, and text + image after the image-text alignment model has been trained; Figure 6 This illustration shows the variation of the first fusion weight and the second fusion weight corresponding to the ID interaction vector and the multimodal interaction vector provided in the embodiments of this application in different object layers; Figure 7 A schematic diagram of the structure of the recommendation model provided in an embodiment of this application is shown; Figure 8 A schematic diagram illustrating the training principle of the recommendation model provided in an embodiment of this application is shown; Figure 9 A flowchart illustrating an information recommendation method provided in an embodiment of this application is shown. Figure 10 This illustration shows a schematic diagram of model training and deployment for the three-stage information recommendation scheme provided in an embodiment of this application; Figure 11 A flowchart illustrating the three-stage information recommendation scheme provided in an embodiment of this application is shown; Figure 12 A flowchart illustrating an information recommendation method provided in an embodiment of this application is shown. Figure 13 This paper shows a schematic diagram of the structure of an information processing device provided in an embodiment of this application; Figure 14 This illustration shows a structural schematic diagram of an information recommendation device provided in an embodiment of this application; Figure 15 A schematic diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0033] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0034] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0035] Information recommendation services are needed in many fields, such as food delivery, home delivery services, and advertising. In these fields, users may submit query terms, and the system needs to search for information on stores, services, or advertisements that match those terms, and then return the search results to the user.

[0036] In recommendation systems of related technologies, a two-stage information recommendation paradigm is typically employed. In the first stage, unstructured data such as images and text of objects are pre-mapped into a unified vector space, generating multimodal vectors for each object. Object identifiers and their corresponding multimodal vectors are then stored in a storage system. In the second stage, after retrieving a candidate object set based on the user's query terms, the multimodal vectors of each object in the candidate set are retrieved from the storage system based on their object identifiers. Then, based on these multimodal vectors, the objects in the candidate set are ranked, and recommendation information is generated from this ranked set.

[0037] However, the multimodal vectors generated in the first stage are static, content-oriented embedding vectors. Once generated and stored, they fail to reflect recent interaction data and lack the ability to perceive user intent. The second stage of ranking, however, needs to adapt to constantly changing user intent. Therefore, the multimodal vectors in related technologies contribute insufficiently to the ranking performance of the second stage, resulting in poor ranking accuracy. Furthermore, the goal of generating multimodal vectors in the first stage is to maintain semantic consistency between different modalities, while the goal of ranking in the second stage is to rank according to the probability of user actions on candidate objects. This inconsistency between the processing goals of the first and second stages leads to low accuracy in ranking based on multimodal vectors.

[0038] Moreover, in the related technologies, although the second-stage ranking process uses multimodal vectors of each object in the candidate object set, the ranking model used in the related technologies has a strong ID-based branch signal after long-term training. Even if multimodal vectors are introduced, the role of multimodal vectors is often suppressed by gating or multilayer perceptrons, resulting in multimodal vectors not playing a substantial role. This leads to poor accuracy of the final information recommendation and poor long-tail coverage.

[0039] Furthermore, fields such as food delivery, home delivery services, and advertising contain a lot of domain-specific descriptive information, and the semantics of this descriptive information may differ across different domains. The first stage typically uses a general image-text alignment scheme to generate multimodal vectors. However, a general image-text alignment scheme is difficult to adapt to the semantics of specific domains, resulting in low accuracy of the generated multimodal vectors, and consequently, low accuracy in the final information recommendation.

[0040] Based on this, embodiments of this application provide an information processing method. In this method, a first training sample is obtained, comprising user historical behavior data, a first candidate object set, and a target object index. The first candidate object set includes first candidate objects recommended based on the user's historical query terms. The target object index is the index of the target object in the first candidate object set that has been manipulated by the user. Based on the first training sample and model parameters of a preset model decoder, the preset model decoder is trained to obtain user behavior prediction information. The user behavior prediction information includes index prediction values, which refer to the index of the first candidate object predicted to be manipulated by the user in the first candidate object set. The model parameters are pre-configured with a first mapping relationship between object identifiers and multimodal vectors. The multimodal vectors are generated by fusing the image and text of the corresponding object. Based on the user behavior prediction information and the target object index, the model parameters are adjusted, and training continues until a first preset convergence condition is met. A second mapping relationship between the adjusted object identifiers and multimodal vectors is obtained from the trained model parameters.

[0041] This method incorporates the first mapping relationship between object identifiers and multimodal vectors as part of the model parameters of a pre-defined model decoder. This allows for optimization and adjustment of the multimodal vectors of each object within the first mapping relationship during the training process of the pre-defined model decoder. The first training samples of the pre-defined model decoder include user historical behavior data, which contains a wealth of information about user behavior and / or user intent. Based on these first training samples, the pre-defined model decoder generates user behavior prediction information. Then, based on this prediction information and the indexes of target objects actually interacted with by the user in the first training samples, the multimodal vectors in the first mapping relationship are adjusted. This allows the adjusted multimodal vectors to capture the correlations between different objects in the user's historical behavior data caused by user behavior and / or user intent. The adjusted multimodal vectors contain richer user context-aware information, ensuring that the multimodal vectors that have learned user intent align with the subsequent ranking stage's goal of ranking based on user intent. This improves the logical reasoning ability of the recommendation system and enhances the accuracy of information recommendation.

[0042] The information processing method provided in this application can be applied to any information recommendation scenario, including but not limited to store recommendations in the food delivery sector, service recommendations in the home delivery sector, and advertising information recommendations in the advertising sector. The objects mentioned in this application can include, but are not limited to, stores, services, advertisements, and products.

[0043] It should be noted that the application scenarios or examples provided in this application are for ease of understanding, and the embodiments of this application do not specifically limit the application of the technical solutions. Furthermore, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0044] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0045] Figure 1 A flowchart illustrating an information processing method provided in an embodiment of this application is shown. Figure 1 As shown, the method specifically includes steps 101-103.

[0046] Step 101: Obtain the first training sample, which includes user historical behavior data, a first candidate object set, and a target object index; the first candidate object set includes first candidate objects recommended based on the user's historical query terms; the target object index is the index of the target object in the first candidate object set that was operated on by the user in the first candidate object set.

[0047] Step 102: Based on the first training sample and the model parameters of the preset model decoder, train the preset model decoder to obtain user behavior prediction information. The user behavior prediction information includes index prediction value, which refers to the index of the first candidate object predicted to be operated by the user in the first candidate object set. The model parameters are pre-configured with a first mapping relationship between object identifiers and multimodal vectors. The multimodal vectors are generated by fusing the image and text of the corresponding object.

[0048] Step 103: Based on user behavior prediction information and target object index, adjust model parameters and continue training until the first preset convergence condition is reached. Obtain the second mapping relationship between the adjusted object identifier and the multimodal vector from the model parameters after training.

[0049] In this embodiment of the application, the method can be applied to a computing device, which can be, but is not limited to, a terminal or a server. The server can include, but is not limited to, a single physical server, a server cluster consisting of multiple physical servers, a cloud server, etc.

[0050] The first training set of the pre-defined model decoder includes multiple first training samples. These first training samples are obtained through data mining from search logs. Search logs are used to record interaction information between users and target domain platforms. Target domain platforms may include, but are not limited to, food delivery platforms, home service platforms, advertising platforms, and content service platforms.

[0051] The interaction between a user and a target domain platform may include, but is not limited to, the user submitting a query term, the target domain platform searching based on the query term and generating a set of candidate objects that can be fed back to the user, and the user performing actions on the target objects in the candidate object set. These actions include, but are not limited to, clicking, adding to cart, making payment, adding to favorites, and staying on the page. Staying on the page can refer to the target object being displayed on the client for a period exceeding a certain threshold, indicating that the user remains on the target object's display page to browse the target object.

[0052] The interaction information recorded in the search log can include user historical behavior data, a set of candidate objects provided to the user by the target domain platform in response to the user's query, and the target object selected by the user from the candidate object set. User historical behavior data may include, but is not limited to, the user's spatiotemporal context and historical object sequences. Spatiotemporal context can refer to the time of the user's interaction with the target domain platform, the user's geographical location, user-side context, and fulfillment-related constraints. User-side context may include, but is not limited to, the device information corresponding to the user's client, weather information at the time of the user's interaction with the target domain platform, and the page position displayed on the client. Fulfillment-related constraints are related to the type of target domain platform and are used to limit the constraints that the objects recommended to the user must meet. As an example, if the target domain platform is a food delivery platform, the fulfillment-related constraints can be delivery-related constraints, specifically including but not limited to delivery range, estimated delivery time, delivery capacity status, and store status.

[0053] In this embodiment, user history behavior data within a preset time period, candidate object sets for different query terms for the user, and the target object selected by the user in each candidate object set are obtained from the search log. For ease of description, any candidate object set in each candidate object set is referred to as the first candidate object set. The first training sample is set to include user history behavior data, the first candidate object set, and a target object index, wherein the target object index can be the sorting number of the target object operated by the user in the object sequence included in the first candidate object set.

[0054] For the same user, multiple first training samples corresponding to that user can be obtained using the above method. For different users, multiple first training samples corresponding to other different users can be obtained using the same method. The obtained first training samples are then combined to form the first training set of the preset model decoder.

[0055] Since the first training sample includes user historical behavior data, which contains user-specific information such as behavioral habits and intentions, using such a first training sample to train the preset model decoder enables the trained preset model decoder to learn information about user behavior and intentions. This allows the preset model decoder to move beyond simply predicting objects that may be operated by the user based on content similarity, and instead to make predictions based on the user's behavioral intentions, effectively improving prediction accuracy.

[0056] In embodiments of this application, a first mapping relationship between object identifiers and multimodal vectors is pre-obtained. The multimodal vectors in the first mapping relationship can be generated through a preset image-text alignment model. For any object in the target domain, the object image and object text of the object are semantically aligned using the preset image-text alignment model, and the features of the aligned object image and object text are fused to form the multimodal vector of the object.

[0057] The acquired first mapping relationship is loaded into the model parameters of the preset model decoder. This first mapping relationship functions as a lookup table within the model parameters. After loading the first mapping relationship into the model parameters of the preset model decoder, the preset model decoder can be trained using the first training samples from the first training set obtained earlier. Specifically, for each first training sample, the first training sample is input into the preset model decoder. Based on the first training sample and the first mapping relationship included in the model parameters, the preset model decoder predicts the objects in the first candidate object set that will be manipulated by the user, and outputs user behavior prediction information. This user behavior prediction information includes at least the index of the predicted object in the first candidate object set, i.e., the aforementioned index prediction value.

[0058] The target object index in the first training sample is the index of the target object that the user actually interacted with, as recorded in the search log, in the first candidate object set. Based on the predicted values ​​of these indices in the user behavior prediction information and the target object indices in the first training sample, the difference between the predicted values ​​and the true values ​​of the preset model decoder can be determined.

[0059] The preset model decoder is trained in batches, with each batch using a certain number of initial training samples. Within the current batch, for each initial training sample, the difference between the predicted value and the true value of the preset model decoder can be determined using the method described above. The model parameters are then adjusted based on these differences. After adjusting the model parameters, the preset model decoder is trained again using the initial training samples from the next batch and the adjusted model. This process iterates until a first preset convergence condition is reached. This first preset convergence condition may include, but is not limited to, the number of iterations reaching a first preset number, or the prediction accuracy of the preset model decoder exceeding a certain threshold, etc.

[0060] After the first preset convergence condition is met, the trained preset model decoder and the trained model parameters are obtained. From the parameter table / embedding table corresponding to the object identifier and the multimodal vector in the trained preset model decoder, the updated first mapping relationship is derived as the aforementioned second mapping relationship.

[0061] The multimodal vectors of objects are used as model parameters of a pre-defined model decoder, which is trained using a first training sample containing user historical behavior data. This user historical behavior data contains a wealth of information about user behavior and / or user intent. The pre-defined model decoder generates index prediction values ​​based on the first training sample. The multimodal vectors used as model parameters are adjusted based on these index prediction values ​​and the indices of target objects actually interacted with by the user in the first training sample. The adjusted multimodal vectors can capture the correlations between different objects caused by user behavior and / or user intent, resulting in richer user context-aware information. This allows the multimodal vectors, which have learned user intent, to be adapted to the goal of ranking candidate objects based on user intent during information recommendation, thereby improving the logical reasoning ability of the recommendation system and increasing the accuracy of information recommendation.

[0062] In some embodiments of this application, user historical behavior data includes the user's spatiotemporal context and a sequence of historical objects; the sequence of historical objects includes objects that the user has previously interacted with. The process of training the preset model decoder may include: generating a fused prompt vector set based on the spatiotemporal context, the sequence of historical objects, a first candidate object set, and a first mapping relationship; this fused prompt vector set is used to represent the content features of objects and user behavior features in a vector space; the fused prompt vector set is input into the preset model decoder to output user behavior prediction information.

[0063] The spatiotemporal context can represent information about a user's past time period, including time, geographical location, and weather. The historical object sequence contains objects that the user has previously interacted with, arranged chronologically according to the user's actions. For each object in the historical object sequence, the sequence may include only the object's identifier, or it may include the object's identifier, object image, object text, and other information.

[0064] In this application, the object image mentioned in the embodiments may include one or more images related to the corresponding object. As an example, if the object is a store, the object image of the store may include the store's logo image, images of one or more goods or services offered by the store, etc. The object text mentioned in this application may also include one or more pieces of text related to the corresponding object. As an example, if the object is a store, the object text of the store may include descriptive information such as the store's name, address, and store features, as well as descriptive information about one or more goods or services offered by the store, etc.

[0065] The first candidate object set may include object identifiers for one or more candidate objects. The object identifier is used to uniquely identify the corresponding object; it may be an object name, object number, etc. The first mapping relationship includes the correspondence between object identifiers and multimodal vectors for different objects. This first mapping relationship may include the object identifiers and corresponding multimodal vectors of each object in the target domain platform.

[0066] This embodiment constructs a fused suggestion vector set, mapping the historical object sequence representing user behavior, the spatiotemporal context representing the environmental information of the user behavior, and the first candidate object set representing the recommended object into the same vector space. By combining these three elements with a unified standard based on the first mapping relationship, the fused suggestion vector set represents both the content features of objects and the user behavior features within the same vector space. This approach breaks down the semantic barriers between user behavior features and multimodal object features in traditional models, enabling the pre-defined model decoder to directly capture the deep correlation between the user's historical behavior and the content features of candidate objects within the same semantic space. This significantly improves the effectiveness of feature fusion and the model's representational capabilities.

[0067] Because the fused cue vector set incorporates spatiotemporal context and historical object sequences, training the pre-defined model decoder using this set enables it to more accurately perceive the user's current real-time state (such as time and location). For example, it allows the decoder to distinguish between a user's different needs for the same type of object during a weekday lunch break and a weekend evening. By utilizing spatiotemporal context and historical object sequences, the pre-defined model decoder can more accurately infer the user's dynamic intent in specific scenarios, thereby improving the accuracy of user behavior prediction.

[0068] Furthermore, since the fusion of the cue vector set includes information from the historical object sequence and the first candidate object set, the model is able to effectively capture the long-term and short-term dependencies in the historical object sequence (e.g., a user orders milk tea after eating spicy hot pot) and calculate the probability of the candidate object in the evolution of the sequence, thereby generating prediction results that are more consistent with the user's behavioral logic.

[0069] In some embodiments of this application, a fusion prompt vector set is generated based on spatiotemporal context, historical object sequence, first candidate object set, and first mapping relationship, including: for each object operated by the user in the historical object sequence, obtaining the multimodal vector of each object operated by the user from the first mapping relationship; for each first candidate object in the first candidate object set, obtaining the multimodal vector of each first candidate object from the first mapping relationship; encoding the spatiotemporal context into a context vector; and combining the context vector, the multimodal vectors of each object operated by the user, and the multimodal vectors of each first candidate object into a fusion prompt vector set.

[0070] The historical object sequence may include object identifiers of each object that the user has interacted with. Based on the object identifiers, the corresponding multimodal vectors are obtained from the first mapping relationship. The first candidate object set may include object identifiers of each first candidate object. Based on the object identifiers of the first candidate objects, the corresponding multimodal vectors are obtained from the first mapping relationship.

[0071] The spatiotemporal context is encoded into a context vector that can be recognized by the preset model decoder. The context vector, the multimodal vectors of each object in the historical object sequence, and the multimodal vectors of each first candidate object in the first candidate object set are arranged and concatenated in a preset order to form a fusion prompt vector set.

[0072] As an example, the above-mentioned preset order could be: the context vector is placed first, then the multimodal vectors of each object in the historical object sequence are arranged according to the order of the objects in the historical object sequence, and then the multimodal vectors of each first candidate object are arranged according to the order of the first candidate objects in the first candidate object set.

[0073] In other implementations, the historical object sequence may include first placeholders for each object the user has interacted with. The first placeholders in the historical object sequence are replaced with the multimodal vectors of each object interacted with in the first mapping relationship. The first candidate object set may include second placeholders for each first candidate object. The second placeholders in the first candidate object set are replaced with the multimodal vectors of each first candidate object in the first mapping relationship. Then, the context vector, the historical object sequence after the replacement operation, and the first candidate object set are arranged and concatenated in the aforementioned preset order to form a fused prompt vector set.

[0074] As an example, let c represent the spatiotemporal context, and S represent the spatiotemporal context. 1：t =[S1, ..., S t [] represents a sequence of historical objects. C 1：m =[C1, ..., C m ] represents the first candidate object set. Then the fusion hint vector set can be expressed as follows: (1): X=[c;e(S1;...;e(S t );e(C1);...;e(C m )]...(1) Where e(S1) represents the multimodal vector of the first object in the historical object sequence, e(S t e(C1) represents the multimodal vector of the t-th object in the historical object sequence; e(C1) represents the multimodal vector of the 1st first candidate object in the first candidate object set. m ) represents the multimodal vector of the m-th first candidate object in the first candidate object set.

[0075] In this embodiment, the historical object sequence and the objects in the first candidate object set are mapped to corresponding multimodal vectors, the user's spatiotemporal context is encoded into a context vector, and they are combined to form a fusion prompt vector set. This eliminates the semantic gap between different modal data, so that the preset model decoder does not need to process complex heterogeneous data structures. It can directly calculate the content association between the user's historical object sequence and the candidate objects in a unified semantic space, which significantly improves the efficiency and depth of feature fusion.

[0076] Furthermore, because the fused cue vector set contains both the object's multimodal vector and the user's context vector, the multimodal vector represents the object's content features, while the context vector represents the scene features of the user's interaction with the object. By fusing the cue vector set, the pre-defined model decoder can no longer rely solely on the user's historical behavior during prediction, but can simultaneously perceive both the object's content and the scene information of the user's interaction with the object. For example, the pre-defined model decoder can learn that the user's preference weight for "hot pot" increases in "cold weather." This joint representation significantly improves the adaptability and accuracy of the recommendation system in complex scenarios.

[0077] Moreover, the fusion of the prompt vector set contains the multimodal vectors of the object, which include the object's image and text semantic information. Therefore, even if a candidate object is a newly added "new object", the preset model decoder can make an accurate prediction through the multimodal vector of the "new object", effectively solving the problem that traditional models have difficulty in handling cold start and long-tail objects, and improving the system's coverage.

[0078] In some embodiments of this application, user behavior prediction information is predicted based on a fused cue vector set using a preset model decoder, which may specifically include: The fused prompt vector set is input into the preset model decoder. The preset model decoder calculates the predicted probability of each first candidate object in the first candidate object set being operated by the user. The predicted probability is calculated based on the hidden state vector and the index embedding matrix. The hidden state vector is used to predict the index prediction value, and the index embedding matrix is ​​adjusted during training based on the target object index and the index prediction value. Based on the predicted probability corresponding to each first candidate object, the index of the first candidate object with the highest predicted probability in the first candidate object set is taken as the index prediction value. The preset model decoder outputs user behavior prediction information, which includes the index prediction value.

[0079] The objective function of the preset model decoder can be expressed as shown in formula (2): ...(2) In formula (1), Here, p represents the model parameters of the preset model decoder, p is the predicted probability that the first candidate object will be manipulated by the user, and c is the spatiotemporal context. Let C be the sequence of historical objects, C be the first candidate object set, and y be the index of the first candidate object that will be operated by the user, i.e., y is the predicted value of the above index.

[0080] The fused prompt vector set shown in formula (1) above is input into the preset model decoder, and the predicted probability of each first candidate object being operated by the user is calculated by formula (3) as follows: ...(3) In formula (3), k is the number of the first candidate object, and M is the total number of first candidate objects in the first candidate object set. Given a set of fused cue vectors X, the preset model decoder predicts the probability that the user will interact with the k-th first candidate object. h represents the hidden state used to predict the final index tag. ∈ R Mmax×d ,in, Let R be the indexed embedding matrix, R be the set of real numbers, Mmax be the number of rows in the matrix, and d be the number of columns in the matrix.

[0081] The predicted probability of each first candidate object is calculated using the above formula (3). The index embedding matrix is ​​a learnable parameter matrix. During the training process of the preset model decoder, the index embedding matrix is ​​adjusted based on the loss value of each training round. Each row of the index embedding matrix can represent a vector representation of a candidate index. The above formula (3) can map the first candidate object set to the limited index space of the index embedding matrix. In this way, the prediction model decoder does not need to directly predict a large number of specific object IDs, but only needs to make predictions from this limited index space, which effectively saves the amount of computation and improves the stability of prediction.

[0082] After calculating the predicted probability of each first candidate object using formula (3) above, the index prediction value with the highest predicted probability is then determined using formula (2) above. The index prediction value output by the preset model decoder can be expressed as follows: .

[0083] In some embodiments, the output of the preset model decoder may include only the index prediction values. The preset model decoder uses index tags. The index of the candidate object is used to represent the prediction target. This transforms the prediction target from a huge space of full object IDs into a finite, dynamic index prediction value, which greatly reduces the complexity of the output space.

[0084] In other embodiments, the user behavior prediction information may further include reasoning process information, which is used to explain the reason why the first candidate object corresponding to the index prediction value will be operated by the user based on the user's historical behavior data. Before the preset model decoder outputs the index prediction value in the above embodiments, the reasoning process information may also be generated and output by the preset model decoder. The output of the preset model decoder may also be as shown in the following formula (4): ...(4) In formula (4), This represents the output of the preset model decoder. For information about the reasoning process, and These are auxiliary start and auxiliary end markers for the reasoning process information, respectively.

[0085] The preset model decoder outputs inference process information and index prediction values. The inference process information is used to explain why the first candidate object corresponding to the index prediction value is selected. The inference process information can help the preset model decoder capture more complex feature interactions. Compared with the method of only outputting the prediction index prediction value, the method of outputting the inference process information first enables the model to make more accurate decisions, improves the prediction accuracy and the stability of model training, and enhances the interpretability of the preset model decoder.

[0086] In this embodiment, the preset model decoder generates an output sequence based on the fused cue vector set, and finally uses index tags. The index of the candidate object is used to represent the prediction target. The prediction target is transformed from a massive space of all object IDs into a finite, dynamic index prediction value, greatly reducing the complexity of the output space. Adding inference process information to the model output allows the model to perform logical deduction before outputting the final index, giving the final predicted index value interpretability. This "thinking chain" mechanism allows the model to combine user history (e.g., "the user ordered spicy hot pot") and current context (e.g., "it's evening") for deep feature cross-referencing, thereby more accurately capturing user intent and improving prediction accuracy.

[0087] In some embodiments of this application, adjusting model parameters based on user behavior prediction information and target object index includes: determining a reward signal for reinforcement learning based on the index prediction value and the target object index; using the reward signal to quantify the degree of consistency between the index prediction value and the target object index; calculating the advantage estimate corresponding to the index prediction value based on the reward signal; calculating the gradient of the model parameters based on the advantage estimate; and adjusting the model parameters using the gradient.

[0088] In some embodiments, the reward signal can be a binary reward based on whether the final index prediction hits the target object index. Specifically, the reward signal can be determined using the following formulas (5) and (6): ...(5) ...(6) In formulas (5) and (6), s represents the state of reinforcement learning and is defined as the fusion cue vector set X. This represents the output of the preset model decoder, where r is the reward signal. Index the target object. This represents the index prediction value that is finally output by the preset model decoder. ∈ A, A= A is a sequence of index tags, and each index tag in A corresponds one-to-one with each first candidate object in the first candidate object set.

[0089] By using the above formula (5), the object operated by the user is regarded as a positive target. If the index prediction value is equal to the target object index, then r = 1. Otherwise, r = 0. During the training process of the preset model decoder, the prediction results corresponding to each first training sample are all reinforced by the reward signal shown in the above formula (5).

[0090] After determining the reward signal using the above method, the advantage estimate is calculated using the following formula (7). Formula (7) performs group-based normalization, which can reduce the variance under sparse rewards: ...(7) In formula (7), Let represent the advantage estimate of the i-th first training sample at position t. This represents the original reward signal obtained for the i-th first training sample. Yes The abbreviation represents the original reward signal. The value after normalization. mean(r) represents the reward signal r = {r1, ..., r} for all samples in the current batch. G The average value of the reward signals is given by}. std(r) represents the standard deviation of all reward signals r in the current batch, which measures the dispersion of the reward signals.

[0091] Where r = {r1, ..., r G The reward is the sum of the samples taken from the same state. In this embodiment, a sequence-level advantage estimate is calculated based on the termination reward. And broadcast it to the model output o corresponding to the i-th first training sample. i The position t of all tags (including intermediate inference information). If std(r) = 0, then Set to 0.

[0092] The above methods encourage models to generate complete logical chains that lead to correct results, rather than simply guessing the final outcome, thus solving the "credit allocation" problem and the sparse reward problem. The method for calculating the advantage estimate compares the relative merits of samples within the same group, eliminating the scaling effect of the reward itself by subtracting the mean and dividing by the standard deviation. This results in more accurate gradient estimation and more stable training.

[0093] After calculating the advantage estimate value in the above manner, the loss value is calculated using the following formulas (8) and (9): ... (8) ... (9) Wherein, in formulas (8) and (9), is the loss value. E[...] represents the expectation, which is used to sample and average the distribution of the variables within the brackets. s is the state of reinforcement learning, is the data distribution of state s. represents G output sequences generated by the old policy model for the same state s. represents the i-th output sequence. is the old policy, that is, the policy at the moment before this update, whose parameters remain unchanged and are used as a reference benchmark. is the probability that the current policy (with parameters θ) generates the t-th token in the i-th sequence under the given state s and historical token , <t , t. is the probability that the old policy generates the t-th token in the i-th sequence under the given state s and historical token , <t , t. That is and are the probabilities that the current policy and the old policy respectively generate the same token, and the ratio of the two measures the magnitude of the policy update. is the advantage estimate value at the t-th position of the i-th sequence, which is used to measure how much better taking this action at the current position is than the "average level".

[0094] is the hyperparameter of the clipping range, which is used to limit the step size of the policy update and prevent the training from being unstable due to too large a single update amplitude. [...] is the KL divergence. The policy is regularized using an unbiased and non-negative estimator of the KL divergence between the current policy and the reference policy, which is used to constrain the current policy from deviating too far from the old policy to ensure the stability of training. is the reference model, which can be the initial reference policy after supervised fine-tuning. β is the coefficient of the KL divergence penalty term, which is a hyperparameter used to control the intensity of regularization. is the reference model under the given state s and historical token , <t , t. Introducing the KL divergence is to prevent the optimized policy from deviating from the reference policy Too far away, thus maintaining the stability and quality of the content generated by the model.

[0095] For each state s, from the old policy A sampled output set {o1, o2, ..., o...} is generated. G}, and optimize the strategy by maximizing the above formula (8). The above o i It can be a short sequence consisting of an auxiliary start marker, inference process information, an auxiliary end marker, and an index prediction value, so |oi| is variable and subject to the maximum generation length constraint.

[0096] After calculating the loss value of the preset model decoder using the above formula (8), the backpropagation function is called to differentiate the loss value to obtain the gradient of the model parameters of the preset model decoder. The model parameters of the preset model decoder are then adjusted using this gradient.

[0097] Since the model parameters of the preset model decoder include the first mapping relationship, the multimodal vectors of the objects in the first mapping relationship are also optimized during the process of adjusting the model parameters using gradients. This results in the optimized multimodal vectors containing information such as user intent and spatiotemporal context. Furthermore, during training, the reward signal is determined by the difference between the index prediction value and the target object index for reinforcement learning, effectively improving the prediction accuracy of the preset model decoder. This embodiment also employs group sampling and within-group relative merit evaluation, effectively reducing variance during training and preventing the preset model decoder from being biased by low-quality samples.

[0098] In some embodiments of this application, before training the preset model decoder based on the model parameters of the first training sample and the preset model decoder, the first training sample is also input into the preset teacher model, and the reasoning reason corresponding to the first training sample is output. The reasoning reason is obtained by reasoning based on the user's historical behavior data and the target object index. The reasoning reason is added to the first training sample to obtain the updated first training sample.

[0099] Before performing reinforcement learning on the preset model decoder, the preset model decoder is warmed up using SFT (Supervised Fine-Tuning) to ensure that the inference steps of the preset model decoder are reasonable and faithful to the selection of the true labels. For each first training sample in the first training set, the inference reason corresponding to each first training sample is generated using the preset teacher model in the manner described above. Then, each first training sample is combined with its corresponding inference reason to form a new first training sample. The updated first training sample can then be used to fine-tune the preset model decoder to generate an output sequence, thereby providing stable and accurate initialization data for subsequent reinforcement learning on the preset model decoder. The output sequence may include index prediction values, or the output sequence may be the sequence shown in formula (4).

[0100] This embodiment utilizes a preset teacher model to generate reasoning reasons corresponding to the first training sample, and adds these reasoning reasons to the first training sample, thereby enriching the training sample. In subsequent fine-tuning or model training processes, the preset model decoder can learn these reasoning reasons, thus learning more complex logical reasoning. This can significantly accelerate the convergence speed of the model, increase the performance ceiling of the preset model decoder, and improve the information density of the training sample and the model training efficiency.

[0101] After warming up the preset model decoder using SFT, the preset model decoder can be optimized based on the reward signal using the optimization schemes described in formulas (5)-(9) above. Compared with the ordinary policy gradient in related technologies, the optimization scheme of this application adopts group-based sampling and intra-group advantage normalization, which can be stably learned under sparse binary rewards. In addition, truncation and KL regularization are applied to the reference policy to prevent destructive updates, which is particularly important when jointly updating the preset model decoder and multimodal vectors.

[0102] To facilitate understanding of the processing procedure of the preset model decoder, the following section combines... Figure 2 The schematic diagram of the preset model decoder is shown for illustration. For example... Figure 2 As shown, the preset model decoder performs generative user behavior alignment. The input to the preset model decoder includes spatiotemporal context, a sequence of historical objects, and a first set of candidate objects. The spatiotemporal context includes temporal and spatial features. The sequence of historical objects reflects user behavior, including multiple target objects S1, S2, ..., S3, which the user has interacted with, arranged chronologically. t Each target object is represented by a corresponding multimodal vector. The first candidate object set includes multiple first candidate objects C1, ..., C2. mEach first candidate object is represented by a corresponding multimodal vector. A fused cue vector set is constructed from the spatiotemporal context, historical object sequence, and first candidate object set. This fused cue vector set is input into the decoder, which outputs the predicted index value y'. Based on the true target object index y and the predicted index value... Determine the reward signal [ =y], the decoder is optimized by reinforcement learning based on the reward signal. The preset model decoder performs generative user behavior alignment, predicting the next object that will be operated by the user from the retrieved first candidate object set, thereby aligning the multimodal vector of the object with the behavior target, and can capture the sequential complementarity between different objects (e.g., a user orders milk tea after eating spicy hot pot).

[0103] By training a pre-defined model decoder, the model parameters are updated during training. These parameters include a first mapping relationship containing multimodal vectors of objects, and an updated index embedding matrix. After training the pre-defined model decoder, optimized multimodal vectors of each object can be derived from its model parameters, and the object identifiers of each object and the optimized multimodal vectors are stored as the aforementioned second mapping relationship. This second mapping relationship serves as the data foundation for subsequent online recommendation models.

[0104] In some embodiments of this application, before training the preset model decoder based on the first training samples and the model parameters of the preset model decoder to obtain user behavior prediction information, the object identifier, object text, and object image of each object can be obtained; based on the object text and object image of each object, a multimodal vector corresponding to each object is generated through a pre-trained image-text alignment model; the object identifier of each object and the multimodal vector of each object are stored to obtain a first mapping relationship; and the first mapping relationship is configured into the model parameters of the preset model decoder.

[0105] The first mapping relationship configured in the model parameters of the preset model decoder mentioned above is pre-generated through the image-text alignment model. The aforementioned object image can include one or more object-related images. For example, in the food delivery field, the object image can be a store image, specifically including the store's logo image, images of various products offered in the store, etc. The aforementioned object text can also include one or more object-related texts. For example, in the food delivery field, the object text can be store text, specifically including the store's descriptive text, descriptive text of various products offered in the store, etc.

[0106] In this embodiment, a text-image alignment model is used to extract rich semantic features from the object image and object text, and then fuse them into a multimodal vector. The multimodal vector not only contains the object's visual features but also incorporates textual descriptive features. Subsequently, the multimodal vector is injected into the model parameters of a preset model decoder, enabling the preset model decoder to more accurately understand the object's fine-grained attributes.

[0107] In some embodiments of this application, the training process of the above-mentioned image-text alignment model may include: obtaining a second training sample, which includes positive samples and negative samples. The positive samples include a first query term corresponding to a first sample object, a first object image of the first sample object, and a first object text semantically matching the first object image; the negative samples include multimodal data that does not match the second sample object or the second query term corresponding to the second sample object; based on the second training sample, generating query term vectors, object text vectors, object image vectors, and image-text fusion vectors corresponding to the second training sample through the image-text alignment model to be trained; calculating the contrastive loss function value based on the query term vectors, object text vectors, object image vectors, and image-text fusion vectors; adjusting the model parameters of the image-text alignment model to be trained based on the contrastive loss function value, and continuing training until a second preset convergence condition is reached to obtain the trained image-text alignment model.

[0108] The first and second sample objects mentioned above can be the same or different. The first and second query terms can also be the same or different. The positive samples include image-text matching, while the negative samples include multimodal data that does not match the second sample object or the second query term corresponding to the second sample object. This multimodal data can include text, images, and fused features of text and images. As an example, a negative sample can include a query term and text where the text does not match the query intent of the query term. A negative sample can also include a query term and an image where the content of the image does not match the query intent of the query term. A negative sample can also include a query term and image-text fusion features where the image-text fusion features do not match the query intent of the query term. A negative sample can also include semantically inconsistent images and text.

[0109] As an example, the query term for a positive sample is "hamburger", the image is an image of a beef hamburger, and the text is "Beef hamburger containing large chunks of beef and fresh vegetables"; the query term for a negative sample is "hamburger", the image is an image of a beef hamburger, and the text is "Taro milk tea, with optional toppings such as brown sugar pearls and glutinous rice".

[0110] In this embodiment, both positive and negative samples used for training the image-text alignment model include query terms, object images, and object text. Samples containing these three parts can be referred to as sample triplet data. Figure 3The diagram illustrates a sample triplet dataset, which is sample data from the food delivery industry. By querying a user's order records, triplet datasets consisting of query terms, store images, and store text can be extracted from the data of orders placed by the user.

[0111] The aforementioned image-text alignment model includes a visual encoder, a text encoder, and a fusion module. After inputting the second training sample into the image-text alignment model, the text encoder encodes the query terms in the second training sample to obtain query term vectors. The text encoder then encodes the object text in the second training sample to obtain object text vectors. The visual encoder encodes the object images in the second training sample to obtain object image vectors. Finally, the fusion module fuses the object text vectors and object image vectors to obtain a fused image-text vector.

[0112] The image-text alignment model utilizes a contrastive learning algorithm to compare query word vectors with object text vectors, query word vectors with object image vectors, and query word vectors with image-text volume vectors. It calculates a contrastive loss function value through contrastive learning, and then adjusts the model parameters based on this value. The model is then iteratively trained using the next batch of training samples and the adjusted parameters until a second preset convergence condition is met, at which point training terminates, yielding the trained image-text alignment model and its parameters.

[0113] The aforementioned second preset convergence condition may include the number of iterations reaching a set number, or the comparison loss function value being less than a set threshold, etc.

[0114] Domain-specific semantics and user intent patterns are typically inherent to that domain. Domain-specific multimodal vectors should simultaneously encode both these semantics and user intent patterns to achieve higher recommendation accuracy when making object recommendations based on multimodal vectors. As an example, an effective multimodal representation for food delivery recommendations must simultaneously encode both domain-specific semantics and user intent patterns. Unlike general domains, food delivery platforms exhibit two key characteristics: (i) highly localized culinary terminology (e.g., "ants climbing a tree" refers to pork vermicelli, not literally insects or trees), which general multimodal models cannot understand; and (ii) users prioritize specific visual / textual attributes during the decision-making process. Based on this, this implementation employs a dual alignment mechanism: 1) domain-specific image-text alignment; and 2) query-store alignment under multi-loss optimization.

[0115] For domain-specific image-text alignment, a training dataset is constructed based on query terms, object images, and object text within the domain to bridge semantic gaps in domain terminology. This training dataset enables the image-text alignment model to accurately understand specific semantics within a particular domain. In this training dataset, positive image-text pairs are extracted from structured object data (e.g., image-text pairs of dishes / shops), while negative samples are generated through intra-batch negative sampling to reduce the impact of spurious negative noise.

[0116] The semantic similarity difference between object images and object text can be naturally represented by positive and negative user interactions. When a user's query in a search scenario leads to a user action, the semantic relevance between the query and the target object of the user's action is significantly stronger than that of samples that have not been acted upon. This provides an effective supervisory signal for contrastive learning. Therefore, positive samples are defined as triplets (query, object image, object text) consisting of query, object image, and object text, corresponding to the target object of the user's final action. Negative samples are taken from objects in the current batch that have not been acted upon by the user. The training dataset is constructed by mining user search-to-action behaviors, with each (query, object image, object text) triplet coming from high-confidence interactions.

[0117] For query-store alignment under multi-loss optimization, contrastive learning is used to compare query terms with object text, query terms with object images, and query terms with multimodal vectors, thereby enabling the image-text alignment model to capture user intent signals.

[0118] Training the image-text alignment model using both positive and negative samples enhances its ability to recognize cross-modal semantic consistency, effectively bridging the semantic gap between the visual and linguistic feature spaces. Through training with negative samples, the model learns to ignore semantically irrelevant noise, improving its robustness in complex real-world scenarios. By generating query word vectors, object text vectors, object image vectors, and image-text fusion vectors, the model maps query words, text, images, and image-text fusion features into a vector space, enabling the image-text object model to use a unified metric to calculate the relevance between different modalities.

[0119] In some embodiments of this application, the comparison loss function value is calculated based on query word vectors, object text vectors, object image vectors, and image-text fusion vectors, including: calculating a first comparison loss value between query words and object text in the second training sample based on query word vectors and object text vectors; calculating a second comparison loss value between query words and object images in the second training sample based on query word vectors and object image vectors; calculating a third comparison loss value based on query word vectors and image-text fusion vectors; and performing a weighted summation of the first comparison loss value, the second comparison loss value, and the third comparison loss value to obtain the comparison loss function value.

[0120] The contrast loss function value can be represented by the following formula (10): ...(10) In formula (10), To compare the loss function values, The first comparison loss value, This is the second contrast loss value. This is the third comparison loss value. , , The weights for the first, second, and third contrastive loss values ​​are used to balance the different modal alignment losses, thereby adapting the image-text alignment model to downstream tasks. Q represents the query term vector, i.e., the user's input intent or search term; Text and Image represent the object text vector and object image vector, respectively. This represents a vector representing the fusion of text and images.

[0121] The first, second, and third contrastive loss values ​​mentioned above can be calculated using the same loss function, which is shown in formula (11): ...(11) Among them, in the public announcement (11), To compare the loss values, b represents the two sides being compared. Let b be the value of the positive sample. Let b be the value of the side represented by b in the j-th negative sample, T be the temperature parameter, and K be the number of negative samples.

[0122] When using formula (11) to calculate the first contrast loss value, The first comparison loss value, To query word vectors, The text vector of the object in the positive sample. Let be the object text vector in the j-th negative sample. The objective function maximizes the positive sample text vector under the temperature parameter T. relative to a set of negative samples The logarithmic probability is used to learn a consistent multimodal representation.

[0123] When using formula (11) to calculate the second contrast loss value, This is the second contrast loss value. To query word vectors, The image vector of the object in the positive sample. Let be the object image vector in the j-th negative sample.

[0124] When using formula (11) to calculate the third contrast loss value, This is the third contrast loss value. To query word vectors, This is the image-text fusion vector in the positive sample. Let be the image-text fusion vector in the j-th negative sample.

[0125] In some embodiments, the image-text alignment model generates an image-text alignment vector corresponding to the second training sample based on the object text vector and object image vector corresponding to the second training sample. The contrast loss between the image and text can also be calculated using the above formula (11). This represents the contrast loss value between the image and the text. For object text vectors, The image vector of the object in the positive sample. Let be the object image vector in the j-th negative sample. The contrast loss value between the image and the text can be obtained by weighted summing the first, second, and third contrast loss values ​​mentioned above to obtain the final contrast loss function value.

[0126] In this embodiment, a first contrastive loss value is calculated between the query term vector and the object text vector. The query term vector represents the user's search intent, while the object text vector represents the textual description of the object. Through contrastive learning, the image-text alignment model can learn the semantic matching between the user's search intent and the object text, thus optimizing the system's text retrieval capabilities. A second contrastive loss value is also calculated between the query term vector and the object image vector. The object image vector represents the object's visual features. Through contrastive learning, the image-text alignment model can learn the semantic matching between the user's search intent (text modality) and the object image (image model). This allows the model to directly cross modalities based on the user's search intent and accurately match visual content, optimizing the system's cross-modal retrieval (text-to-image search) capabilities. By simultaneously optimizing these two aspects, the "image-text mismatch" situation in the image-text alignment model is effectively reduced.

[0127] This embodiment also calculates a third contrastive loss value between the query term vector and the image-text fusion vector. The image-text fusion vector typically contains richer higher-order features than a single modality (e.g., the interaction between image region features and text entities). The third contrastive loss value requires that the user's query term not only match the surface image and text but also match the deeper "image-text fusion logic," which can significantly improve the model's accuracy when handling complex queries. This embodiment weights and sums the loss values ​​from multiple contrastive learning methods, enabling the image-text alignment model to learn more general and robust feature representations, thus maintaining stable performance in various scenarios.

[0128] In some embodiments of this application, positive samples used for training the image-text alignment model can be obtained in the following manner: First, a first preset number of query terms most frequently used by the user are obtained from the query terms used to query the first sample object; one query term is obtained from the first preset number of query terms as the first query term corresponding to the first sample object; a second preset number of target resources are determined from the multiple resources corresponding to the first sample object, where the target user is the user who queries the first sample object using the first query term; the first image of the first sample object and the second images of the second preset number of target resources are both used as the first object image of the first sample object; the descriptive text corresponding to the first image and the descriptive text corresponding to the second image are both used as the first object text; positive samples are obtained based on the first query term, the first object image, and the first object text.

[0129] The search log records the user's past search terms, the set of candidate objects recommended by the system based on those search terms, and the target objects that the user actually interacted with within that set. If the target object provides resources, the search log also records the target resources that the user actually interacted with. As an example, in the food delivery industry, the search log records the user's past search terms (such as "hamburger"), multiple shops recommended based on those search terms, and the target products from the shops where the user actually placed orders.

[0130] Based on log records from multiple users, the mapping relationship between query terms, target objects, and target resources can be extracted. Any target object in this mapping relationship can be used as the first sample object. The query terms corresponding to the first sample object are obtained from this mapping relationship. These query terms may contain duplicates; the frequency of each query term is counted, and this frequency represents the usage frequency of the user using the query term to query the first sample object. The query terms corresponding to the first sample object are sorted in descending order of usage frequency to obtain a query term sequence, which contains no duplicate query terms. A first preset number of query terms are selected from this query term sequence. This first preset number can be, but is not limited to, 8 or 10. Any of the selected query terms can be used as the first query term.

[0131] From the mapping relationship between the query term, target object, and target resource, identify the records containing the first query term. Count the number of times each target resource appears in each record; this count represents the number of times the target resource has been operated on by the target user. Sort the target resources in each record in descending order of frequency of occurrence to obtain a resource sequence, where no duplicate target resources exist. Select a second preset number of target resources from the top of this resource sequence; this second preset number can be, but is not limited to, 5 or 10.

[0132] The first image of the first sample object and the second images of a second preset number of target resources are both used as the first object images of the first sample object. The descriptive text corresponding to the first image and the descriptive text corresponding to the second image are both used as the first object text of the first sample object. The first query term, the first object image, and the first object text are combined to form a positive sample.

[0133] User behavior data typically contains a large number of random clicks or accidental actions. By filtering the most frequently used query terms and the target resources with the most operations, low-quality, occasional behaviors can be filtered out, improving the accuracy and representativeness of the training data. This ensures that the image-text alignment model primarily learns the most typical and common "query-object" relationships, making the model more robust. By introducing the image and text of the target resource and merging them with the image and text of the first sample object, the features of the sample are essentially expanded, making the features learned by the model more comprehensive. This allows the model to better capture the object's behavior in different scenarios, reducing overfitting and enhancing the generalization ability of the image-text alignment model.

[0134] To understand the processing of the above image-text alignment model, the following will combine... Figure 4 The schematic diagram illustrating the principle of the text-image alignment model is shown below. For example... Figure 4As shown, the image-text alignment model performs domain-specific query-image-text alignment. The model includes a text encoder, a visual encoder, and a fusion module. The input sample data includes query terms, object text, and object images, such as... Figure 4 The query term can be "Store A", the object text can be "Store A mainly sells hamburgers, fries, desserts, etc.", and the object image can include Store A's logo, images of the hamburgers, fries, desserts, etc. offered by Store A, etc. Both the query term and object text are input into a text encoder, which outputs query term vectors and object text vectors, respectively. The object image is input into a visual encoder, which outputs an object image vector. Then, both the object text vector and the object image vector are input into a fusion module, which outputs an image-text fusion vector. Based on the query term vector, object text vector, object image vector, and image-text fusion vector, a first contrastive loss value is calculated between the query term vector and the object text vector through contrastive learning. The second contrast loss value between query word vectors and object image vectors. And the third contrast loss value between query word vectors and image-text fusion vectors. Then, the three contrastive loss values ​​are weighted and summed to obtain the total contrastive loss function value. Subsequently, the comparison loss function value can be used. Adjust the model parameters of the image-text alignment model, and then use the next batch of samples and the adjusted model parameters to continue training the image-text alignment model until the training is completed and a trained image-text alignment model is obtained.

[0135] By training the image-text alignment model, it is possible to maximize the similarity between matched images and text (e.g., a hamburger image and its corresponding description "juicy beef hamburger with lettuce"), while minimizing the similarity between mismatched images and text (e.g., a sushi image paired with an unrelated hamburger description). This training objective of the image-text alignment model ensures the consistency of visual and textual embeddings within a domain (such as food delivery, home delivery services, advertising, etc.). Figure 5 The diagram shows the distribution of multimodal features in the vector space before and after training. (a) shows that before the text-image alignment model was trained, the multimodal features of text, image, and text + image occupied batch-independent and non-overlapping regions in the vector space. (b) shows that after the training, the distribution of the multimodal features showed significant overlap, indicating that cross-modal consistency was achieved through training.

[0136] The positive samples used for training come from user behavior. As an example, in the food delivery field, matching query terms with stores that users have previously purchased from strengthens the relevance between query terms and stores. Negative samples are selected from stores that users have not purchased from. These negative samples are custom-sampled based on domain knowledge (e.g., store category, cuisine type), and they provide clear supervisory signals.

[0137] During training, each modality is processed separately before the fusion module, which preserves fine-grained visual details that are crucial for the recognition of target resources (such as dishes) (e.g., recognizing cilantro among ingredients). Then, the fusion module fuses the object images and object text included in the samples to obtain an image-text fusion vector. In essence, it aligns the semantics expressed by the matching object images with the semantics expressed by the object text, achieving domain-specific image-text semantic alignment.

[0138] During training, the image-text alignment model uses contrastive learning to align the semantics of query terms with the text vectors of objects (e.g., matching "spicy" with a shop's description of "Sichuan hot pot, spicy flavor"), thereby enhancing the semantic consistency between user intent and text features. By contrastively learning query term vectors with object image vectors, it ensures that query intent matches the visual features of the object (e.g., matching "bright, lively dining environment" with a shop's colorful, bustling interior image), strengthening the consistency between intent and visual features. The image-text fusion vector integrates visual and text features; through contrastive learning, it aligns query term vectors with the image-text fusion vector, ensuring that user intent matches the overall impression of the object. The contrastive loss values ​​calculated by these various contrastive learning methods are jointly optimized with weighted contributions, thereby refining the multimodal vectors of the object and achieving accurate intent-aware alignment.

[0139] After training the image-text alignment model using the above method, it can be used to generate multimodal vectors for objects published on the target domain platform. Specifically, object images and object text can be obtained from the target domain platform and input into the trained image-text alignment model. The text encoder in the image-text alignment model encodes the object text into an object text vector, and the visual encoder encodes the object image into an object image vector. The fusion module fuses the object text vector and object image vector to obtain the object's multimodal vector. For the multimodal vectors of each object on the target domain platform, the object identifier and the corresponding multimodal vector are stored as a first mapping relationship. Before training the preset model decoder in the previous embodiments, the first mapping relationship is configured into the model parameters of the preset model decoder.

[0140] After training the preset model decoder using the preceding embodiments, an optimized second mapping relationship is obtained from the model parameters of the trained preset model decoder. Subsequently, during the online information recommendation service provided to users, the second mapping relationship can provide data support for the online recommendation model.

[0141] In some embodiments of this application, after obtaining the adjusted second mapping relationship between the object identifier and the multimodal vector from the model parameters after training, the method may further include: obtaining a third training sample, the third training sample including the sample ID interaction vector and the sample multimodal interaction vector of the third sample object; the sample multimodal interaction vector is obtained based on the second mapping relationship; generating a first sample fusion vector of the third sample object through the concatenation layer in the recommendation model to be trained based on the sample ID interaction vector and the sample multimodal interaction vector; predicting a first probability that the third sample object will be operated by the user through the recommendation main module in the recommendation model to be trained based on the first sample fusion vector; predicting a second probability that the third sample object will be operated by the user through the auxiliary multilayer perceptron in the recommendation model to be trained based on the sample ID interaction vector and the sample multimodal interaction vector; calculating a target loss function value based on the first probability and the second probability of each third training sample in the current training batch; adjusting the model parameters of the recommendation model to be trained based on the target loss function value, and continuing training until a third preset convergence condition is reached to obtain a trained recommendation model.

[0142] Among them, the sample ID interaction vector is a feature representation vector based on IDs. It represents the most basic and direct identifying features of the sample users and the third sample objects that have interacted with them. It reflects the statistical patterns of the sample users' historical behavior, such as which object IDs the sample users have clicked in the past and the historical click rate of the object IDs. The sample ID interaction vector mainly captures mature and statistically significant collaborative signals.

[0143] The sample multimodal interaction vector is a multimodal feature representation vector that represents the content features of sample users and third-party sample objects, reflecting semantic understanding, such as what the store's images look like, what the text descriptions say, what image styles and keywords users pay attention to, etc. The sample multimodal interaction vector captures visual and semantic similarity. The purpose of the sample multimodal interaction vector is to use content information to compensate for the insufficiency of ID behavior data (i.e., to solve the problems of cold start and poor long-tail coverage).

[0144] This embodiment predicts the probability of a third sample object being operated on by a sample user based on sample ID interaction vectors and sample multimodal interaction vectors. The sample ID interaction vector captures the historical behavioral association between the sample user and the third sample object, excelling at handling popular objects with abundant historical data. The sample multimodal interaction vector, on the other hand, captures the content understanding signal of the semantic and visual content association between the sample user and the third sample object, excelling at understanding fine-grained attributes. Simultaneously using these two vectors for prediction achieves complementary dual signals, leveraging both the accuracy of historical statistical patterns and the richness of content features, thereby significantly improving the accuracy of operation probability prediction.

[0145] For newly listed objects (cold start) or niche objects (long tail objects), traditional models often perform poorly due to a lack of sufficient user interaction ID data. However, this application's embodiment introduces sample multimodal interaction vectors on top of sample ID interaction vectors. This enables matching of an object's image and text features with the current user's search intent, even without historical click data, thereby improving the exposure opportunities and information recommendation diversity for long tail or cold start objects.

[0146] Since the sample ID interaction vector and sample multimodal interaction vector are calculated based on the user identifier of the sample user and the object identifier of the third sample object, the calculated sample ID interaction vector and sample multimodal interaction vector will be different for the same third sample object for different sample users. This allows the recommendation model to capture the personalized intent of a specific user for a specific object (e.g., user A focuses on image style, user B focuses on brand ID), thereby achieving highly personalized ranking.

[0147] In some embodiments of this application, the above-mentioned sample ID interaction vector can be generated in the following manner, specifically including: obtaining the user behavior sequence corresponding to the sample user based on the user identifier of the sample user, wherein the user behavior sequence includes the first object identifier of the object that the sample user has previously operated on; obtaining the second object identifier of the third sample object; and generating the sample ID interaction vector corresponding to the third sample object based on the user behavior sequence and the second object identifier through a preset ID feature interaction function.

[0148] The search log can record the user's user identifier and the object identifiers of objects that the user has previously interacted with. Based on the user identifier of the sample user, the object identifiers of objects that the sample user interacted with within a preset time period are obtained from the sample user's search log. The obtained object identifiers are arranged in chronological order of the sample user's interaction time to obtain the aforementioned user behavior sequence.

[0149] The above sample ID interaction vector can be obtained using the following formula (12): ...(12) In formula (12), The interaction vector for the sample ID corresponding to the third sample object. For the user behavior sequence corresponding to the sample users, For the second object identifier, This indicates the preset ID interaction function.

[0150] For each third sample object, the sample ID interaction vector corresponding to each third sample object is obtained in the manner described above.

[0151] Traditional ID features often focus only on the "user ID" itself, which is a static attribute. However, this application introduces user behavior sequences, which reflect which objects a user has recently interacted with. Introducing user behavior sequences allows the ranking model to perceive changes in user intent, breaking through the limitations of static user ID profiling and enabling the recommendation model's results to quickly adapt to changes in user intent. By generating a sample ID interaction vector corresponding to a third sample object based on the user behavior sequence and the second object identifier, the recommendation model can learn complex transition patterns. For example, the recommendation model might learn that "a user who ordered spicy hot pot might next order milk tea." This ID sequence-based interaction can uncover more accurate conversion logic than simple content matching.

[0152] Generating sample ID interaction vectors based on user behavior sequences ensures that even for the same third sample object (i.e., the second object has the same identifier), the generated sample ID interaction vectors will be completely different for different sample users' user behavior sequences. Therefore, predicting the operation probability of the third sample object based on the sample ID interaction vector generated in this way can ensure that the predicted operation probability is highly personalized, effectively improving the prediction accuracy and thus improving the accuracy of the final trained recommendation model.

[0153] In some embodiments of this application, the multimodal interaction vector of a sample can be obtained in the following manner: based on the user identifier of the sample user who has operated on the third sample object, obtain the first object identifier of each object that the sample user has previously operated on; based on each first object identifier, obtain the multimodal vector corresponding to each first object identifier from a second mapping relationship; based on the second object identifier of the third sample object, obtain the multimodal vector corresponding to the third sample object from the second mapping relationship; based on the multimodal vectors corresponding to each first object identifier and the multimodal vectors corresponding to the third sample object, generate the multimodal interaction vector corresponding to the third sample object by using a preset multimodal feature interaction function.

[0154] The first object identifier of each object that the sample user has previously operated on is the object identifier of each object in the user behavior sequence mentioned above. The acquisition process is the same as the acquisition process mentioned above, and will not be repeated here.

[0155] The second mapping relationship stores the mapping relationship between object identifiers and multi-model vectors. The multi-modal vectors in the second mapping relationship can be multi-modal vectors that are first generated by the image-text alignment model trained in the previous embodiment and then optimized and adjusted during the training process of the preset model decoder in the previous embodiment.

[0156] From the second mapping relationship, obtain the multimodal vectors of each object that the sample user has previously operated on in the user behavior sequence, and obtain the multimodal vector corresponding to the third sample object. Then, generate sample multimodal interaction vectors based on these multimodal vectors.

[0157] The above sample multimodal interaction vector can be obtained by the following formula (13): ...(13) In formula (13), This is the sample multimodal interaction vector corresponding to the third sample object. Each first object is identified by its corresponding multimodal vector, which is the multimodal vector corresponding to the user behavior sequence. This is the multimodal vector corresponding to the third sample object. This represents a preset multimodal feature interaction function.

[0158] For each third sample object, the sample multimodal interaction vector corresponding to each third sample object is obtained in the manner described above.

[0159] Multimodal interaction vectors are generated based on the multimodal vectors of objects previously interacted with by sample users and the multimodal vectors of third-party sample objects. This allows the recommendation model to compare not only "users" and "objects," but also the "content features of objects interacted with in the past" and the "content features of objects," resulting in a more granular capture of user behavior. By pre-setting multimodal feature interaction functions, the distance or correlation between the multimodal vectors of objects previously interacted with by sample users and the multimodal vectors of third-party sample objects in the feature space can be calculated. This captures deeper user behavioral logic more effectively than simple ID matching.

[0160] For newly listed objects or long-tail objects, although these objects may lack historical ID interaction data (i.e., no user has interacted with them), they have images and text descriptions, thus allowing for corresponding multimodal vectors. Through the above embodiments, as long as the multimodal vectors of new or long-tail objects are semantically similar to the multimodal vectors of objects previously interacted with by the user, the model can accurately predict the user's likely actions on the object. This greatly alleviates the problem of cold-start objects and long-tail objects failing to be recommended.

[0161] The recommendation model includes a concatenation layer and a main recommendation module. The concatenation layer concatenates the sample ID interaction vector and the multimodal interaction vector of the third sample object. The first sample fusion vector obtained by concatenation is input into the main recommendation module, and the first probability corresponding to the third sample object is output.

[0162] In some embodiments of this application, based on the sample ID interaction vector and the sample multimodal interaction vector, a second probability of a third sample object being operated by a user is predicted using an auxiliary multilayer perceptron in the recommendation model to be trained. This includes: performing a gradient adjustment operation on the sample ID interaction vector, wherein the gradient adjustment operation is used to control the impact of parameter updates on the sample ID interaction vector during backpropagation; determining a first fusion weight and a second fusion weight corresponding to the sample ID interaction vector and the sample multimodal interaction vector respectively through a gating network based on the sample ID interaction vector; performing a weighted fusion of the sample ID interaction vector and the sample multimodal interaction vector after the gradient adjustment operation based on the first fusion weight and the second fusion weight to obtain a second sample fusion vector corresponding to the third sample object; and inputting the second sample fusion vector into the auxiliary multilayer perceptron to output the second probability of the third sample object being operated by the user.

[0163] Gradient adjustment operations can include, but are not limited to, gradient blocking, gradient scaling, and orthogonality constraints. Gradient blocking means that when the sample ID interaction vector is used as a constant input to the auxiliary multilayer perceptron (MLP), the model parameters on the feature extraction path containing the sample ID interaction vector are not adjusted during gradient backpropagation to adjust the model parameters during the training of the recommendation model. Gradient scaling refers to adjusting the intensity of the gradient backpropagation from the auxiliary MLP to the main recommendation module. For example, multiplying the gradient of the ID feature branch by a coefficient less than 1. Orthogonality constraints involve adding a constraint term to the loss function so that the gradient direction of the auxiliary MLP is as perpendicular as possible to the gradient direction of the main recommendation module, thereby reducing conflicts (i.e., reducing interference) between the two during parameter updates.

[0164] In traditional approaches, ID features are typically accumulated over a long period and have very strong signals; while multimodal features (such as images and text) are relatively sparse and difficult to train. Without intervention, gating mechanisms tend to give ID features extremely high weights and multimodal features extremely low weights. This leads to an imbalance between ID features and multimodal features. This imbalance causes gating collapse, causing the multimodal branch to be effectively ignored and no longer learned, thus failing to fully leverage the advantages of multimodal features.

[0165] Based on this, embodiments of this application introduce an auxiliary multilayer perceptron (ALP). In the ALP, a gradient adjustment operation is used on the sample ID interaction vector to adjust the ALP's dependence on ID features. Specifically, by blocking the gradient, the ALP's dependency path on ID features is directly severed. Since ID features cannot be updated through the ALP's gradient, the recommendation model is forced to optimize the multimodal interaction vector to reduce the ALP's loss. This allows the multimodal branches to be fully trained, effectively preventing the multimodal path from being marginalized during fusion.

[0166] ID features typically represent the most direct collaborative filtering signals for a user (e.g., user A has purchased product B). In the auxiliary multilayer perceptron, gradient adjustment is performed on the sample ID interaction vector. For cold-start or long-tailed objects, the gradient backpropagation of the ID feature branch can be reduced to decrease ID feature sharing and enhance the learning strength of multimodal features in cold-start or long-tailed object scenarios. Especially when gradient blocking is applied to the sample ID interaction vector, the sample ID interaction vector is only used as input but does not receive gradients. This design prevents noisy gradients generated by the auxiliary multilayer perceptron from corrupting the mature patterns already learned in the ID features. In other words, the auxiliary multilayer perceptron can use ID features as contextual references but cannot modify them, ensuring the stability of ID features in the main recommendation module.

[0167] For cold-start or long-tail objects, because few users have previously interacted with these objects, their ID interaction features are scarce and often unreliable. In such cases, multimodal interaction features must be relied upon. The embodiments of this application forcibly enhance the representational capability of multimodal interaction vectors by assisting a multilayer perceptron. Through this training mechanism, the multimodal features learned by the recommendation model will contain richer and more discriminative semantic information. When encountering long-tail or cold-start objects with missing or weak ID features, the recommendation model can still make accurate recommendations based on high-quality multimodal interaction vectors, thereby significantly improving the generalization ability and robustness of the recommendation model.

[0168] The recommendation model in this embodiment incorporates an ID-based gating network that dynamically balances the contributions of ID-based and multimodal-based vectors according to data features. The input to this gating network is the sample ID interaction vector. A first fusion weight is calculated from the sample ID interaction vector through a linear transformation of the activation function. Then, a second fusion weight is calculated from the first fusion weight to the sample multimodal interaction vector. Finally, the first and second fusion weights are used to perform a weighted fusion of the sample ID interaction vector and the sample multimodal interaction vector of the third sample object.

[0169] The first fusion weight of the above sample ID interaction vector can be calculated using the following formula (14): ...(14) In formula (14), The first fusion weight of the sample ID interaction vector. This represents the activation function. For sample ID interaction vectors, and These are trainable parameters.

[0170] After calculating the first fusion weight of the sample ID interaction vector of the third sample object using formula (14), calculate (1- The second fusion weight of the sample multimodal interaction vector of the third sample object is obtained. Then, the second sample fusion vector of the third sample object is obtained by the following formula (15).

[0171] ...(15) In formula (15), The second sample fusion vector for the third sample object.

[0172] After obtaining the second sample fusion vector of the third sample object through the above process, the recommendation model predicts the probability of the third sample object being operated on by the current user based on the second sample fusion vector. Specifically, the recommendation model includes a main recommendation module and an auxiliary multilayer perceptron. A concatenation layer concatenates the sample ID interaction vector and the sample multimodal interaction vector of the third sample object, and the resulting first sample fusion vector is input into the main recommendation module to obtain the first probability predicted by the main recommendation module that the third sample object will be operated on by the current user. The second sample fusion vector of the third sample object is then input into the auxiliary multilayer perceptron, which outputs the second probability predicted by the auxiliary multilayer perceptron that the third sample object will be operated on by the current user.

[0173] For different third sample objects, this embodiment dynamically calculates the first fusion weight and the second fusion weight corresponding to the sample ID interaction vector and the sample multimodal interaction vector, respectively, based on the sample ID interaction vector of the third sample object. Compared with the traditional scheme that uses a fixed ratio to fuse ID features and multimodal features, this embodiment dynamically determines the fusion weight, realizing dynamic feature fusion that varies from object to object, making the recommendation model more adaptive.

[0174] The gating network design in this embodiment can adaptively adjust the weights between ID-based signals and multimodal signals according to data features. For data-rich "head objects", their ID interaction vectors are more reliable, and the gating network will prioritize ID features. For "long-tail" or cold-start objects, more attention will be paid to multimodal features to alleviate the problem of sparse behavior.

[0175] For example, for popular objects with abundant ID data, the recommendation model automatically assigns a higher first fusion weight to the ID interaction vector, thus relying primarily on historical statistical patterns to predict the probability of operation. Conversely, for cold-start objects or long-tail objects with sparse ID data, the recommendation model automatically assigns a higher second fusion weight to the multimodal interaction vector, thus relying primarily on content semantic matching. Figure 6 The diagram illustrates the variations of the first and second fusion weights for ID interaction vectors and multimodal interaction vectors across different object layers. Squares represent the first fusion weight of the ID interaction vector, and circles represent the second fusion weight of the multimodal interaction vector. L1-L6 represent different object layers arranged in descending order of ID feature data richness, with L1 representing the tail object layer and L6 representing the head object layer. Figure 6 The diagram visually demonstrates that the second fusion weight of the multimodal interaction vector gradually increases from L6 at the head to L1 at the tail. In other words, for popular IDs (with abundant data and sufficient training), the model assigns higher weights to the ID interaction vector. Conversely, for less popular IDs (cold-start or long-tail objects), the model assigns higher weights to the multimodal interaction vector.

[0176] After dynamically determining the weights corresponding to the sample ID interaction vector and the sample multimodal interaction vector, the two are fused using the determined weights. The resulting second sample fusion vector contains both the precise collaborative signal of the ID feature and the semantic generalization signal of the multimodal feature. This makes the recommendation model accurate enough when recommending "head objects" (without wasting traffic) and intelligent enough when recommending "long-tail objects" (able to detect changes in user behavior habits), thereby improving the overall click-through rate and conversion rate of the recommendation model.

[0177] In some embodiments of this application, the target loss function value is calculated based on the first probability and the second probability of each third training sample in the current training batch, including: calculating the main loss value based on the first probability of each third training sample in the current training batch and the truth label of each third training sample; the truth label is used to characterize whether the corresponding third sample object has been operated by the user; calculating the auxiliary loss value based on the second probability of each third training sample in the current training batch and the truth label of each third training sample; and weighted summing the main loss value and the auxiliary loss value to obtain the target loss function value.

[0178] The truth label of the third sample object indicates whether the third sample object has been interacted with by the user. If the third sample object has been interacted with by the user, its truth label can be 1. Otherwise, the truth label of the third sample object can be 0.

[0179] The main loss value mentioned above can be calculated using the following formula (16): ...(16) In formula (16), The primary loss value is given by N, where N is the number of samples in the current training batch, and i is the sample index. Let i be the truth label of the i-th third sample object. This represents the first probability of the i-th third sample object predicted by the main recommendation module.

[0180] The aforementioned auxiliary loss value can be calculated using the following formulas (17) and (18): ...(17) ...(18) In formulas (17) and (18), As an auxiliary loss value, This is the second probability of the i-th third sample object predicted by the multilayer perceptron. Let i be the sample multimodal interaction vector corresponding to the i-th third sample object. Let i be the sample ID interaction vector corresponding to the i-th third sample object. Indicates blocking gradient, This represents an auxiliary multilayer perceptron. Formula (18) is illustrated with a blocking gradient. In embodiments that employ gradient adjustment operations other than blocking gradients, the specific formulaic representation of the auxiliary multilayer perceptron predicting the second probability can be designed based on the specific gradient adjustment operations adopted.

[0181] In formulas (17) and (18), the auxiliary loss value is a concatenated vector formed by the sample multimodal interaction vector and the sample ID interaction vector. This is calculated and is obtained from the concatenated vector. The algorithm blocks gradient propagation on the ID path and directly supervises the multimodal path to predict user action probabilities. This design ensures that gradients can still flow through the multimodal branch even if the main loss favors ID features.

[0182] The target loss function value mentioned above can be calculated using the following formula (19): ...(19) In formula (19), The target loss function value, This is a hyperparameter used to control the weights of the auxiliary loss values.

[0183] In this embodiment, the main loss value ensures that the recommendation model can fit complex user behavior patterns, pursuing the final prediction accuracy. The auxiliary loss value, as an additional supervisory signal, forces the recommendation model to maintain sensitivity to basic content features while learning complex behavior patterns. This combined effect of the main and auxiliary loss values ​​is equivalent to adding regularization constraints to the recommendation model, effectively reducing overfitting on training data and making it more stable when facing cold-start or long-tailed data. In the assisted multilayer perceptron, due to the gradient adjustment operation on the sample ID interaction vector, the auxiliary loss value actually focuses more on training multimodal features. Especially in the embodiment using gradient-blocking sample ID interaction vectors, the auxiliary loss value actually undertakes the task of specifically training multimodal features. If only the main loss value is used, the recommendation model might be lazy and only learn ID features. Introducing the auxiliary loss value requires the recommendation model to consider both objectives simultaneously, thus ensuring that multimodal features are fully learned and optimized, solving the problem of imbalance between ID features and multimodal features when only the main loss is used.

[0184] During the training phase of the recommendation model, the model comprises two prediction branches: a main recommendation module and an auxiliary multilayer perceptron. The main recommendation module predicts the first probability that a third sample object will be interacted with by the user based on the first sample fusion vector. The auxiliary multilayer perceptron predicts the second probability that a third sample object will be interacted with by the user based on the sample ID interaction vector and the sample multimodal interaction vector. Specifically, the auxiliary multilayer perceptron performs gradient adjustment on the sample ID interaction vector. This gradient adjustment operation controls the impact of the auxiliary multilayer perceptron on the parameter updates of the sample ID interaction vector during backpropagation. As an example, the gradient adjustment operation can include gradient blocking. In this case, the sample ID interaction vector is used as an auxiliary input feature; it can be understood as being used as a constant and not participating in the update of model parameters. During training, when adjusting model parameters, the gradient is not backpropagated to the feature extraction path of the sample ID interaction vector.

[0185] By introducing an auxiliary multilayer perceptron, the recommendation model can predict two probabilities for the same third sample object. Upon completion of training for a batch, the target loss function value for that batch is calculated based on the two predicted probabilities for each third sample object within that batch. The model parameters are then adjusted based on this target loss function value. The recommendation model is then trained again using the next batch of samples and the adjusted model parameters until a third preset convergence condition is met, resulting in a well-trained recommendation model. The third preset convergence condition can be, but is not limited to, the recommendation model reaching a set number of iterations, or the target loss function value being less than a set threshold, etc.

[0186] Traditional recommendation models only generate a predicted probability for each sample before calculating the loss. However, the embodiments in this application add an auxiliary multilayer perceptron (MLP) to predict a second probability using ID interaction vectors and multimodal interaction vectors. The APP essentially adds an extra constraint to the recommendation model, introducing an auxiliary supervision signal. This forces the lower-level feature extraction layer (the layer that generates ID interaction vectors and multimodal interaction vectors) to not only serve the main recommendation module but also directly support the APP's predictions. This helps prevent the recommendation model from overfitting to the training data and improves the model's generalization ability.

[0187] The target loss function is calculated based on the first probability predicted by the main recommendation module and the second probability predicted by the auxiliary multilayer perceptron. During backpropagation, the loss function includes not only the error of the main task but also the error of the auxiliary task. This means that when updating the model parameters of the recommendation model, it can obtain gradient signals from two directions, thus gaining richer gradient information. Introducing the auxiliary multilayer perceptron helps accelerate the convergence process of the entire model and improves the training efficiency of the recommendation model.

[0188] To facilitate understanding the training process of the above recommendation model, the following will combine... Figure 7 The structural diagram of the recommendation model shown is as follows: Figure 8 The training principle of the recommendation model is illustrated in the diagram shown below. Figure 7 As shown, the recommendation model includes a multimodal interaction layer, an ID feature interaction layer, a concatenation layer, a gating network, a main recommendation module, and an auxiliary multilayer perceptron. The connection relationships between the various parts of the recommendation model are as follows: Figure 7 As shown. Figure 8 As shown, the training objective of the recommendation model is to achieve multimodal embedding fusion for the recommendation system. The input data of the recommendation model includes user behavior sequences, sample object IDs, multimodal vectors corresponding to the user behavior sequences, and multimodal vectors of the sample objects. The input data also includes user profile features, user spatiotemporal context features, etc., which... Figure 8 Not shown in the image.

[0189] The user behavior sequence includes the IDs of objects the user has previously interacted with. The multimodal vectors corresponding to the user behavior sequence and the multimodal vectors of the sample objects are fused, and the resulting fused multimodal vector is embedded into the input multimodal interaction layer, outputting the multimodal interaction vector corresponding to the sample object. The user behavior sequence and the IDs of the sample objects are fused, and the resulting IDs are embedded into the input ID feature interaction layer, outputting the ID feature interaction vector corresponding to the sample object. A gating network is used to calculate the fusion weights corresponding to the multimodal interaction vector and the ID feature interaction vector based on the ID feature interaction vector corresponding to the sample object. Based on these fusion weights, the multimodal interaction vector corresponding to the sample object and the ID feature interaction vector with blocked gradients are fused, outputting the second sample fusion vector corresponding to the sample object. The second sample fusion vector is input into an auxiliary multilayer perceptron, outputting the second probability corresponding to the sample object. The ID feature interaction vector and the multimodal interaction vector corresponding to the sample object are input into a concatenation layer, outputting the first sample fusion vector corresponding to the sample object. The first sample fusion vector is input into the recommendation main module, outputting the first probability corresponding to the sample object.

[0190] Based on the first and second probabilities of the sample object, the target loss function value is calculated using the formula (19) above. Then, the model parameters of the recommendation model are adjusted based on the target loss function value, and training continues until the training is completed, resulting in a fully trained recommendation model.

[0191] In the embodiments of this application, the relative contribution of multimodal features is adapted to object characteristics (such as long-tail objects or popular objects). For example, the ID features of long-tail objects often suffer from data sparsity, making their multimodal features more valuable than those of popular objects. The recommendation model in the embodiments of this application can dynamically balance ID features and multimodal features to fully leverage their complementary advantages.

[0192] Some embodiments of this application also provide an information recommendation method, see [link to relevant documentation]. Figure 9 The flowchart shown illustrates the information recommendation method, which specifically includes the following steps: Step 201: Based on the current user's current query term, obtain the second candidate object set corresponding to the current query term.

[0193] Step 202: Based on the current user's user identifier and the second candidate object set, obtain the ID interaction vector corresponding to each second candidate object in the second candidate object set.

[0194] Step 203: Based on the current user's user identifier, the second candidate object set, and the second mapping relationship, obtain the multimodal interaction vectors corresponding to each second candidate object in the second candidate object set; the second mapping relationship is obtained from the model parameters of the pre-trained preset model decoder; the first mapping relationship between object identifiers and multimodal vectors is pre-configured in the model parameters before the preset model decoder is trained; the model parameters are adjusted based on user behavior prediction information and target object index during the training process of the preset model decoder; the user behavior prediction information is output by the preset model decoder based on the user's historical behavior data and the first candidate object set, and the target object index is the index of the target object to be operated in the first candidate object set; the user behavior prediction information includes the index prediction value, which refers to the index of the first candidate object predicted to be operated in the first candidate object set.

[0195] Step 204: Based on the ID interaction vector and multimodal interaction vector corresponding to each second candidate object, predict the operation probability of each second candidate object being operated by the current user.

[0196] Step 205: Based on the operation probability corresponding to each second candidate object, sort each second candidate object to obtain object recommendation information, and send the object recommendation information to the current user.

[0197] The executing entity of this method can be a server, which may include, but is not limited to, a single physical server, a server cluster consisting of multiple physical servers, or a cloud server. The executing entity of this method may be the same as or different from the executing entity of the information processing method provided above.

[0198] Steps 202 and 203 above can be executed simultaneously or in any order. Figure 9The illustration shows steps 202 and 203 being executed simultaneously. The aforementioned second mapping relationship can be an optimized mapping relationship between object identifiers and multimodal vectors derived from the parameter table / embedding table of the trained preset model decoder after training in any of the preceding embodiments. The process of obtaining the second mapping relationship can be referred to the preceding embodiments and will not be repeated here.

[0199] Among them, the ID interaction vector is a feature representation vector based on IDs. It represents the most basic and direct identifying features of the current user and the second candidate object, reflecting the statistical patterns of the user's historical behavior, such as which object IDs the current user has clicked in the past and the historical click rate of the object IDs. The ID interaction vector mainly captures mature and statistically significant collaborative signals.

[0200] Multimodal interaction vectors are multimodal feature representation vectors that represent the content features of the current user and the second candidate object, reflecting semantic understanding, such as what the store's images look like, what the text descriptions say, what image styles users pay attention to, and what keywords they follow. Multimodal interaction vectors capture visual and semantic similarity. The purpose of multimodal interaction vectors is to use content information to compensate for the insufficiency of ID behavior data (i.e., to solve the problems of cold start and poor long-tail coverage).

[0201] This embodiment predicts the probability of the current user interacting with a second candidate object based on ID interaction vectors and multimodal interaction vectors. ID interaction vectors capture the historical behavioral correlation between the current user and the second candidate object, excelling at handling popular objects with abundant historical data. Multimodal interaction vectors, on the other hand, capture the content understanding signal of the semantic and visual content correlation between the current user and the second candidate object, excelling at understanding fine-grained attributes. Simultaneously employing these two vectors for prediction achieves complementary dual signals, leveraging both the accuracy of historical statistical patterns and the richness of content features, thereby significantly improving the accuracy of operation probability prediction.

[0202] For newly listed objects (cold start) or niche objects (long tail objects), traditional models often perform poorly due to a lack of sufficient user interaction ID data. However, this application's embodiment introduces multimodal interaction vectors on top of ID interaction vectors. This allows for matching of an object's image and text features with the current user's search intent, even without historical click data, thereby improving the exposure opportunities and information recommendation diversity for long tail or cold start objects.

[0203] Since the ID interaction vector and multimodal interaction vector are calculated jointly based on the current user's user identifier and the second candidate object, the calculated ID interaction vector and multimodal interaction vector will be different for the same second candidate object for different users. This allows the recommendation model to capture the unique preferences of a specific current user for a specific second candidate object (e.g., user A focuses on image style, user B focuses on brand ID), thereby achieving highly personalized ranking.

[0204] In some embodiments of this application, the aforementioned ID interaction vector can be generated in the following manner, specifically including: obtaining the user behavior sequence corresponding to the current user based on the user identifier, wherein the user behavior sequence includes the third object identifier of the object that the current user has previously operated on; obtaining the fourth object identifier of the target candidate object, wherein the target candidate object is any second candidate object in the second candidate object set; and generating the ID interaction vector corresponding to the target candidate object based on the user behavior sequence and the fourth object identifier through a preset ID feature interaction function.

[0205] The search log can record the user's user identifier and the object identifiers of objects that the user has previously interacted with. Based on the current user's user identifier, the object identifiers of objects that the current user interacted with within a preset time period are retrieved from the current user's search log. The retrieved object identifiers are then arranged in chronological order of the user's actions to obtain the aforementioned user behavior sequence.

[0206] The ID interaction vector mentioned above can also be obtained using formula (12) mentioned earlier, and will not be repeated here. For each second candidate object in the second candidate object set, the ID interaction vector corresponding to each second candidate object is obtained in the above manner.

[0207] Traditional ID features often focus only on the "user ID" itself, which is a static attribute. However, this application's embodiment introduces user behavior sequences, which reflect which objects the user has recently interacted with. Introducing user behavior sequences allows the ranking model to perceive recent changes in user intent, breaking through the limitations of static user ID profiling and helping the recommendation model's results to quickly adapt to changes in user intent. Based on user behavior sequences and fourth object identifiers, ID interaction vectors corresponding to target candidate objects are generated, enabling the ranking model to learn complex transition patterns. For example, the ranking model might learn that "a user who ordered spicy hot pot might next order milk tea." This interaction based on ID sequences can uncover more accurate conversion logic than simple content matching.

[0208] Generating ID interaction vectors based on user behavior sequences ensures that even for the same candidate object (i.e., the fourth object identifier is the same), the generated ID interaction vectors will be completely different for different users' user behavior sequences. Therefore, predicting the operation probability of candidate objects based on the ID interaction vectors generated in this way can ensure that the predicted operation probability is highly personalized and effectively improve the prediction accuracy.

[0209] In some embodiments of this application, multimodal interaction vectors can be obtained in the following manner: based on the user identifier of the current user, obtain the third object identifiers of each object that the current user has previously operated on; based on each third object identifier, obtain the multimodal vector corresponding to each third object identifier from a second mapping relationship; based on the fourth object identifier of the target candidate object, obtain the multimodal vector corresponding to the target candidate object from the second mapping relationship; the target candidate object is any second candidate object in the second candidate object set; based on the multimodal vectors corresponding to each third object identifier and the multimodal vectors corresponding to the target candidate object, generate the multimodal interaction vector corresponding to the target candidate object through a preset multimodal feature interaction function.

[0210] The third object identifier of each object that the user has previously operated on is the object identifier of each object in the user behavior sequence mentioned above. The acquisition process is the same as the acquisition process mentioned above, and will not be repeated here.

[0211] The second mapping relationship stores the mapping relationship between object identifiers and multi-model vectors. The multi-modal vectors in the second mapping relationship can be multi-modal vectors that are first generated by the image-text alignment model trained in the previous embodiment and then optimized and adjusted during the training process of the preset model decoder in the previous embodiment.

[0212] From the second mapping relationship, obtain the multimodal vectors of each object that the user has previously operated on in the user behavior sequence, and obtain the multimodal vectors corresponding to the target candidate objects. Then, generate multimodal interaction vectors based on these multimodal vectors. The above multimodal interaction vectors can be obtained by formula (13) mentioned above, and will not be repeated here. For each second candidate object in the second candidate object set, obtain the multimodal interaction vector corresponding to each second candidate object in the above manner.

[0213] Multimodal interaction vectors are generated based on the multimodal vectors of objects previously interacted with by the user and the multimodal vectors of target candidate objects. This allows the ranking model to compare not only "user" and "object," but also the "content features of historically interacted objects" and the "content features of the current candidate object," resulting in a more granular capture of user behavior. By pre-setting multimodal feature interaction functions, the distance or correlation between the multimodal vectors of previously interacted objects and the multimodal vectors of the current candidate object in the feature space can be calculated. This captures deeper user behavioral logic more effectively than simple ID matching.

[0214] For newly listed objects or long-tail objects, although these objects may lack historical ID interaction data (i.e., no user has interacted with them), they have images and text descriptions, thus allowing for corresponding multimodal vectors. Through the above embodiments, as long as the multimodal vectors of new or long-tail objects are semantically similar to the multimodal vectors of objects previously interacted with by the user, the model can accurately predict the user's likely actions on the object. This greatly alleviates the problem of cold-start objects and long-tail objects failing to be recommended.

[0215] After obtaining the ID interaction vector and multimodal interaction vector corresponding to each second candidate object in the second candidate object set through the above method, the operation probability of each second candidate object being operated by the current user can be predicted using a pre-trained recommendation model. The recommendation model includes a concatenation layer, which concatenates the ID interaction vector and multimodal interaction vector of the second candidate object. The concatenated vector is then input into the main recommendation module of the recommendation model, which outputs the operation probability corresponding to the second candidate object.

[0216] For each second candidate object in the second candidate object set, the probability of the current user interacting with each second candidate object is predicted according to the above process. The second candidate objects are then sorted in descending order of their interaction probabilities, and this sorted set is used as the final object recommendation information. Alternatively, a certain number of the top-ranked second candidate objects in the sorted set can be used as the final object recommendation information.

[0217] In this embodiment, the image-text alignment model, the preset model decoder, and the recommendation model can be combined to obtain various information recommendation schemes. In some embodiments, the image-text alignment model and the recommendation model can also be combined to form an information recommendation scheme. In this scheme, the image-text alignment model generates multimodal vectors for each object based on the object image and object text of an object in a specific domain, and stores a first mapping relationship between the object identifier and the multimodal vector. The recommendation model searches for a second set of candidate objects based on the user's current query term. Based on the first mapping relationship, the second candidate objects in the second set are sorted to obtain object recommendation information, which is then fed back to the user.

[0218] In this embodiment, the image-text alignment model learns semantic information specific to a particular domain, aligning and fusing the visual and textual features of an object to obtain its multimodal vector. Furthermore, during the training phase, the model aligns the user's query terms with visual features, textual features, and multimodal features respectively. Therefore, the multimodal vector generated by the image-text alignment model can capture user intent. The recommendation model ranks candidate objects based on the multimodal vectors generated by the image-text alignment model. Since the multimodal vectors contain user intent, which aligns with the recommendation model's goal of ranking based on user intent, it effectively improves the accuracy of ranking, thereby enhancing the precision of information recommendation.

[0219] In other embodiments, these three elements can be combined to form a three-stage information recommendation scheme based on generative alignment learning. In this scheme, the first stage uses an image-text alignment model to generate multimodal vectors for each object based on object images and object text within a specific domain, and stores a first mapping relationship between object identifiers and multimodal vectors. In the second stage, the first mapping relationship is configured into the model parameters of a preset model decoder. The preset model decoder is trained using the user's spatiotemporal context, historical object sequences, a first candidate object set, and the target object index. During training, the first mapping relationship in the model parameters is optimized to obtain an optimized second mapping relationship. In the third stage, the recommendation model searches for a second candidate object set based on the user's current query term. Based on the second mapping relationship, each second candidate object in the second candidate object set is sorted to obtain object recommendation information, which is then fed back to the user.

[0220] In this embodiment, the image-text alignment model and the preset model decoder are trained and deployed offline, while the recommended model is trained offline and deployed in advance. Figure 10This diagram illustrates the model training and deployment of the three-stage information recommendation scheme. For offline training, the image-text alignment model (Stage 1) and the pre-defined model decoder (Stage 2) are trained to generate multimodal vectors. These multimodal vectors are then used to train the recommendation model (Stage 3). For online service, object identifiers and the multimodal vectors generated during offline training are stored as key-value pairs in a key-value table. The trained recommendation model is then deployed online. The multimodal vectors from the key-value table, user and object features stored in the database, user context features, and a candidate object set obtained based on the current user's query terms are input into the online-deployed recommendation model, outputting a sorted object list. This deployment method maintains the low-cost characteristics of "offline generation + online table lookup" throughout the entire process, only adding key-value lookup and lightweight fusion overhead, making it suitable for industrial latency budgets.

[0221] The three-stage approach described above is a domain-adaptive, generative alignment framework that comprises three stages, such as... Figure 11 As shown: Phase 1 (Domain-Adaptive Cross-Modal Alignment) constructs domain-specific query-image-text triples from large-scale user search logs and aligns the multimodal vectors using contrastive learning. This early-stage alignment captures user intent signals and constructs user-centric multimodal vectors based on real-world interaction data.

[0222] Phase 2 (Generative User Behavior Alignment) employs a generative reinforcement learning paradigm—specifically, prediction of the next object within a sequence of user behaviors. Transformation-based reward-driven continuous optimization of multimodal vectors bridges the gap between pre-training and fine-tuning, ensuring intrinsic alignment with downstream ranking objectives.

[0223] Phase 3 (Multimodal Vector Fusion for Recommendation Systems) introduces an adaptive gating mechanism to dynamically balance the contributions of ID-based and multimodal vector-based methods. An auxiliary supervision term adjusts the gating weights to maintain the effectiveness of multimodal methods during long-term ID-dominated training, thereby mitigating performance degradation.

[0224] This three-stage approach enables the unification of multimodal pre-training, behavior-driven alignment, and adaptive fusion within a unified three-stage recommendation process.

[0225] In this embodiment, a generative reinforcement learning alignment stage is introduced during the training of the pre-defined model decoder to address the mismatch between static multimodal vectors and dynamic user behavior goals. In this stage, a multimodal behavior alignment dataset is constructed based on historical user interactions and optimized directly with conversion rate rewards. This ensures that the pre-trained distribution naturally aligns with the downstream ranking goal. In the three-stage scheme described above, the first stage utilizes query-image-text triples extracted from search logs to inject user intent into the training process as early as possible, reducing the semantic gap. In the third stage, an adaptive gating mechanism with an auxiliary multilayer perceptron is employed to retain informative multimodal signals in an ID-driven training mode. This design enables the learning of fine-grained content representations that maintain semantic consistency across different modalities. Dynamic user intent is captured through behavior modeling optimized by reinforcement learning on the pre-defined model decoder, and robust feature integration is achieved through adaptive fusion of multimodal and ID features. This three-stage generative alignment framework effectively bridges the gap between content-level pre-training and behavior-driven downstream ranking goals, providing a deployable solution for multimodal recommendation scenarios.

[0226] In this embodiment, during the training of the image-text alignment model, query-image-text triples are constructed based on user search logs. Through joint training using multiple losses (query-image, query-text, and query-fusion vectors), fine-grained alignment of domain intent and content is enhanced. The first mapping relationship between object identifiers and multimodal vectors obtained through the image-text alignment model is used as part of the model parameters of the preset model decoder. This allows for the optimization and adjustment of the multimodal vectors of each object in the first mapping relationship during the training of the preset model decoder. The first training sample of the pre-defined model decoder includes user historical behavior data, which contains a wealth of information about user behavior and / or user intent. Based on this first training sample, the pre-defined model decoder generates user behavior prediction information. Then, based on this prediction information and the indexes of target objects actually interacted with by the user in the first training sample, it adjusts the multimodal vector in the first mapping relationship. This allows the adjusted multimodal vector to capture the correlations between different objects in the user's historical behavior data caused by user behavior and / or user intent. The adjusted multimodal vector contains richer user context-aware information, ensuring that the multimodal vector, having learned the inherent logic of user behavior, aligns with the subsequent ranking stage's goal of ranking based on user intent. This improves the logical reasoning ability of the recommendation system and enhances the accuracy of information recommendation. During the training of the recommendation model, gating weights are generated based on ID interactions, and gradient adjustment is performed on the ID branch. The auxiliary loss of the multimodal branch suppresses gating collapse, improving the long-term effectiveness of multimodality in the ID-dominated model. The following example from the food delivery sector will illustrate the recommendation scheme for the above three stages.

[0227] Phase 1: Domain-Adaptive Cross-Modal Alignment Training Extract store images and corresponding text fields (such as titles, categories, descriptions, etc.) and product images and corresponding text fields from structured content to form image-text pairs. Mine the interaction chain of "query term → exposure → click / order and other conversion events" from user search logs, and construct query-image-text triplet samples by combining the query term with the images and text of the stores that generated the conversion event. Construct positive samples by including the (image, text) corresponding to the target store where a user ultimately generated a conversion event under a given query term. Construct negative samples by including the (image, text) of other stores that did not generate conversion events under the given query term within the same training batch with the query term. In the process of constructing the samples described above, for each store, a set of high-confidence queries related to it can be retained. For example, a certain number of queries with the highest usage frequency can be selected, and / or, images and corresponding text fields of a certain number of products with the highest conversion rates can be selected.

[0228] It can also filter out non-core categories or non-main product categories for store content to reduce noise. For example, for a store that mainly sells hamburgers, images and corresponding text fields for products such as beverages and napkins can be filtered out.

[0229] To construct negative samples, momentum queues, cross-batch caching, and other methods can be used for negative sampling to reduce the impact of pseudo-negative samples. After constructing the sample set for training the image-text alignment model using the above method, the samples are input into the image-text alignment model. For any sample, the visual encoder in the image-text alignment model encodes the image into an image vector; the text encoder encodes the text into a text vector; and the fusion module generates a store fusion vector based on the image vector and the text vector.

[0230] During the training of the image-text alignment model, at least the following alignment objectives should be optimized simultaneously: 1. Image-text alignment loss: Maximize the similarity of matching image-text pairs and minimize the similarity of non-matching image-text pairs; 2. Query-Text Alignment Loss: Makes the query term vector similar to the text vector of the target store and dissimilar to the text vector of the negative sample.

[0231] 3. Query-Image Alignment Loss: Makes the query term vector similar to the image vector of the target store and dissimilar to the image vector of the negative sample. 4. Query-fusion vector alignment loss: Makes the query term vector similar to the fusion vector of the target store and dissimilar to the fusion vector of the negative sample. The various contrastive learning losses mentioned above can be summed by weight to obtain a store fusion vector that takes into account both user intent and content. After training the image-text alignment model in the above manner, the trained image-text alignment model is used to process the image-text data of each store on the food delivery platform to generate multimodal vectors for each store.

[0232] Phase Two: Post-Training After Generative User Behavior Alignment Phase two is used to train the pre-defined model decoder. The training task is to predict the index of the next store in the store candidate set that will generate a conversion event, based on the deliverable store candidate set. The multimodal vectors of each store generated in Phase one are configured into the model parameters of the pre-defined model decoder in Phase two so that the multimodal vectors of each store can be optimized and adjusted during the training process in this phase.

[0233] First, construct training samples for the pre-defined model decoder. The training samples include at least the following: 1. User's spatiotemporal context: including time, location, delivery-related constraints, user-side context, etc.; 2. User's historical store sequence: The sequence of stores where the user has recently interacted / placed orders. Each store in this sequence is represented by the first placeholder. 3. Candidate store set: A set of candidate stores generated by the recall module that meet the delivery and / or serviceability constraints. Each candidate store in this set is represented by a second placeholder. 4. Real Store Index: The index position of the target store where a conversion event occurred within the candidate store set. The goal of the preset model decoder is to predict the index of the next store where a conversion event will occur, but instead of directly outputting the store ID, it outputs the index of the predicted store in the candidate store set.

[0234] After obtaining training samples using the above method, a fusion prompt vector set is constructed based on the training samples. The specific construction method includes: encoding the spatiotemporal context into a context vector; replacing the first placeholder corresponding to each store in the historical store sequence with the multimodal vector generated for each store by the image-text alignment model in Phase 1; replacing the second placeholder corresponding to each candidate store in the candidate store set with the multimodal vector generated for each candidate store by the image-text alignment model in Phase 1; and combining the context vector, the historical object sequence after the replacement operation, and the candidate store set after the replacement operation into a fusion prompt vector set. The fusion prompt vector set corresponding to each training sample is used to train a preset model decoder, which can simultaneously utilize the user's spatiotemporal context and the store's multimodal vector. The fused suggestion vector set is input into the preset model decoder, and the output of the preset model decoder is the predicted index value of the store. Let the size of the candidate store set be M, and construct the index set {<index1>...<index2>}. M The final decision generated by the preset model decoder is an index in this index set, which indicates the selection of the corresponding candidate store.

[0235] In some implementations, the output of the preset model decoder may include not only the index prediction value of a store, but also inference process information to explain why the index prediction value of the store is output.

[0236] The training strategy for the pre-defined model decoder employs supervised fine-tuning and reward-driven alignment optimization. Supervised fine-tuning refers to training the pre-defined model decoder using log samples (c, S, C, y) under supervision, enabling it to learn to output the correct index prediction value given a fused cue vector set. Here, c represents the spatiotemporal context, S represents the historical store sequence, C represents the candidate store set, and y represents the true store index.

[0237] Reward-driven alignment optimization can refer to strategy optimization training based on supervised fine-tuning. The reward signal is defined by the conversion event. If the index prediction value output by the preset model decoder matches the real store index y of the actual conversion, the reward is positive (e.g., 1); otherwise, the reward signal is 0. The reward signal can be a binary conversion of 1 or 0. In some alternative embodiments, the reward signal value can also be determined by weighted summation of different conversion events. For example, different weights can be assigned to different conversion events such as clicks, orders, and add-to-cart, and the reward signal can be calculated through weighted calculation.

[0238] During the reward-driven alignment optimization process, pruning and reference policy regularization constraints are applied to policy updates to maintain training stability. After the training in Phase 2 is completed, the optimized multimodal vectors of each store are exported from the model parameters of the pre-trained model decoder and used as frozen embeddings in the subsequent ranking phase in Phase 3. In Phase Two, the multimodal vectors of stores are shifted from a static semantic space to a behavioral target space by utilizing a "set of deliverable and / or servicable candidate stores + conversion rewards". The index of a store within the candidate store set is used to avoid directly predicting large-scale dynamic store IDs, adapting to industrial-scale environments and addressing cold start / long-tail issues. The optimized multimodal vectors of each store are exported and used as frozen embeddings, maintaining low online deployment costs.

[0239] Phase 3: Adaptive Gated Fusion in the Sorting Phase During online sorting, for each candidate store, the frozen multimodal vector of the store is queried by store ID; the ID feature vector of the candidate store, the context feature vector of the user, and the ID representation of the user behavior sequence are obtained; after ID interaction modeling and multimodal interaction modeling respectively, the ID interaction vector and multimodal interaction vector corresponding to the candidate store are obtained.

[0240] The gating network in the recommendation model takes the ID interaction vector as input and generates g (0~1) gating weights for the ID interaction vector. Based on the gating weights, the ID interaction vector and the multimodal interaction vector are fused to output the fused vector corresponding to the candidate store. This design enables the recommendation model to dynamically allocate gating weights according to the richness / stability of ID information. It improves the multimodal contribution on long-tail / cold-start samples and retains the ID advantage on head samples. Based on the above fused vector, the recommendation main module of the recommendation model is used to predict the conversion probability of the candidate store and calculate the main loss L_main.

[0241] To avoid multimodal degradation caused by long-term gating bias towards the ID branch during training, an auxiliary prediction branch is introduced. The input to this auxiliary prediction branch is a concatenated vector, obtained by concatenating the multimodal interaction vector with the ID interaction vector containing the stopping gradient. The auxiliary prediction branch predicts the conversion probability of candidate stores and calculates the auxiliary loss L_aux. The total loss of the recommendation model is L = L_main + λ·L_aux, where λ can be fixed or adaptively adjusted according to the training stage / sample sparsity.

[0242] Among them, the ID interaction vector with stopping gradient ensures that the auxiliary loss mainly drives the multimodal branch learning, avoiding the ID branch from "swallowing" the gradient.

[0243] In this embodiment, the recommendation model dynamically allocates gating weights based on the ID interaction vector. Specifically, the allocation can be based on the ID interaction vector, the feature dimensions included in the ID interaction vector, or the sequence position of the ID features in the ID interaction vector. For the auxiliary multilayer perceptron, an ID interaction vector with a stopping gradient is used, where the stopping gradient can also be replaced by other gradient adjustment methods such as scaling gradient or orthogonal constraint.

[0244] Embodiments of this application also provide an information recommendation method, which is applied to a client-side application. See [link to relevant documentation]. Figure 12 The flowchart shown illustrates an information recommendation method that may include the following steps: Step 301: Send the current user's current query term to the server.

[0245] Step 302: Receive object recommendation information returned by the server based on the current query term.

[0246] The recommended information for this object is obtained through the information recommendation method provided above for the application server.

[0247] Since the object recommendation information is obtained through the recommendation model in the previous embodiment, the recommendation model uses multimodal vectors optimized and adjusted during the training of the preset model decoder during information processing. The first training sample of the preset model decoder includes user historical behavior data, which contains a large amount of information about user behavior and / or user intent. The preset model decoder generates user behavior prediction information based on the first training sample including user historical behavior data, and then adjusts the multimodal vectors in the first mapping relationship based on the user behavior prediction information and the index of target objects actually operated by the user in the first training sample. This allows the adjusted multimodal vectors to capture the correlation between different objects caused by user behavior and / or user intent in the user historical behavior data, and to contain richer user context-aware information. As a result, the multimodal vectors that have learned the internal logic of user behavior can be consistent with the goal of sorting based on user intent in the subsequent sorting stage, thereby improving the logical reasoning ability of the recommendation model and improving the accuracy of information recommendation. In the training of the recommendation model, the gating weights are generated based on ID interaction, and gradient adjustment operations are performed on the ID branch. By using auxiliary loss on the multimodal branch, gating collapse is suppressed, thereby improving the long-term effectiveness of multimodality in the ID-dominated model.

[0248] Corresponding to the methods provided in the foregoing embodiments, this application also provides an information processing apparatus, such as... Figure 13 As shown, the device includes: The first acquisition module 401 is used to acquire a first training sample, which includes user historical behavior data, a first candidate object set, and a target object index; the first candidate object set includes first candidate objects recommended based on the user's historical query terms; the target object index is the index of the target object in the first candidate object set that was operated on by the user in the first candidate object set. The training module 402 is used to train the preset model decoder based on the first training sample and the model parameters of the preset model decoder to obtain user behavior prediction information. The user behavior prediction information includes index prediction values, which refer to the index of the first candidate object predicted to be operated by the user in the first candidate object set. The model parameters are pre-configured with a first mapping relationship between object identifiers and multimodal vectors. The multimodal vectors are generated by fusing the image and text of the corresponding object. The parameter adjustment module 403 is used to adjust the model parameters based on user behavior prediction information and target object index, continue training until the first preset convergence condition is reached, and obtain the second mapping relationship between the adjusted object identifier and the multimodal vector from the model parameters after training.

[0249] The specific implementation details of each module of the device can be found in the specific operation details in the method embodiment above, and will not be repeated here.

[0250] The information processing apparatus and the information processing method provided in the above embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0251] Corresponding to the method provided in the foregoing embodiments, this application also provides an information recommendation device for executing an information recommendation method provided in the preceding embodiments. For example... Figure 14 As shown, the device includes: The second acquisition module 501 is used to acquire a second candidate object set corresponding to the current query term based on the current user's current query term; acquire the ID interaction vector corresponding to each second candidate object in the second candidate object set based on the current user's user identifier and the second candidate object set; and acquire the multimodal interaction vector corresponding to each second candidate object in the second candidate object set based on the current user's user identifier, the second candidate object set, and the second mapping relationship. The second mapping relationship is obtained from the model parameters of the pre-trained preset model decoder. A first mapping relationship between object identifiers and multimodal vectors is pre-configured in the model parameters before the preset model decoder is trained. The model parameters are adjusted based on user behavior prediction information and target object index during the training process of the preset model decoder. The user behavior prediction information is output by the preset model decoder based on the user's historical behavior data and the first candidate object set. The target object index is the index of the target object to be operated in the first candidate object set. The user behavior prediction information includes an index prediction value, which refers to the index of the predicted first candidate object to be operated in the first candidate object set. The prediction module 502 is used to predict the probability of each second candidate object being operated by the current user based on the ID interaction vector and multimodal interaction vector corresponding to each second candidate object respectively. The sorting module 503 is used to sort each second candidate object based on the operation probability corresponding to each second candidate object to obtain object recommendation information. The sending module 504 is used to send the object recommendation information to the current user.

[0252] The information recommendation device and the information recommendation method provided in the above embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0253] Figure 15 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 15As shown, the electronic device includes a memory 601 and a processor 602. The memory 601 stores a computer program that can run on the processor 602. When the processor 602 executes the computer program, it implements the method described in the above embodiments. The number of memories 601 and processors 602 can be one or more. In a specific implementation, the electronic device may also include a communication interface 603 for communicating with external devices and exchanging data.

[0254] In practical implementation, if the memory 601, processor 602, and communication interface 603 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 15 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0255] Optionally, in a specific implementation, if the memory 601, processor 602 and communication interface 603 are integrated on a single chip, the memory 601, processor 602 and communication interface 603 can communicate with each other through an internal interface.

[0256] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0257] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0258] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0259] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in this application.

[0260] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0261] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0262] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0263] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0264] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0265] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0266] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution client, apparatus, or device (such as a computer-based client, a processor-included client, or other client that can fetch and execute instructions from the instruction execution client, apparatus, or device), or in conjunction with such instruction execution client, apparatus, or device.

[0267] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution client. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0268] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0269] The above are merely exemplary embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An information processing method characterized by comprising: include: Obtain a first training sample, which includes user historical behavior data, a first candidate object set, and a target object index; The first candidate object set includes first candidate objects recommended based on the user's historical query terms; the target object index is the index of the target object operated on by the user in the first candidate object set. Based on the first training sample and the model parameters of the preset model decoder, the preset model decoder is trained to obtain user behavior prediction information. The user behavior prediction information includes an index prediction value, which refers to the index of the first candidate object that is predicted to be operated by the user in the first candidate object set. The model parameters are pre-configured with a first mapping relationship between object identifiers and multimodal vectors. The multimodal vector is generated by fusing the image and text of the corresponding object; Based on the user behavior prediction information and the target object index, the model parameters are adjusted, and training continues until the first preset convergence condition is met. The second mapping relationship between the adjusted object identifier and the multimodal vector is obtained from the model parameters after training.

2. The method of claim 1, wherein, The user's historical behavior data includes the user's spatiotemporal context and a sequence of historical objects; the sequence of historical objects includes objects that the user has previously interacted with. The step of training the preset model decoder based on the first training samples and the model parameters of the preset model decoder to obtain user behavior prediction information includes: Based on the spatiotemporal context, the historical object sequence, the first candidate object set, and the first mapping relationship, a fusion prompt vector set is generated; the fusion prompt vector set is used to represent the content features and user behavior features of objects in the vector space. The fused prompt vector set is input into the preset model decoder, and the user behavior prediction information is output.

3. The method according to claim 2, characterized in that, The step of generating a fusion prompt vector set based on the spatiotemporal context, the historical object sequence, the first candidate object set, and the first mapping relationship includes: For each object that the user has operated on in the historical object sequence, the multimodal vector of each object operated on by the user is obtained from the first mapping relationship; For each first candidate object in the first candidate object set, the multimodal vector of each first candidate object is obtained from the first mapping relationship; The spatiotemporal context is encoded into a context vector; The context vector, the multimodal vectors of each object operated by the user, and the multimodal vectors of each first candidate object are combined into the fusion prompt vector set.

4. The method according to claim 2, characterized in that, The step of inputting the fused prompt vector set into the preset model decoder and outputting the user behavior prediction information includes: The fused prompt vector set is input into the preset model decoder. The preset model decoder calculates the predicted probability of each first candidate object in the first candidate object set being operated by the user. The predicted probability is calculated based on the hidden state vector and the index embedding matrix. The hidden state vector is used to predict the index prediction value. The index embedding matrix is ​​adjusted during training based on the target object index and the index prediction value. Based on the predicted probability corresponding to each first candidate object, the index of the first candidate object with the highest predicted probability in the first candidate object set is used as the index prediction value. The user behavior prediction information is output by the preset model decoder, and the user behavior prediction information includes the index prediction value.

5. The method according to any one of claims 1-4, characterized in that, The step of adjusting the model parameters based on the user behavior prediction information and the target object index includes: Based on the index prediction value and the target object index, a reward signal is determined for reinforcement learning; the reward signal is used to quantify the degree of consistency between the index prediction value and the target object index. Based on the reward signal, calculate the advantage estimate corresponding to the index prediction value; Based on the advantage estimate, calculate the gradient of the model parameters; The model parameters are adjusted using the gradient.

6. The method according to any one of claims 1-4, characterized in that, Before training the preset model decoder based on the first training samples and the model parameters of the preset model decoder to obtain user behavior prediction information, the method further includes: Retrieve the object identifier, object text, and object image for each object; Based on the object text and object image of each object, a pre-trained image-text alignment model is used to generate multimodal vectors corresponding to each object. The first mapping relationship is obtained by correspondingly storing the object identifiers of each object and the multimodal vectors of each object; Configure the first mapping relationship into the model parameters of the preset model decoder.

7. The method according to claim 6, characterized in that, The training process of the image-text alignment model includes: Obtain a second training sample, which includes positive samples and negative samples. The positive samples include a first query term corresponding to a first sample object, a first object image of the first sample object, and first object text that semantically matches the first object image. The negative samples include multimodal data that does not match the second sample object or the second query term corresponding to the second sample object. Based on the second training sample, the query word vector, object text vector, object image vector and image-text fusion vector corresponding to the second training sample are generated through the image-text alignment model to be trained; Based on the query word vector, the object text vector, the object image vector, and the image-text fusion vector, calculate the contrast loss function value; Based on the contrast loss function value, the model parameters of the image-text alignment model to be trained are adjusted, and training continues until the second preset convergence condition is reached, thus obtaining the trained image-text alignment model.

8. The method according to any one of claims 1-4, characterized in that, After obtaining the second mapping relationship between the adjusted object identifier and the multimodal vector from the trained model parameters, the method further includes: Obtain a third training sample, which includes a sample ID interaction vector and a sample multimodal interaction vector of a third sample object; the sample multimodal interaction vector is obtained based on the second mapping relationship. Based on the sample ID interaction vector and the sample multimodal interaction vector, the first sample fusion vector of the third sample object is generated through the concatenation layer in the recommendation model to be trained. Based on the first sample fusion vector, the first probability that the third sample object will be operated by the user is predicted through the recommendation main module in the recommendation model to be trained; Based on the sample ID interaction vector and the sample multimodal interaction vector, the second probability of the third sample object being operated by the user is predicted through the auxiliary multilayer perceptron in the recommendation model to be trained. Calculate the target loss function value based on the first probability and the second probability of each of the third training samples in the current training batch; Based on the target loss function value, the model parameters of the recommendation model to be trained are adjusted, and training continues until the third preset convergence condition is met, thus obtaining the trained recommendation model.

9. The method according to claim 8, characterized in that, The step of predicting the second probability that the third sample object will be operated by the user based on the sample ID interaction vector and the sample multimodal interaction vector, using the auxiliary multilayer perceptron in the recommendation model to be trained, includes: A gradient adjustment operation is performed on the sample ID interaction vector, the gradient adjustment operation being used to control the impact on the parameter updates of the sample ID interaction vector during backpropagation; Based on the sample ID interaction vector, a first fusion weight and a second fusion weight corresponding to the sample ID interaction vector and the sample multimodal interaction vector are determined by a gating network. Based on the first fusion weight and the second fusion weight, the sample ID interaction vector and the sample multimodal interaction vector after performing gradient adjustment operation are weighted and fused to obtain the second sample fusion vector corresponding to the third sample object; The second sample fusion vector is input into the auxiliary multilayer perceptron, which outputs the second probability that the third sample object is operated by the user.

10. An information recommendation method, characterized in that, include: Based on the current user's current query term, obtain the second candidate object set corresponding to the current query term; Based on the current user's user identifier and the second candidate object set, obtain the ID interaction vector corresponding to each second candidate object in the second candidate object set; Based on the current user's user identifier, the second candidate object set, and the second mapping relationship, obtain the multimodal interaction vector corresponding to each second candidate object in the second candidate object set; The second mapping relationship is obtained from the model parameters of the pre-trained preset model decoder; Before training the preset model decoder, a first mapping relationship between object identifiers and multimodal vectors is pre-configured in the model parameters; The model parameters are adjusted based on user behavior prediction information and target object index during the training process of the preset model decoder; the user behavior prediction information is output by the preset model decoder based on user historical behavior data and a first candidate object set; the target object index is the index of the target object to be operated on in the first candidate object set; the user behavior prediction information includes an index prediction value, which refers to the index of the first candidate object predicted to be operated on in the first candidate object set. Based on the ID interaction vector and the multimodal interaction vector corresponding to each of the second candidate objects, the probability of each of the second candidate objects being operated by the current user is predicted. Based on the operation probability corresponding to each of the second candidate objects, the second candidate objects are sorted to obtain object recommendation information, and the object recommendation information is sent to the current user.

Images