A data processing method and related apparatus

Abstract

A data processing method can be applied to the field of artificial intelligence, and the method comprises: obtaining first log data and second log data of a user; the first log data comprises first attribute information of a first item, and the second log data comprises second attribute information of a second item; the occurrence time of the first log data is earlier than that of the second log data; a first feature representation is obtained by processing a first embedding corresponding to the first attribute information through a first encoder; a second feature representation is obtained by processing a second embedding corresponding to the second attribute information through a second encoder; a difference between the first feature representation and the second embedding and a difference between the second feature representation and the first embedding are used to construct a loss; and the first encoder and the second encoder are updated according to the loss. The application can improve the prediction accuracy of the model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a data processing method and related apparatus. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0003] Personalized recommendation systems play a crucial role in many online service platforms, from online advertising and online retail to music and video recommendations. To provide users with accurate and customized services, these systems attempt to recommend products that users might be interested in based on their historical interaction data. A common approach is to construct a dynamic sequence of user interactions with products and then capture the diverse and dynamic patterns of user interests using sequence models. This idea can be naturally described as a Next-Item Prediction task (i.e., predicting the next item a user might interact with), modeled using an autoregressive model.

[0004] However, in real-world scenarios, user behavior is not strictly ordered. For example, after purchasing an iPad, a user might click on a pencil, an iPad case, and headphones, but the user is likely clicking these three products randomly without a strict order. Therefore, the assumption of ordered behavior in current mainstream sequence recommendation systems leads to a loss of contextual information in user behavior modeling, because future information (interactions that occur after the user interacts with the target item) also provides rich background information to help the model train.

[0005] Recently, researchers have demonstrated that utilizing past and future contextual information during training significantly improves recommendation performance compared to autoregressive models. Inspired by advancements in natural language processing, they proposed the BERT4Rec model, based on an autoencoder sequence recommendation algorithm. Employing an MLM training approach, it predicts masked item interactions based on users' past and future interaction records. Compared to unidirectional autoregressive models (such as SASRec), BERT4Rec attempts to break the constraints of behavioral orderliness by incorporating future information into the user behavior modeling process, achieving significant results.

[0006] Although BERT4Rec incorporates both past and future information into the training process through the MLM task, it suffers from a significant training-inference gap. During training, past and future interaction records are used as context to predict masked items, while during inference, only past interaction records can be used to predict the user's next possible interaction. This contextual difference between training and inference can cause model bias during inference and lead to potential performance degradation. Summary of the Invention

[0007] This application provides a data processing method that can improve the prediction accuracy of a model.

[0008] In a first aspect, this application provides a data processing method, the method comprising: acquiring first log data and second log data of a user; the first log data including first attribute information of a first item, and the second log data including second attribute information of a second item; the occurrence time of the first log data being earlier than that of the second log data; processing a first embedding corresponding to the first attribute information through a first encoder to obtain a first feature representation; processing a second embedding corresponding to the second attribute information through a second encoder to obtain a second feature representation; the difference between the first feature representation and the second embedding, and the difference between the second feature representation and the first embedding, are used to construct a loss; and updating the first encoder and the second encoder according to the loss.

[0009] Updating the first encoder and the second encoder based on the loss constructed above enables the first encoder to predict future information based on historical information, and the second encoder to predict historical information based on future information. When updating the first encoder and the second encoder, target constraints are applied to the first encoder and the second encoder. The target constraints are used to minimize the difference between the intermediate outputs of the first encoder and the second encoder. In other words, the ability of the second encoder can be distilled into the first encoder, so that the first encoder can also have the ability to predict historical information based on future information. The updated first encoder can be used for model inference.

[0010] While existing technologies incorporate future behavioral information through MLM tasks, they also introduce significant training-inference discrepancies. Unlike existing recommendation models, this application employs two independent encoders to model past and future interaction information separately, while simultaneously promoting mutual learning between the two encoders by constraining the multi-scale interest representations captured by both encoders. This overcomes the limitation of insufficient modeling in existing technologies and improves the model's prediction accuracy.

[0011] In one possible implementation, when updating the first encoder and the second encoder, the first encoder and the second encoder are subject to a target constraint that is designed to minimize the difference between the intermediate outputs of the first encoder and the second encoder.

[0012] In one possible implementation, processing the first attribute information through the first encoder includes: processing the first attribute information and the second attribute information through the first encoder according to first self-attention information; the first self-attention information indicates that the second attribute information is masked and the first attribute information is not masked; processing the second attribute information through the second encoder includes: processing the first attribute information and the second attribute information through the second encoder according to second self-attention information; the second self-attention information indicates that the first attribute information is masked and the second attribute information is not masked.

[0013] In one possible implementation, the updated first encoder is used for model inference.

[0014] In one possible implementation, the target constraint is specifically KL divergence. This application uses KL divergence to constrain the multi-scale user interest representations captured by the past and future encoders, thereby enabling mutual learning of knowledge from past and future behaviors.

[0015] In one possible implementation, the first embedding and the second embedding are obtained by processing the first attribute information and the second attribute information respectively through the same embedding layer.

[0016] In one possible implementation, the first encoder and the second encoder include multiple attention heads, and the receptive fields corresponding to the attention information in different attention heads are different. In the self-attention mechanism calculation process, this application uses a self-attention mask to set receptive fields of different lengths for different self-attention heads, thereby capturing user interest representations at different scales.

[0017] In one possible implementation, the first attribute information and the second attribute information include at least one of the following: item name, developer, installation package size, category, and rating.

[0018] Secondly, this application provides a data processing apparatus, the apparatus comprising:

[0019] The processing module is used to acquire the user's first log data and second log data; the first log data includes first attribute information of a first item, and the second log data includes second attribute information of a second item; the occurrence time of the first log data is earlier than that of the second log data.

[0020] The first embedding corresponding to the first attribute information is processed by the first encoder to obtain the first feature representation;

[0021] The second embedding corresponding to the second attribute information is processed by the second encoder to obtain the second feature representation; the difference between the first feature representation and the second embedding, and the difference between the second feature representation and the first embedding are used to construct the loss;

[0022] An update module is used to update the first encoder and the second encoder according to the loss; and when updating the first encoder and the second encoder, the first encoder and the second encoder are subject to a target constraint, which is used to minimize the difference between the intermediate outputs of the first encoder and the second encoder.

[0023] In one possible implementation, the objective constraint is a part of the loss.

[0024] In one possible implementation, the processing module is specifically configured to process the first attribute information and the second attribute information through a first encoder based on the first self-attention information; the first self-attention information indicates that the second attribute information is masked and the first attribute information is not masked.

[0025] The processing module is specifically used to process the first attribute information and the second attribute information through the second encoder according to the second self-attention information; the second self-attention information indicates that the first attribute information is masked and the second attribute information is not masked.

[0026] In one possible implementation, the updated first encoder is used for model inference.

[0027] In one possible implementation, the objective constraint is specifically the KL divergence.

[0028] In one possible implementation, the first embedding and the second embedding are obtained by processing the first attribute information and the second attribute information respectively through the same embedding layer.

[0029] In one possible implementation, the first encoder and the second encoder include multiple attention heads, and the receptive fields corresponding to the attention information in different attention heads are different.

[0030] In one possible implementation, the first attribute information and the second attribute information include at least one of the following: item name, developer, installation package size, category, and rating.

[0031] Thirdly, embodiments of this application provide a data processing apparatus, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform any of the optional methods described in the first aspect above.

[0032] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first aspect and any optional method.

[0033] Fifthly, embodiments of this application provide a computer program product, including code, which, when executed, is used to implement the first aspect and any optional method described above.

[0034] Sixthly, this application provides a chip system including a processor for supporting a data processing device in implementing the functions involved in the foregoing aspects, such as transmitting or processing data involved in the foregoing methods; or, information. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution device or training device. This chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0035] Figure 1 A structural diagram illustrating the main framework of artificial intelligence;

[0036] Figure 2 A schematic diagram of a system architecture provided for an embodiment of this application;

[0037] Figure 3 A schematic diagram of a system architecture provided for an embodiment of this application;

[0038] Figure 4A A schematic diagram illustrating a recommended scenario provided in an embodiment of this application;

[0039] Figure 4B A schematic diagram of a network provided for an embodiment of this application;

[0040] Figure 5 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0041] Figure 6A This is a schematic diagram of a model;

[0042] Figure 6B This is a schematic diagram of a model;

[0043] Figure 6C This is a schematic diagram of a model;

[0044] Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0045] Figure 8 A schematic diagram of an execution device provided in an embodiment of this application;

[0046] Figure 9 A schematic diagram of a training device provided in an embodiment of this application;

[0047] Figure 10 This is a schematic diagram of a chip provided in an embodiment of this application. Detailed Implementation

[0048] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0049] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0050] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0051] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1 The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0052] (1) Infrastructure

[0053] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0054] (2) Data

[0055] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0056] (3) Data processing

[0057] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0058] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training of data by symbolizing and formalizing it.

[0059] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0060] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0061] (4) General ability

[0062] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0063] (5) Smart Products and Industry Applications

[0064] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0065] This application's embodiments can be applied to the field of information recommendation, including but not limited to e-commerce product recommendations, search engine result recommendations, app store recommendations, music recommendations, and video recommendations. The recommended items in various application scenarios can also be referred to as "objects" for ease of subsequent description. That is, in different recommendation scenarios, the recommended object can be an app, a video, music, or a specific product (such as the presentation interface of an online shopping platform, which displays different products based on different users; this can essentially be presented through the recommendation results of a recommendation model). These recommendation scenarios typically involve user behavior log collection, log data preprocessing (e.g., quantization, sampling), sample set training to obtain a recommendation model, and analysis and processing of the objects (such as apps, music, etc.) involved in the scenarios corresponding to the training sample items based on the recommendation model. For example, if the samples selected in the recommendation model training stage come from the user's operational behavior towards the recommended app in a mobile app store, then the recommendation model trained in this way is applicable to the aforementioned mobile app app store, or can be used for recommending terminal apps in other types of terminal app app stores. The recommendation model will eventually calculate the recommendation probability or score of each object to be recommended. The recommendation system selects the recommendation results according to certain selection rules, such as sorting them according to the recommendation probability or score. The results are then presented to the user through the corresponding application or terminal device. The user interacts with the objects in the recommendation results to generate user behavior logs, etc.

[0066] Reference Figure 4A In the recommendation process, when a user interacts with the recommendation system, a recommendation request is triggered. The system inputs this request and its related feature information into the deployed recommendation model, then predicts the click-through rate (CTR) of the user for all candidate items. Subsequently, the candidate items are sorted in descending order based on the predicted CTR and displayed sequentially in different positions as the recommendation result for the user. Users browse the displayed items and perform user actions, such as browsing, clicking, and downloading. These user actions are stored in logs as training data, and the parameters of the recommendation model are periodically updated through an offline training module to improve the model's recommendation performance.

[0067] For example, when a user opens the app store, the recommendation module is triggered. This module predicts the likelihood of the user downloading a given set of candidate apps based on the user's download history, click history, app characteristics, and environmental factors such as time and location. Based on the predictions, the app store displays apps in descending order of probability, thus increasing the likelihood of app downloads. Specifically, apps more likely to be downloaded are listed first, while those less likely are listed last. User behavior is also logged and used to train and update the prediction model's parameters through an offline training module.

[0068] For example, in applications related to lifelong partners, historical data from users across domains such as video, music, and news can be used to construct a cognitive brain, mimicking the mechanisms of the human brain, and build a framework for a lifelong learning system. Lifelong partners can record past events based on system and application data, understand current intentions, predict future actions or behaviors, and ultimately provide intelligent services. In the current first phase, user behavior data (including information such as SMS messages, photos, and email events) obtained from music apps, video apps, and browser apps is used to build a user profile system and implement learning and memory modules based on user information filtering, association analysis, cross-domain recommendation, and causal reasoning to construct a personal knowledge graph for each user.

[0069] The application architecture of this application embodiment will be described next.

[0070] See appendix Figure 2This invention provides a recommendation system architecture 200. A data acquisition device 260 is used to collect samples. A training sample can consist of multiple feature information (or described as attribute information, such as user attributes and item attributes). The feature information can be of various types, specifically including user feature information, object feature information, and tag features. User feature information is used to characterize user characteristics, such as gender, age, occupation, hobbies, etc. Object feature information is used to characterize the features of the objects pushed to the user. Different recommendation systems correspond to different objects, and the types of features to be extracted for different objects are also different. For example, the object features extracted from the training samples of an app market can be the app's name (identifier), type, size, etc. The object features mentioned in the training samples of e-commerce apps can include the product name, category, price range, etc. Tag features are used to indicate whether a sample is a positive or negative example. Typically, the tag features of a sample can be obtained from the user's actions on the recommended object. Samples where the user has performed actions on the recommended object are positive examples, while samples where the user has not performed actions or has only browsed the recommended object are negative examples. For example, if a user clicks, downloads, or purchases a recommended object, the tag feature is 1, indicating that the sample is a positive example; if the user has not performed any actions on the recommended object, the tag feature is 0, indicating that the sample is a negative example. After collection, the samples can be stored in database 230. Some or all of the feature information in the samples in database 230 can also be directly obtained from the client device 240, such as user feature information, user action information on objects (used to determine type identification), and object feature information (such as object identification). The training device 220 trains and obtains a model parameter matrix based on the samples in database 230 to generate a recommendation model 201 (e.g., the feature extraction network and neural network in this embodiment). The following describes in more detail how the training device 220 trains to obtain the model parameter matrix used to generate the recommendation model 201. The recommendation model 201 can be used to evaluate a large number of objects to obtain the score of each object to be recommended. Furthermore, it can recommend a specified or preset number of objects from the evaluation results of a large number of objects. The calculation module 211 obtains the recommendation results based on the evaluation results of the recommendation model 201 and recommends them to the client device through the I / O interface 212.

[0071] In this embodiment, the training device 220 can select positive and negative samples from the sample set in the database 230 and add them to the training set. Then, the recommendation model is used to train the samples in the training set to obtain the trained recommendation model. The implementation details of the calculation module 211 can be found in [reference needed]. Figure 5 A detailed description of the method embodiments shown.

[0072] After training the model parameter matrix based on the samples, the training device 220 uses it to construct the recommendation model 201 and then sends the recommendation model 201 to the execution device 210. Alternatively, the model parameter matrix can be directly sent to the execution device 210, where the recommendation model is constructed for use in the corresponding system. For example, a recommendation model trained based on video-related samples can be used to recommend videos to users on video websites or apps, while a recommendation model trained based on app-related samples can be used to recommend apps to users in app stores.

[0073] The execution device 210 is equipped with an I / O interface 212 for data interaction with external devices. The execution device 210 can obtain user characteristic information from the client device 240 through the I / O interface 212, such as user identifier, user identity, gender, occupation, and hobbies. This information can also be obtained from the system database. The recommendation model 201 recommends target objects to the user based on the user characteristic information and the characteristic information of the objects to be recommended. The execution device 210 can be located on a cloud server or on the user client.

[0074] The execution device 210 can access data, code, etc., in the data storage system 250, and can also store output data into the data storage system 250. The data storage system 250 can be located within the execution device 210, can be set up independently, or can be located in other network entities; there can be one or multiple such systems.

[0075] The calculation module 211 uses the recommendation model 201 to process the user feature information and the feature information of the object to be recommended. For example, the calculation module 211 uses the recommendation model 201 to analyze and process the user feature information and the feature information of the object to be recommended, thereby obtaining the score of the object to be recommended. The objects to be recommended are sorted according to the score, and the objects ranked higher will be recommended to the client device 240.

[0076] Finally, I / O interface 212 returns the recommendation results to client device 240 and presents them to the user.

[0077] At a deeper level, the training device 220 can generate corresponding recommendation models 201 based on different sample feature information for different targets, so as to provide users with better results.

[0078] It is worth noting that, attached Figure 2 This is merely a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationships between the devices, components, modules, etc. shown in the diagram do not constitute any limitation. For example, in the attached diagram... Figure 2In this context, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 may also be placed within the execution device 210.

[0079] In this embodiment, the training device 220, the execution device 210, and the client device 240 may be three different physical devices. Alternatively, the training device 220 and the execution device 210 may be on the same physical device or a cluster, or the execution device 210 and the client device 240 may be on the same physical device or a cluster.

[0080] See appendix Figure 3 This is a system architecture 300 proposed in an embodiment of the present invention. In this architecture, the execution device 210 is implemented by one or more servers, optionally in conjunction with other computing devices, such as data storage, routers, load balancers, etc. The execution device 210 can be deployed on a single physical site or distributed across multiple physical sites. The execution device 210 can use data in the data storage system 250 or call program code in the data storage system 250 to implement the object recommendation function. Specifically, the information of the object to be recommended is input into the recommendation model. The recommendation model generates an estimated score for each object to be recommended, and then sorts them in descending order of the estimated scores. The object to be recommended is then recommended to the user according to the sorting result. For example, the top 10 objects in the sorting result are recommended to the user.

[0081] The data storage system 250 is used to receive and store the parameters of the recommendation model sent by the training device, as well as the data for storing the recommendation results obtained through the recommendation model. It may also include the program code (or instructions) required for the normal operation of the storage system 250. The data storage system 250 can be a distributed storage cluster consisting of one or more devices deployed outside the execution device 210. In this case, when the execution device 210 needs to use data on the storage system 250, the storage system 250 can send the required data to the execution device 210, and the execution device 210 receives and stores (or caches) the data. Alternatively, the data storage system 250 can be deployed within the execution device 210. When deployed within the execution device 210, the distributed storage system can include one or more storage devices. Optionally, when multiple storage devices exist, different storage devices are used to store different types of data. For example, the model parameters of the recommendation model generated by the training device and the recommendation results obtained through the recommendation model can be stored on two different storage devices.

[0082] Users can interact with execution device 210 by operating their respective user devices (e.g., local device 301 and local device 302). Each local device can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, smart camera, smart car or other type of cellular phone, media consumption device, wearable device, set-top box, game console, etc.

[0083] Each user's local device can interact with the execution device 210 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0084] In another implementation, execution device 210 can be implemented by a local device. For example, local device 301 can implement the recommendation function of execution device 210 based on the recommendation model to obtain user feature information and provide recommendation results to the user, or provide services to the user of local device 302.

[0085] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0086] 1. Click-through rate (CTR)

[0087] Click probability, also known as click-through rate, refers to the ratio of the number of clicks to the number of impressions of recommended information (e.g., recommended items) on a website or application. Click-through rate is usually an important metric for evaluating recommendation systems.

[0088] 2. Personalized Recommendation System

[0089] A personalized recommendation system refers to a system that analyzes a user's historical data (such as the operation information in the embodiments of this application) using machine learning algorithms, and uses this data to predict new requests and provide personalized recommendation results.

[0090] 3. Offline training

[0091] Offline training refers to a module in a personalized recommendation system that iteratively updates the parameters of the recommendation model according to the algorithm learned by the machine learning, based on the user's historical data (such as the operation information in the embodiments of this application), until the set requirements are met.

[0092] 4. Online Inference

[0093] Online prediction refers to using a model trained offline to predict a user's preference for recommended items in the current context, based on the characteristics of the user, the item, and the context, and to predict the probability that the user will choose the recommended item.

[0094] For example, Figure 3 This is a schematic diagram of the recommendation system provided in an embodiment of this application. For example... Figure 3 As shown, when a user enters the system, a recommendation request is triggered. The recommendation system inputs this request and its related information (such as the operation information in this embodiment) into the recommendation model, and then predicts the user's selection rate for items within the system. Further, the items are sorted in descending order based on the predicted selection rate or a function based on that selection rate; that is, the recommendation system can display items in different positions sequentially as recommendations to the user. The user browses items in different positions and performs user actions, such as browsing, selecting, and downloading. Simultaneously, the user's actual behavior is stored in a log as training data, and the parameters of the recommendation model are continuously updated through an offline training module to improve the model's predictive performance.

[0095] For example, a user opening the app store on a smart device (e.g., a mobile phone) triggers the app store's recommendation system. The app store's recommendation system predicts the probability of the user downloading each recommended candidate app based on the user's historical behavior logs, such as historical download records and user selection records, as well as the app store's own characteristics, such as environmental features like time and location. Based on the calculation results, the app store's recommendation system can display candidate apps in descending order of predicted probability values, thereby increasing the download probability of candidate apps.

[0096] For example, apps with a predicted high user selection rate can be displayed at the top of the recommendation list, while apps with a predicted low user selection rate can be displayed at the bottom.

[0097] The recommended model mentioned above can be a neural network model. The following is an introduction to the relevant terms and concepts of neural networks that may be involved in the embodiments of this application.

[0098] (1) Neural Network

[0099] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0100]

[0101] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0102] (2) Deep Neural Networks

[0103] Deep Neural Networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is therefore quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows: It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0104] (3) Loss Function

[0105] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0106] (4) Backpropagation algorithm

[0107] Backpropagation (BP) can be used during training to correct the parameters in the initial model, thereby reducing the model's error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters in the initial model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining optimal model parameters, such as the weight matrix.

[0108] (5) Machine Learning Systems

[0109] Based on input data and labels, the parameters of a machine learning model are trained using optimization methods such as gradient descent, and the trained model is then used to predict unknown data.

[0110] (6) Personalized recommendation system

[0111] This system uses machine learning algorithms to analyze and model users' historical data, and then uses this data to predict new user requests and provide personalized recommendations.

[0112] (7) Transformer layer

[0113] Reference Figure 4B , Figure 4B This is a schematic diagram of a transformer layer architecture, such as Figure 4B As shown, the neural network includes an embedding layer and at least one transformer layer. The at least one transformer layer can be N transformer layers (N being an integer greater than 0). Each transformer layer includes sequentially adjacent attention layers, add and normalize layers, feed-forward layers, and add and normalize layers. In the embedding layer, the current input is embedded to obtain multiple feature vectors. In the attention layer, P input vectors are obtained from the layer above the transformer layer. Using any first input vector among the P input vectors as the center, based on the correlation between each input vector within a preset attention window and the first input vector, an intermediate vector corresponding to the first input vector is obtained. This process determines P intermediate vectors corresponding to the P input vectors. In the pooling layer, the P intermediate vectors are merged into Q output vectors, where the multiple output vectors obtained from the last transformer layer in at least one transformer layer are used as feature representations of the current input.

[0114] Next, we will explain each of the above steps in detail with specific examples.

[0115] First, in the embedding layer, the current input is embedded to obtain multiple feature vectors.

[0116] The embedding layer can be called the input embedding layer. The current input can be text, such as a paragraph or a sentence. The text can be Chinese, English, or other languages. After obtaining the current input, the embedding layer can perform embedding processing on each word in the current input to obtain the feature vector of each word. In some embodiments, such as... Figure 4BAs shown, the embedding layer includes an input embedding layer and a positional encoding layer. In the input embedding layer, word embedding processing can be performed on each word in the current input to obtain word embedding vectors for each word. In the positional encoding layer, the position of each word in the current input can be obtained, and then position vectors can be generated for the positions of each word. In some examples, the position of each word can be the absolute position of each word in the current input. Taking the current input "When should Huabei be repaid?" as an example, the position of "When" can be represented as the first position, the position of "should" can be represented as the second position, and so on. In some examples, the position of each word can be the relative position between each word. Still taking the current input "When should Huabei be repaid?" as an example, the position of "When" can be represented as before "should", the position of "should" can be represented as after "When" and before "be", and so on. When the word embedding vectors and position vectors of each word in the current input are obtained, the position vectors of each word can be combined with the corresponding word embedding vectors to obtain word feature vectors, that is, multiple feature vectors corresponding to the current input are obtained. The multiple feature vectors can be represented as an embedding matrix with a preset dimension. It can be set that the number of feature vectors in the multiple feature vectors is M, and the preset dimension is H dimensions, then the multiple feature vectors can be represented as an M×H embedding matrix.

[0117] Secondly, P input vectors are obtained from the layer above the first transformer layer. Taking any first input vector among the P input vectors as the center, based on the correlation degree between each input vector within the preset attention window range and the first input vector, an intermediate vector corresponding to the first input vector is obtained, and thus P intermediate vectors corresponding to the P input vectors are determined. The attention layer can also be referred to as a multi-head attention layer. In one example, the attention layer can be a fixed window multi-head attention layer.

[0118] In some embodiments, the first transformer layer can be the layer below the above embedding layer, and the P input vectors are the multiple feature vectors obtained from the embedding layer. In some embodiments, at least one transformer layer in the neural network provided in the embodiments of this specification further includes a second transformer layer. The second transformer layer is the layer above the first self-attention, and the P input vectors are the P output vectors output by the second transformer layer. In the last transformer layer in the neural network, the multiple output vectors obtained through the above steps can be used as the feature representation of the current input. This feature representation is a feature representation suitable for computer processing of the current input.

[0119] (8) Attention mechanism

[0120] Attention mechanisms mimic the internal processes of biological observation—aligning internal experience with external senses to increase the precision of observation in specific areas. They enable the rapid sifting of high-value information from a large volume of data using limited attentional resources. Attention mechanisms can quickly extract important features from sparse data and are therefore widely used in natural language processing tasks, particularly machine translation. Self-attention mechanisms, an improvement on attention mechanisms, reduce reliance on external information and are better at capturing the internal correlations of data or features. The core idea of ​​attention mechanisms can be rewritten as follows:

[0121] In this formula, Lx = ||Source|| represents the length of the Source. The meaning is that the elements in the Source are imagined as a series of data pairs. Given a Query element in the Target, the similarity or relevance between the Query and each Key is calculated to obtain the weight coefficient of the Value corresponding to each Key. Then, the Values ​​are weighted and summed to obtain the final Attention value. Therefore, the Attention mechanism essentially performs a weighted sum of the Values ​​of the elements in the Source, while the Query and Key are used to calculate the weight coefficients of their corresponding Values. Conceptually, Attention can be understood as selectively filtering a small amount of important information from a large amount of information and focusing on this important information, ignoring most of the unimportant information. The focusing process is reflected in the calculation of the weight coefficients; the larger the weight, the more focused it is on its corresponding Value. That is, the weight represents the importance of the information, and the Value is the corresponding information. Self-attention can be understood as intra attention. The attention mechanism occurs between the elements of the Target (Query) and all elements of the Source. Self-attention refers to the attention mechanism that occurs between elements within the Source or between elements within the Target. It can also be understood as the attention calculation mechanism in the special case where Target = Source. The specific calculation process is the same, only the calculation object changes.

[0122] Personalized recommendation systems play a crucial role in many online service platforms, from online advertising and online retail to music and video recommendations. To provide users with accurate and customized services, these systems attempt to recommend products that users might be interested in based on their historical interaction data. A common approach is to construct a dynamic sequence of user interactions with products and then capture the diverse and dynamic patterns of user interests using sequence models. This idea can be naturally described as a Next-Item Prediction task (i.e., predicting the next item a user might interact with), modeled using an autoregressive model.

[0123] However, in real-world scenarios, user behavior is not strictly ordered. For example, after purchasing an iPad, a user might click on a pencil, an iPad case, and headphones, but the user is likely clicking these three products randomly without a strict order. Therefore, the assumption of ordered behavior in current mainstream sequence recommendation systems leads to a loss of contextual information in user behavior modeling, because future information (interactions that occur after the user interacts with the target item) also provides rich background information to help the model train.

[0124] Recently, researchers have demonstrated that utilizing past and future contextual information during training significantly improves recommendation performance compared to autoregressive models. Inspired by advancements in natural language processing, they proposed the BERT4Rec model, based on an autoencoder sequence recommendation algorithm. Employing an MLM training approach, it predicts masked item interactions based on users' past and future interaction records. Compared to unidirectional autoregressive models (such as SASRec), BERT4Rec attempts to break the constraints of behavioral orderliness by incorporating future information into the user behavior modeling process, achieving significant results.

[0125] Although BERT4Rec incorporates both past and future information into the training process through the MLM task, it suffers from a significant training-inference gap. During training, past and future interaction records are used as context to predict masked items, while during inference, only past interaction records can be used to predict the user's next possible interaction. This contextual difference between training and inference can cause model bias during inference and lead to potential performance degradation.

[0126] To address the aforementioned problems, this application provides a data processing method.

[0127] Reference Figure 5 , Figure 5 An embodiment of a data processing method provided in this application is illustrated, such as... Figure 5As shown, an embodiment of this application provides a data processing method, including:

[0128] 501. Obtain the user's first log data and second log data; the first log data includes the first attribute information of the first item, and the second log data includes the second attribute information of the second item; the occurrence time of the first log data is earlier than that of the second log data.

[0129] In this embodiment of the application, the execution subject of step 501 can be a terminal device, which can be a portable mobile device, such as, but not limited to, mobile or portable computing devices (such as smartphones), personal computers, server computers, handheld devices (such as tablets) or laptop devices, multiprocessor systems, game consoles or controllers, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, mobile computing and / or communication devices with wearable or accessory form factors (such as watches, glasses, headphones or earphones), network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0130] In this embodiment of the application, the entity executing step 501 can be a cloud-side server. The server can receive user operation data sent from the terminal device, and thus the server can obtain the user's operation data.

[0131] For ease of description, the form of the executing entity will not be distinguished below, and will be described as training device.

[0132] In one possible implementation, when training the target neural network, it is necessary to obtain training samples. Taking a recommendation model as an example, the training samples may include attribute information of users and items, and the attribute information may be user operation data.

[0133] The user's operation data can be obtained based on the interaction records between the user and the items (such as the user's behavior log). This operation data can include the user's actual operation records on each item, and can include the user's attribute information, the attribute information of each item, and the operation type of the user's operation on the multiple items (such as clicking, downloading, etc.).

[0134] The user's attribute information can be attributes related to the user's preferences, such as at least one of gender, age, occupation, income, hobbies, and education level. Gender can be male or female, age can be a number between 0 and 100, occupation can be teacher, programmer, chef, etc., hobbies can be basketball, tennis, running, etc., and education level can be primary school, junior high school, high school, university, etc. This application does not limit the specific type of user attribute information.

[0135] The items can be physical or virtual, such as applications (APPs), audio and video, web pages, and news information. The attribute information of the items can be at least one of the following: item name, developer, installation package size, category, and rating. For example, if the item is an application, the category can be chat, parkour game, office, etc., and the rating can be a score or comment on the item. This application does not limit the specific type of attribute information of the items.

[0136] In one possible implementation, a user's first log data and a second log data can be obtained; the first log data includes first attribute information of a first item, and the second log data includes second attribute information of a second item; the occurrence time of the first log data is earlier than that of the second log data. The first item may include one or more items, and the second item may include one or more items.

[0137] In a specific scenario, the items that a user has interacted with can form an item sequence x = [x1, x2, x3, ..., x...]. n In this sequence, the first item is the item preceding the second item.

[0138] In one possible implementation, the attribute information of items in the sequence can first be mapped into low-dimensional continuous representation vectors through an embedding layer. Specifically, the first attribute information of the first item is mapped to a first embedding, and the second attribute information of the second item is mapped to a second embedding. Then, the low-dimensional representation sequence x of the items is used as input to the model. The model may include a first encoder and a second encoder.

[0139] 502. The first embedding corresponding to the first attribute information is processed by the first encoder to obtain the first feature representation.

[0140] 503. The second embedding corresponding to the second attribute information is processed by the second encoder to obtain the second feature representation; the difference between the first feature representation and the second embedding, and the difference between the second feature representation and the first embedding are used to construct the loss.

[0141] In one possible implementation, the first encoder and the second encoder may include multiple transformer layers connected in series.

[0142] Next, we will introduce an example of an attention network:

[0143] In one possible implementation, the transformer-based encoder may include sequentially connected embedding layers and multiple transformer layers. The number of transformer layers can be set as needed. The encoder determines the feature vector corresponding to the current node based on the N output vectors obtained from each transformer layer.

[0144] In the embedding layer, the current input is embedded to obtain multiple feature vectors. The core feature of the transformer model lies in its unique attention mechanism. (See reference...) Figure 6B , Figure 6B This is a schematic diagram of a transformer layer structure. The transformer layers of various neural networks in the embodiments of this application can be referenced from this diagram. Figure 6B The structure shown in the figure, such as Figure 6B As shown, the transformer layer consists of a multi-head attention layer, an add & normalization layer, a feed forward layer, and another add & normalization layer, which are sequentially adjacent to each other.

[0145] The multi-head attention layer obtains N input vectors X from the layer above it. l This can also be represented as matrix X. Employing a self-attention mechanism, it transforms (or interacts with) each vector based on the correlation between them, resulting in N output vectors, which can also be represented as matrix Y. It can be understood that when this multi-head attention layer is directly connected to the embedding layer, for example... Figure 6B In a transformer layer directly connected to the embedding layer, the input vector it receives is the embedding vector output by the embedding layer; when this multi-head attention layer is a multi-head attention layer included in subsequent transformer layers, for example... Figure 6B The transformer layer directly connected to the previous transformer layer includes a multi-head attention layer, whose input vector is the output vector of the previous transformer layer. In the multi-head attention layer, the MHA layer includes multiple attention heads (e.g., ...). Figure 6C The following are Head 1, Head 2, ..., Head N shown in the figure.

[0146] Figure 6C This is a schematic diagram illustrating the operation of an attention head, showing how the attention head transforms an input matrix X into an output matrix Y. For example... Figure 6CAs shown, the first transformation matrix Q, the second transformation matrix K, and the third transformation matrix V are applied to N input vectors respectively.<X1,X2,…,XN> The input vectors Xi are transformed to obtain the first intermediate vector (q vector), second intermediate vector (k vector), and third intermediate vector (v vector) corresponding to each input vector. Operationally, the input matrix X, composed of N input vectors, can be linearly transformed using the first transformation matrix Q, the second transformation matrix K, and the third transformation matrix V, respectively, to obtain the Q matrix, K matrix, and V matrix of the input matrix. These matrices are then split to obtain the q vector, k vector, and v vector corresponding to each input vector. For any i-th input vector Xi among the N input vectors, the correlation degree between the i-th input vector Xi and each input vector Xj is determined based on the dot product operation between the first intermediate vector (q vector, qi) corresponding to the i-th input vector and each second intermediate vector (k vector, kj) corresponding to each input vector Xj. Although the dot product result of qi and kj can be directly used to determine the correlation degree, a more classic approach is to first divide the dot product result by a constant, then perform a softmax operation, and use the result as the correlation degree between the input vector Xi and Xj (that is, the correlation degree between the Q vector and the K vector), i.e.:

[0147]

[0148] Therefore, the correlation degree α between the i-th input vector Xi and each of the input vectors Xj can be used as the basis for determining the correlation degree α between the i-th input vector Xi and each of the input vectors Xj. i,j As a weighting factor, the third intermediate vector (v vector, vj) corresponding to each input vector Xj is weighted and combined to obtain the i-th combined vector Ci corresponding to the i-th input vector Xi:

[0149]

[0150] Therefore, we can obtain a vector sequence of N combined vectors corresponding to N input vectors.<C1,C2,…,CN> Or matrix C. Based on this combined vector sequence, N output vectors can be obtained. Specifically, in one embodiment, the vector sequence of N combined vectors can be directly used as N output vectors, i.e., Yi = Ci. In this case, the output matrix Y is the combined vector matrix C, which can also be written as:

[0151]

[0152] The above describes the processing flow of an attention head. In the MHA architecture, the MHA layer maintains m sets of transformation matrices. Each set of transformation matrices includes the aforementioned first transformation matrix Q, second transformation matrix K, and third transformation matrix V, allowing the above operations to be performed in parallel to obtain m combined vector sequences (i.e., m matrices C). Each vector sequence includes N combined vectors obtained based on a set of transformation matrices. In this case, the MHA layer concatenates the m combined vector sequences to obtain a concatenated matrix; then, it transforms this concatenated matrix using the fourth transformation matrix W to obtain the final output matrix Y. This output matrix Y can be split into N output vectors.<Y1,Y2,…,YN> Through the above operations, the MHA layer performs transformation operations based on the correlation between the N input vectors to obtain N output vectors.

[0153] In one possible implementation, processing the first attribute information through the first encoder includes: processing the first attribute information and the second attribute information through the first encoder according to first self-attention information; the first self-attention information indicates that the second attribute information is masked and the first attribute information is not masked; processing the second attribute information through the second encoder includes: processing the first attribute information and the second attribute information through the second encoder according to second self-attention information; the second self-attention information indicates that the first attribute information is masked and the second attribute information is not masked.

[0154] In one possible implementation, a first embedding corresponding to the first attribute information can be processed by a first encoder to obtain a first feature representation; a second embedding corresponding to the second attribute information can be processed by a second encoder to obtain a second feature representation; the difference between the first feature representation and the second embedding, and the difference between the second feature representation and the first embedding, are used to construct the loss.

[0155] Updating the first and second encoders based on the constructed loss allows the first encoder to predict future information based on historical information, and the second encoder to predict historical information based on future information. Furthermore, when updating the first and second encoders, a target constraint is applied to them. This target constraint minimizes the difference in intermediate outputs between the first and second encoders, effectively distilling the capabilities of the second encoder into the first encoder, enabling the first encoder to also predict historical information based on future information. The updated first encoder can then be used for model inference.

[0156] To fully utilize future information while mitigating potential training-inference discrepancies, embodiments of this application propose a dual network model (including a first encoder and a second encoder), such as... Figure 6A As shown, the dual network employs two independent encoders to model past and future user behaviors, respectively, while sharing the same embedding layer. The encoders utilize a Transformer architecture, employing self-attention masks to ensure that behavioral information is processed from left to right or right to left; in other words, the self-attention masks of the two encoders are also dual. Regarding the training task, the target tasks of the two encoders are also dual: the encoder processing past interaction information predicts the next possible interaction item (original task), while the encoder responsible for future interaction information predicts the previous possible interaction item before the occurrence of the behavioral sequence (dual task). In this way, future information is modeled separately, explicitly decoupled from the modeling of past information.

[0157] In one possible implementation, the first encoder and the second encoder include multiple attention heads, and the receptive fields corresponding to the attention information in different attention heads are different. In the self-attention mechanism calculation process, this application uses a self-attention mask to set receptive fields of different lengths for different self-attention heads, thereby capturing user interest representations at different scales.

[0158] 504. Update the first encoder and the second encoder according to the loss; and when updating the first encoder and the second encoder, the first encoder and the second encoder are subject to a target constraint, the target constraint being used to minimize the difference between the intermediate outputs of the first encoder and the second encoder.

[0159] In one possible implementation, when updating the first encoder and the second encoder, the first encoder and the second encoder are subject to a target constraint that is designed to minimize the difference between the intermediate outputs of the first encoder and the second encoder.

[0160] In one possible implementation, the target constraint is specifically KL divergence. This application uses KL divergence to constrain the multi-scale user interest representations captured by the past and future encoders, thereby enabling mutual learning of knowledge from past and future behaviors.

[0161] In one possible implementation, the objective constraint may also be a part of the loss.

[0162] The core device in the embodiments of this application can be as follows: Figure 6AAs shown, it consists of dual networks and a bi-directional information transferring (BLT). The two encoders in the dual networks are responsible for modeling past and future behavioral information, respectively, and they share the same set of embedding layer parameters. The BLT uses KL divergence to achieve multi-scale representation distillation between past and future information, thereby enhancing the utilization of future contextual information.

[0163] Dual networks explicitly decouple the processing of past and future information in a direct manner, but this also affects the mutual learning between the two encoders to some extent. In fact, the user preferences captured by the two encoders corresponding to the past and future are often related or even complementary, so mutual learning between the two encoders can further improve model performance. Furthermore, in real-world scenarios, user interests are often dynamic and multi-scale, as there are often both stable long-term interests and dynamic short-term interests. Therefore, this invention employs a multi-scale multi-head self-attention mechanism to capture the user's multi-scale interests and uses KL divergence to constrain the multi-scale user interest representations captured in the past and future encoders, thereby promoting mutual learning between the two. The multi-scale multi-head self-attention mechanism captures multi-scale interests through different receptive field lengths in the self-attention mask, such as... Figure 6A As shown in (c)(d) in the diagram.

[0164] The following is a specific example of a data processing method in an embodiment of this application. In a specific scenario, the products that a user has interacted with form a product sequence x = [x1, x2, x3, ..., x...]. n This technical solution can predict, based on a user's interaction history data, the product that the user is most likely to interact with in n+1 interactions under a given type of interaction.

[0165] Specifically, the user's item sequence is first mapped into a low-dimensional continuous representation vector through an embedding layer. Then, the low-dimensional item representation sequence x is used as the input to the model, and the overall dual network processes the data to obtain the potential interactive items for the user's next target action.

[0166] The specific process of dual networks can be divided into the following steps:

[0167] (1) Input the product sequence x into the shared Embedding layer to obtain the product low-dimensional representation sequence e. Then, perform self-attention operations on each pair of positions in the product low-dimensional representation sequence e from left to right (processing past information) or from right to left (processing future information) through two encoders, one for the past and one for the future, to update the product representation.

[0168] (2) Apply layer normalization and residual connection to the updated product representation sequence to ensure gradient stability.

[0169] (3) The low-dimensional representation of commodities learns behavioral-specific semantic information through a behavior-specific feedforward neural network.

[0170] (4) Finally, the low-dimensional representation of the goods obtained by the output of the dual network is used to perform dual training tasks for the two encoders.

[0171] The dual network described above decouples the processing of past and future behaviors, but this also hinders mutual learning between the two. Introducing constraints can facilitate the transfer of knowledge between the two encoders during training. The specific process can be divided into the following steps:

[0172] (1) In the self-attention mechanism calculation process, a self-attention mask is used to set different receptive fields of different lengths for different self-attention heads, thereby capturing user interest representations of different scales.

[0173] (2) KL divergence is used to constrain the multi-scale user interest representations captured by the past and future encoders, thereby enabling mutual learning of knowledge in past and future behaviors.

[0174] This application's embodiments have undergone extensive experimentation on multiple publicly available datasets for recommending user behavior sequences. The experimental setup is as follows:

[0175] Performance was evaluated using the Amazon Beauty, Sports, Toys, and Yelp datasets.

[0176] Use industry-recognized testing metrics (HR, the higher the better), (NDCG, the higher the better), and (MRR, the higher the better).

[0177] Several existing technologies were compared:

[0178] (1) Sequence recommendation models: Caser, GRU4CTR, HGN, RepeatNet, SASRec, BERT4Rec, and FMLP-Rec

[0179] (2) Graphical models: SRGNN and GCSAN

[0180] (3) Comparative learning models: S3-Rec and CLEA

[0181] Table 1 compares the performance of the recommendations. Bold text indicates the best results, and underlined text indicates the second-best results (i.e., the best baseline). "*" indicates a statistically significant improvement compared to the best baseline (i.e., p-value < 0.05).

[0182] Table 1

[0183]

[0184] The following conclusions can be drawn from the experiment:

[0185] In terms of recommendation accuracy, this invention achieved the best experimental results in the three metrics of Hit Rate, NDCG, and MRR, demonstrating the significant effectiveness of the sequence recommendation system proposed in this invention.

[0186] Furthermore, the modules in this invention were migrated as plug-ins to existing work, and a compatibility analysis experiment was conducted. The results are shown in Table 2 below.

[0187] Table 2

[0188]

[0189] Applying the solutions from this invention to existing work can significantly improve the effectiveness of existing work, which further proves the effectiveness and universality of this invention.

[0190] While existing technologies incorporate future behavioral information through MLM tasks, they also introduce significant training-inference discrepancies. Unlike existing recommendation models, this application employs two independent encoders to model past and future interaction information separately, while simultaneously promoting mutual learning between the two encoders by constraining the multi-scale interest representations captured by both encoders. This overcomes the limitation of insufficient modeling in existing technologies and improves the model's prediction accuracy.

[0191] The following describes a data processing apparatus provided in the embodiments of this application from the perspective of the apparatus itself, referring to... Figure 7 , Figure 7 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application, such as... Figure 7 As shown, an embodiment of this application provides a data processing apparatus 700 comprising:

[0192] Processing module 701 is used to acquire first log data and second log data from the user; the first log data includes first attribute information of a first item, and the second log data includes second attribute information of a second item; the occurrence time of the first log data is earlier than that of the second log data.

[0193] The first embedding corresponding to the first attribute information is processed by the first encoder to obtain the first feature representation;

[0194] The second embedding corresponding to the second attribute information is processed by the second encoder to obtain the second feature representation; the difference between the first feature representation and the second embedding, and the difference between the second feature representation and the first embedding are used to construct the loss;

[0195] For a detailed description of the processing module 701, please refer to the description of steps 501 to 503 in the above embodiments, which will not be repeated here.

[0196] The update module 702 is used to update the first encoder and the second encoder according to the loss; and when updating the first encoder and the second encoder, the first encoder and the second encoder are subject to a target constraint, which is used to minimize the difference between the intermediate outputs of the first encoder and the second encoder.

[0197] For a detailed description of the update module 702, please refer to the description of step 504 in the above embodiment, which will not be repeated here.

[0198] In one possible implementation, the objective constraint is a part of the loss.

[0199] In one possible implementation, the processing module is specifically configured to process the first attribute information and the second attribute information through a first encoder based on the first self-attention information; the first self-attention information indicates that the second attribute information is masked and the first attribute information is not masked.

[0200] The processing module is specifically used to process the first attribute information and the second attribute information through the second encoder according to the second self-attention information; the second self-attention information indicates that the first attribute information is masked and the second attribute information is not masked.

[0201] In one possible implementation, the updated first encoder is used for model inference.

[0202] In one possible implementation, the objective constraint is specifically the KL divergence.

[0203] In one possible implementation, the first embedding and the second embedding are obtained by processing the first attribute information and the second attribute information respectively through the same embedding layer.

[0204] In one possible implementation, the first encoder and the second encoder include multiple attention heads, and the receptive fields corresponding to the attention information in different attention heads are different.

[0205] In one possible implementation, the first attribute information and the second attribute information include at least one of the following: item name, developer, installation package size, category, and rating.

[0206] The following describes an execution device provided in an embodiment of this application. Please refer to [link / reference]. Figure 8 , Figure 8This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 800 can specifically be a mobile phone, tablet, laptop, smart wearable device, server, etc., and is not limited thereto. The execution device 800 implements... Figure 5 The data processing method in the corresponding embodiment has functionalities. Specifically, the execution device 800 includes: a receiver 801, a transmitter 802, a processor 803, and a memory 804 (wherein the execution device 800 may have one or more processors 803), wherein the processor 803 may include an application processor 8031 ​​and a communication processor 8032. In some embodiments of this application, the receiver 801, transmitter 802, processor 803, and memory 804 may be connected via a bus or other means.

[0207] Memory 804 may include read-only memory and random access memory, and provides instructions and data to processor 803. A portion of memory 804 may also include non-volatile random access memory (NVRAM). Memory 804 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0208] The processor 803 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus but also power buses, control buses, and status signal buses. However, for clarity, all buses in the diagram are referred to as the bus system.

[0209] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 803. Processor 803 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 803 or by instructions in software form. The processor 803 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor or microcontroller, a vision processing unit (VPU), a tensor processing unit (TPU), or other processors suitable for AI computation. It may further include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. This processor 803 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 804, and processor 803 reads the information in memory 804 and, in conjunction with its hardware, completes steps 501 to 503 in the above embodiments.

[0210] Receiver 801 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 802 can be used to output digital or character information through the first interface; transmitter 802 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 802 may also include a display device such as a display screen.

[0211] This application also provides a training device; please refer to [link / reference]. Figure 9 , Figure 9This is a schematic diagram of a training device provided in an embodiment of this application. Specifically, the training device 900 is implemented by one or more servers. The training device 900 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 99 (e.g., one or more processors) and a memory 932, and one or more storage media 930 (e.g., one or more mass storage devices) for storing application programs 942 or data 944. The memory 932 and storage media 930 can be temporary or persistent storage. The program stored in the storage media 930 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the training device. Furthermore, the CPU 99 may be configured to communicate with the storage media 930 and execute the series of instruction operations in the storage media 930 on the training device 900.

[0212] The training device 900 may also include one or more power supplies 926, one or more wired or wireless network interfaces 950, one or more input / output interfaces 958; or one or more operating systems 941, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0213] Specifically, the training device can perform steps 501 to 503 in the above embodiments.

[0214] This application also provides a computer program product that, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0215] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0216] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0217] For details, please refer to Figure 10 , Figure 10 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1000. The NPU 1000 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1003, which is controlled by the controller 1004 to extract matrix data from the memory and perform multiplication operations.

[0218] The NPU 1000 achieves this through the cooperation of its various internal components. Figure 5 The data processing method provided in the described embodiments.

[0219] More specifically, in some implementations, the arithmetic circuitry 1003 within the NPU 1000 includes multiple processing engines (PEs). In some implementations, the arithmetic circuitry 1003 is a two-dimensional pulsating array. The arithmetic circuitry 1003 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuitry 1003 is a general-purpose matrix processor.

[0220] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1002 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1001 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1008.

[0221] Unified memory 1006 is used to store input and output data. Weight data is directly transferred to weight memory 1002 via Direct Memory Access Controller (DMAC) 1005. Input data is also transferred to unified memory 1006 via DMAC.

[0222] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1009.

[0223] The Bus Interface Unit (BIU) 1010 is used by the instruction fetch memory 1009 to fetch instructions from external memory, and also by the memory access controller 1005 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0224] The DMAC is mainly used to move input data from external memory DDR to unified memory 1006, or to weight data to weight memory 1002, or to input data to input memory 1001.

[0225] The vector computation unit 1007 includes multiple processing units that further process the output of the computation circuit 1003 when needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0226] In some implementations, the vector computation unit 1007 can store the processed output vector in the unified memory 1006. For example, the vector computation unit 1007 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1003, such as performing linear interpolation on the feature planes extracted by the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1007 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 1003, for example, for use in subsequent layers of the neural network.

[0227] The instruction fetch buffer 1009 connected to the controller 1004 is used to store the instructions used by the controller 1004;

[0228] Unified memory 1006, input memory 1001, weight memory 1002, and instruction fetch memory 1009 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0229] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0230] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0231] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0232] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0233] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data processing method, characterized in that, The method includes: Obtain the user's first log data and second log data; the first log data includes the first attribute information of a first item, and the second log data includes the second attribute information of a second item; the first log data occurred earlier than the second log data. The first embedding corresponding to the first attribute information is processed by the first encoder to obtain the first feature representation; The second encoder processes the second embedding corresponding to the second attribute information to obtain the second feature representation; the self-attention mask of the first encoder and the self-attention mask of the second encoder are dual to each other, and the difference between the first feature representation and the second embedding, as well as the difference between the second feature representation and the first embedding, are used to construct the loss. The first encoder and the second encoder are updated based on the loss; and when updating the first encoder and the second encoder, a target constraint is applied to the first encoder and the second encoder to minimize the difference between the intermediate outputs of the first encoder and the second encoder.

2. The method according to claim 1, characterized in that, The target constraint is a part of the loss.

3. The method according to claim 1 or 2, characterized in that, The step of processing the first attribute information through the first encoder includes: processing the first attribute information and the second attribute information through the first encoder according to the first self-attention information; the first self-attention information indicates that the second attribute information is masked and the first attribute information is not masked; The step of processing the second attribute information through the second encoder includes: processing the first attribute information and the second attribute information through the second encoder according to the second self-attention information; the second self-attention information indicates that the first attribute information is masked and the second attribute information is not masked.

4. The method according to any one of claims 1 to 2, characterized in that, The updated first encoder is used for model inference.

5. The method according to any one of claims 1 to 2, characterized in that, The target constraint is specifically the KL divergence.

6. The method according to any one of claims 1 to 2, characterized in that, The first embedding and the second embedding are obtained by processing the first attribute information and the second attribute information respectively through the same embedding layer.

7. The method according to any one of claims 1 to 2, characterized in that, The first encoder and the second encoder include multiple attention heads, and the receptive fields corresponding to the attention information in different attention heads are different.

8. The method according to any one of claims 1 to 2, characterized in that, The first attribute information and the second attribute information include at least one of the following: item name, developer, installation package size, category, and rating.

9. A data processing apparatus, characterized in that, The device includes: The processing module is used to acquire the user's first log data and second log data; the first log data includes first attribute information of a first item, and the second log data includes second attribute information of a second item; the occurrence time of the first log data is earlier than that of the second log data. The first embedding corresponding to the first attribute information is processed by the first encoder to obtain the first feature representation; The second encoder processes the second embedding corresponding to the second attribute information to obtain the second feature representation; the self-attention mask of the first encoder and the self-attention mask of the second encoder are dual to each other, and the difference between the first feature representation and the second embedding, as well as the difference between the second feature representation and the first embedding, are used to construct the loss. An update module is used to update the first encoder and the second encoder according to the loss; and when updating the first encoder and the second encoder, the first encoder and the second encoder are subject to a target constraint, which is used to minimize the difference between the intermediate outputs of the first encoder and the second encoder.

10. The apparatus according to claim 9, characterized in that, The target constraint is a part of the loss.

11. The apparatus according to claim 9 or 10, characterized in that, The processing module is specifically used to process the first attribute information and the second attribute information through the first encoder according to the first self-attention information; the first self-attention information indicates that the second attribute information is masked and the first attribute information is not masked; The processing module is specifically used to process the first attribute information and the second attribute information through the second encoder according to the second self-attention information; the second self-attention information indicates that the first attribute information is masked and the second attribute information is not masked.

12. The apparatus according to any one of claims 9 to 10, characterized in that, The updated first encoder is used for model inference.

13. The apparatus according to any one of claims 9 to 10, characterized in that, The target constraint is specifically the KL divergence.

14. The apparatus according to any one of claims 9 to 10, characterized in that, The first embedding and the second embedding are obtained by processing the first attribute information and the second attribute information respectively through the same embedding layer.

15. The apparatus according to any one of claims 9 to 10, characterized in that, The first encoder and the second encoder include multiple attention heads, and the receptive fields corresponding to the attention information in different attention heads are different.

16. The apparatus according to any one of claims 9 to 10, characterized in that, The first attribute information and the second attribute information include at least one of the following: item name, developer, installation package size, category, and rating.

17. A computing device, characterized in that, The computing device includes a memory and a processor; the memory stores code, and the processor is configured to retrieve the code and execute the method as described in any one of claims 1 to 8.

18. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 8.

19. A computer program product, comprising code, characterized in that, When the code is executed, it is used to implement the method as described in any one of claims 1 to 8.

Images