A data processing method and related apparatus

Abstract

This application proposes a data processing method and related apparatus for selecting the embedding size of input feature data based on the information-compressed embedding vector. The method includes: converting input feature data into an N-dimensional first embedding vector, where the input feature data includes user behavior data and / or item feature data; inferring from the N-dimensional first embedding vector using a neural network model to obtain an N-dimensional second embedding vector, where the information content of the i-th dimension in the N-dimensional second embedding vector is negatively correlated with the value of i; determining the target dimension of each input feature data based on a preset total dimension and the N-dimensional second embedding vector of each input feature data, where the preset total dimension is the sum of the vector dimensions allocated to each input feature data, and the target dimension is used to obtain a target embedding vector, which is the embedding vector of the input feature data converted to the target dimension. The target embedding vector is used by the recommendation system to predict the first prediction result to be recommended to the user.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a data processing method and related apparatus. Background Technology

[0002] With the rapid development of the internet, the amount of information online has increased dramatically, leading to information overload for users. To help users find content of interest amidst this vast sea of ​​information, recommendation systems have emerged. These systems analyze users' historical behavior and interests to provide personalized recommendations, thereby improving user experience and platform profitability. In recommendation systems, embedding techniques are used to transform user behaviors, items, and other entities into vectors. The recommendation system model then learns from these vector features to obtain recommendations, achieving personalized feedback. Embedding size is a crucial parameter in embedding techniques, determining the dimension of the transformed vector. The selection and adjustment of the embedding size significantly impact model performance.

[0003] Currently, the selection of embedding size usually requires generating multiple embedding vectors of different dimensions for each input feature, and then selecting the embedding size of the input feature based on the weights of the embedding vectors of different dimensions, or using a pruning-based method to prune the feature dimensions of the input feature, thereby reducing the dimension of the embedding vector of the input feature.

[0004] However, when it is necessary to increase the dimension of candidate features, a new embedding vector needs to be generated for each input feature, and multiple embedding vectors of different dimensions need to be generated for each input feature, which will bring computational and storage burdens. Summary of the Invention

[0005] This application provides a data processing method and related apparatus for selecting the embedding size of input feature data based on the information-compressed embedding vector.

[0006] In a first aspect, this application provides a data processing method, which includes: firstly, converting input feature data into an N-dimensional first embedding vector, wherein the input feature data includes user behavior data and / or item feature data, and N is an integer greater than or equal to 2; subsequently, inferring from the N-dimensional first embedding vector using a neural network model to obtain an N-dimensional second embedding vector, wherein the information content of the i-th dimension in the N-dimensional second embedding vector is negatively correlated with the value of i, and i is less than or equal to N; then, determining the target dimension of each input feature data according to a preset total dimension and the N-dimensional second embedding vector of each input feature data, wherein the preset total dimension is the sum of the vector dimensions allocated to each input feature data, and the target dimension is used to obtain a target embedding vector, which is an embedding vector of the input feature data converted into the target dimension, and the target embedding vector is used by the recommendation system to predict the first prediction result to be recommended to the user.

[0007] In this embodiment, the input feature data can be transformed into an N-dimensional embedding vector. The target dimension of the input feature data can then be determined based on this N-dimensional embedding vector. This eliminates the need to first transform the input feature data into embedding vectors corresponding to each candidate dimension and store them, thus reducing storage pressure. Furthermore, by compressing the representational information of the input feature data as much as possible into the beginning of the embedding vector, and gradually reducing the amount of information from the beginning to the end, the input feature data can be optimized to select the smallest possible dimension based on the preset total dimension, while retaining the maximum representational information.

[0008] In one possible implementation, before reasoning about the N-dimensional first embedding vector through a neural network model to obtain the N-dimensional second embedding vector, the method may further include: randomly truncating the N-dimensional first embedding vector to obtain multiple sub-embedded vectors of different dimensions.

[0009] In this embodiment, it is only necessary to convert the input feature data into an N-dimensional first embedding vector, and then obtain multiple embedding vectors of different dimensions by randomly truncating the N-dimensional first embedding vector. It is not necessary to store the embedding vector of each candidate dimension of the input feature data, but only the N-dimensional first embedding vector needs to be stored, thereby reducing the storage space occupied and reducing storage pressure.

[0010] In one possible implementation, the method may further include: transforming multiple sub-embedded vectors of different dimensions into multiple M-dimensional third embedded vectors through a mapping matrix, wherein the multiple M-dimensional third embedded vectors are the first embedded vectors.

[0011] In this embodiment, since the input dimension of the neural network model is fixed, sub-embedded vectors of different dimensions can be mapped to third embedded vectors of the same dimension through a mapping matrix, which facilitates the subsequent representation learning of embedded vectors of the same dimension by the neural network model.

[0012] In one possible implementation, the aforementioned determination of the target dimension of each input feature data based on the preset total dimension and the N-dimensional second embedding vector of each input feature data may include: calculating the contribution of the first K-dimensional embedding vectors in the N-dimensional second embedding vector to the second prediction result, obtaining N contribution values, wherein the second prediction result is obtained by the recommendation system through reasoning on the N-dimensional second embedding vector; and determining the target dimension of each input feature data based on the preset total dimension and the N contribution values ​​of each input feature data.

[0013] In one possible implementation, the aforementioned calculation of the contribution of the first K-dimensional embedding vectors in the N-dimensional second embedding vector to the second prediction result, resulting in N contribution values, may include: using feature attribution to calculate the contribution of the first K-dimensional embedding vectors in the N-dimensional second embedding vector to the second prediction result, resulting in N contribution values.

[0014] In this embodiment, feature attribution is a primary method for explaining black-box neural network models, and can be used to explain the importance of input features to network prediction. Feature attribution is used to calculate the contribution of the first K dimensions of the N-dimensional second embedding vector to the prediction result, providing a basis for subsequently selecting the embedding size of the input feature data.

[0015] In one possible implementation, the aforementioned determination of the target dimension of each input feature data based on the preset total dimension and the N contribution values ​​of each input feature data may include: determining the target dimension of each input feature data using an integer programming algorithm based on the preset total dimension and the N contribution values ​​of each input feature data.

[0016] In this embodiment, an integer programming algorithm can be used to determine the target embedding vector dimension of each input feature data under the premise of a preset total dimension, so as to select the smallest possible embedding size for each input feature data without affecting the model inference.

[0017] Secondly, this application provides a data processing method, comprising: a transformation module for transforming input feature data into an N-dimensional first embedding vector, wherein the input feature data includes user behavior data and / or item feature data, and N is an integer greater than or equal to 2; an inference module for inferring from the N-dimensional first embedding vector using a neural network model to obtain an N-dimensional second embedding vector, wherein the information content of the i-th dimension in the N-dimensional second embedding vector is negatively correlated with the value of i, and i is less than or equal to N; and a determination module for determining the target dimension of each input feature data based on a preset total dimension and the N-dimensional second embedding vector of each input feature data, wherein the preset total dimension is the sum of the vector dimensions allocated to each input feature data, the target dimension is used to obtain a target embedding vector, the target embedding vector is the embedding vector of the input feature data transformed into the target dimension, and the target embedding vector is used by the recommendation system to predict the first prediction result to be recommended to the user.

[0018] In one possible implementation, before inferring the N-dimensional first embedding vector through the neural network model to obtain the N-dimensional second embedding vector, the aforementioned inference module is also used to randomly truncate the N-dimensional first embedding vector to obtain multiple sub-embedded vectors of different dimensions.

[0019] In one possible implementation, the aforementioned inference module is further configured to transform multiple sub-embedded vectors of different dimensions into multiple M-dimensional third embedded vectors through a mapping matrix, wherein the multiple M-dimensional third embedded vectors are the first embedded vectors.

[0020] In one possible implementation, the aforementioned determining module is specifically used to: calculate the contribution of the first K dimensions of the N-dimensional second embedding vector to the second prediction result, and obtain N contribution values, wherein the second prediction result is obtained by the recommendation system through reasoning on the N-dimensional second embedding vector; and determine the target dimension of each input feature data according to the preset total dimension and the N contribution values ​​of each input feature data.

[0021] In one possible implementation, the aforementioned determining module is specifically used to: calculate the contribution of the first K-dimensional embedding vectors in the N-dimensional second embedding vector to the second prediction result using feature attribution method, and obtain N contribution values.

[0022] In one possible implementation, the aforementioned determining module is specifically used to: determine the target dimension of each input feature data using an integer programming algorithm based on the preset total dimension and the N contribution values ​​of each input feature data.

[0023] Thirdly, this application provides a data processing apparatus, which includes: a processor, a memory, an input / output device, and a bus; the memory stores computer instructions; when the processor executes the computer instructions in the memory, the memory stores computer instructions; when the processor executes the computer instructions in the memory, it is used to implement a method as described in the first aspect or any possible implementation of the first aspect.

[0024] Fourthly, embodiments of this application provide a computer-readable storage medium. The computer-readable storage medium stores computer instructions; when the computer instructions are executed on a computer, the computer causes the computer to perform the method as described in the first aspect or any possible implementation thereof.

[0025] Fifthly, embodiments of this application provide a computer program product. This computer program product includes a computer program or instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect or any possible implementation thereof.

[0026] The technical effects of the second to fifth aspects or any of their possible implementations can be found in the first aspect or the related possible implementations of the first aspect, and will not be repeated here. Attached Figure Description

[0027] Figure 1 This application uses a schematic diagram of an artificial intelligence main framework.

[0028] Figure 2 A schematic diagram of a system architecture is provided for this application;

[0029] Figure 3 This is a schematic diagram of the architecture of a recommendation system;

[0030] Figure 4 This is a flowchart illustrating a data processing method proposed in an embodiment of this application;

[0031] Figure 5 This is a schematic diagram illustrating the process of randomly truncating and mapping N, which is the first embedding vector.

[0032] Figure 6 The contribution of each input feature data to the prediction result in different dimensions;

[0033] Figure 7 This is a schematic diagram of the structure of a data processing device proposed in an embodiment of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0035] The method provided in this application can be applied to artificial intelligence (AI) scenarios. AI is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to have the functions of perception, reasoning, and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0036] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1 The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0037] (1) Infrastructure

[0038] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0039] (2) Data

[0040] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0041] (3) Data processing

[0042] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0043] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0044] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0045] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0046] (4) General ability

[0047] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0048] (5) Smart Products and Industry Applications

[0049] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0050] To facilitate understanding, some concepts involved in this application will be introduced first.

[0051] 1. Recommendation System

[0052] The recommendation system uses machine learning algorithms to analyze and learn from users' historical click behavior data, then predicts new user requests and returns a personalized list of recommended items.

[0053] 2. Embedding

[0054] Embedding is a technique that maps high-dimensional data to a low-dimensional space, and it is widely used in fields such as Natural Language Processing (NLP), computer vision, recommender systems, and social network analysis. It involves mapping high-dimensional data (such as text, images, and audio) to a low-dimensional vector space through mathematical transformations, enabling more efficient data processing and analysis. These low-dimensional vectors can capture the inherent features and relationships in the original data that are difficult to measure and compute directly.

[0055] 3. Embedding size

[0056] Embedding size refers to the size or dimension of a vector when converting discrete features (such as user IDs, item IDs, etc.) into a vector representation. In machine learning and deep learning, embedding size determines the amount of information and complexity contained in the vector. For example, in recommender systems, user IDs and item IDs are often represented in the form of embeddings so that the model can process these discrete features.

[0057] 4. Characteristic Attribution Method

[0058] Feature attribution is a machine learning explanation technique. It aims to evaluate the contribution of each input feature to the model's prediction. It is primarily used to explain the model's predictions for a given input and attribute them to the features of that input. The attribution is proportional to the feature's contribution to the prediction.

[0059] 5. Integer Programming Algorithm

[0060] Integer programming algorithms are algorithms used to solve linear programming or broader mathematical programming problems, requiring all or some of the variables in the solution vector to take integer values. Integer programming algorithms have wide applications in industrial and engineering design, scientific research, computer design, system reliability, coding, and economic analysis. For example, they can be used to solve combinatorial optimization problems such as the backpack problem, fixed-cost problems, location problems, and investment problems.

[0061] To make the technical solution of this application clearer and easier to understand, the system architecture of this application will be described below with reference to the accompanying drawings.

[0062] See Figure 2 This application provides a system architecture 200. As shown in the system architecture 200, the data acquisition device 260 can be used to collect training data. After the data acquisition device 260 collects the training data, it stores the training data in the database 230. The training device 220 trains the neural network model 201 based on the training data maintained in the database 230.

[0063] It should be noted that in practical applications, the training data maintained in database 230 may not all come from the data acquisition device 260; it may also be received from other devices. Furthermore, it should be noted that the training device 220 may not necessarily train the neural network model 201 entirely based on the training data maintained in database 230; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0064] The calculation module 211 includes a neural network model 201 and a determination module 202. The data processing method mentioned in this application can be executed by the calculation module 211. The neural network model 201 is used to compress the information content of the embedding vector as much as possible into the head of the embedding vector. The neural network model 201 can infer an N-dimensional second embedding vector from an N-dimensional first embedding vector, where the information content of each dimension in the N-dimensional second embedding vector gradually decreases as the dimension increases. For example, the information content of the first dimension in the second embedding vector is greater than the information content of the second dimension. After obtaining the information-compressed second embedding vector, the determination module 202 can determine the target dimension of each input feature vector based on the preset total dimension and the second embedding vector of each input feature data. The data processing method proposed in this application embodiment can be applied to different devices, such as those used in… Figure 2 The execution device 210 shown can be a terminal device such as a mobile phone, watch, wristband, tablet computer, and vehicle-mounted device, or a device with computing capabilities such as a CPU, GPU, and FPGA. Figure 2 In this embodiment, the execution device 210 is equipped with a transceiver 212, which may include an input / output (I / O) interface or other wireless or wired communication interfaces for data interaction with external devices. Taking the I / O interface as an example, the user can input data to the I / O interface through the client device 240. For example, in the following embodiments of this application, the client device 240 can send the required input feature data to the execution device 210, requesting the execution device 210 to output the target dimension of the input feature data. This target dimension is used to obtain the target embedding vector. Subsequently, multiple target embedding vectors can be input into the recommendation system 213 to predict the first prediction result to be recommended to the user.

[0065] During the preprocessing of input data by the execution device 210, or during the calculation module 211 of the execution device 210 performing calculations and other related processes, the execution device 210 can call data, code, etc. in the data storage system 250 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 250.

[0066] Finally, I / O interface 212 returns the processing result to client device 240, thereby providing it to the user.

[0067] It is worth noting that, attached Figure 2 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 2 In this context, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 may also be placed within the execution device 210.

[0068] The data processing method proposed in this application can be applied to machine learning-based scenarios, such as recommendation systems and natural language processing. For example, a click-through rate (CTR) prediction scenario in a recommendation system will be used as an example. Click-through rate prediction is a typical scenario in machine learning applications, and its main structure is as follows... Figure 3 As shown, it includes a display list, logs, an offline training module, and an online prediction module.

[0069] Recommendation systems first acquire user behavior data, such as browsing, clicking, commenting, and downloading, and store this data in logs. The system then uses this data, including user behavior logs, for offline model training. After training converges, a prediction model is generated. This model is deployed in an online service environment and provides recommendation results based on user requests, item features, and contextual information. Users then provide feedback on these recommendations, forming user data.

[0070] In the recommendation system process, the system can record thousands of user and item features. Before predicting user and item feature data using a prediction model, these features are typically encoded into embedding vectors, which are then input into the prediction model to estimate the user click probability. However, due to the limited computing resources of online services, the data processing method proposed in this application can be used to determine the embedding size of the input features for model inference, thereby reducing the computational load of the online service model and improving the model's inference efficiency.

[0071] The method flow provided in this application will be described below in conjunction with the aforementioned system architecture.

[0072] See Figure 4 The following is a flowchart illustrating a data processing method provided in this application.

[0073] Step 401: Convert the input feature data into an N-dimensional first embedding vector;

[0074] The data processing method provided in this application can be applied not only to recommendation systems, but also to various machine learning-based application scenarios such as image recognition and natural language processing, without being limited to any specific application here. For example, in a recommendation system, the input feature data can be user behavior feature data and item feature data, while in the field of image recognition, the input feature data can be image features, and in the field of natural language processing, the input feature data can be text or other data.

[0075] In this embodiment, the input feature data can first be converted into an N-dimensional first embedding vector, where N can be the maximum dimension set according to user needs. This means that an embedding vector of maximum length N is allocated to the input feature data to store its representational information, so that embedding vectors of other dimensions can be obtained subsequently based on this first embedding vector. For example, when N is 30, each input feature data can be converted into a 30-dimensional first embedding vector.

[0076] In this embodiment of the application, the input feature data is transformed into an N-dimensional first embedding vector, where the value of N is set to determine the search range for the dimension search of the embedding vector, and the target dimension of the input feature data can be determined from 1 to N dimensions.

[0077] Step 402: Infer the N-dimensional first embedding vector through a neural network model to obtain the N-dimensional second embedding vector;

[0078] In this context, the information content of the i-th dimension in the N-dimensional second embedding vector is negatively correlated with the value of i, meaning that as the dimension increases, the information content of the i-th dimension in the N-dimensional second embedding vector decreases.

[0079] Optionally, before obtaining the N-dimensional second embedding vector, the N-dimensional first embedding vector can be randomly truncated to obtain multiple sub-embedding vectors, which are embedding vectors of multiple different dimensions. For example, randomly truncating the N-dimensional first embedding vector can yield multiple sub-embedding vectors such as 1-dimensional sub-embedding vector, ..., K-dimensional sub-embedding vector, ...

[0080] In this embodiment, the input feature data can be converted into only an N-dimensional first embedding vector. By randomly truncating the first embedding vector, multiple embedding vectors of different dimensions can be obtained for model inference, thus avoiding the storage burden caused by generating embedding vectors of different dimensions from the input feature data.

[0081] Subsequently, since the input to the neural network model is of fixed dimension, the resulting multiple sub-embedding vectors can be transformed into multiple M-dimensional third embedding vectors using a mapping matrix. This mapping matrix is ​​an N×M matrix used to map sub-embedding matrices of different dimensions to embedding vectors of the same dimension. The value of M is typically set by the user based on the value of N. The process of randomly truncating and mapping the first embedding vector (where N is the first embedding vector) is as follows: Figure 5 As shown. By Figure 5 It can be seen that after truncating each input feature data and mapping it into an embedding vector of the same dimension, the embedding vectors of multiple input feature data can be combined to obtain sample data, and the obtained sample data can be input into the neural network model for inference.

[0082] Optionally, after obtaining multiple M-dimensional third embedding vectors, an N-dimensional second embedding vector can be obtained by reasoning through the multiple M-dimensional third embedding vectors using a neural network model. This allows the important information of the input feature data to be compressed into the head of the N-dimensional second embedding vector, and the amount of information in each dimension gradually decreases from the head to the tail.

[0083] Before using a neural network model to infer the N-dimensional first embedding vector, the neural network model needs to be trained so that it can infer the information-compressed N-dimensional second embedding vector from the N-dimensional first embedding vector.

[0084] During the training of a neural network model, multiple N-dimensional sample embedding vectors can be obtained from multiple sample feature data. In each training round, the N-dimensional sample embedding vectors can be randomly truncated to obtain multiple sub-sample embedding vectors of different dimensions. Since the input dimension of the neural network model is fixed, the sub-sample embedding vectors of different dimensions can be mapped to multiple sub-embedded vectors of the same dimension through a mapping matrix. Since each sub-embedded vector includes information from the first dimension of the sample embedding vector, the neural network model updates the weights of the first dimension of the sample embedding vector most frequently in each training round, followed by updating the weights of the second dimension. Therefore, the first dimension has the greatest impact on the output performance of the neural network model, followed by the second dimension. Consequently, when the neural network model learns the representation information from the sample embedding vectors, the information is first placed in the first dimension, then the second dimension, and so on, with the Nth dimension containing the least amount of information. Therefore, the trained neural network model can compress the information of the input N-dimensional first embedding vector to obtain an N-dimensional second embedding vector. The amount of information in each dimension of the N-dimensional second embedding vector gradually decreases as the dimension increases, so as to compress the representation information of the embedding vector into a small number of dimensions as much as possible.

[0085] Step 403: Determine the target dimension of each input feature data based on the preset total dimension and the N-dimensional second embedding vector of each input feature data.

[0086] After compressing the N-dimensional first embedding vector to obtain the N-dimensional second embedding vector, the vector dimension of each input feature data can be searched according to the preset total dimension and the vector dimension range of each input feature data, and finally the target dimension of each input feature data is determined.

[0087] Optionally, the contribution of the first K dimensions of the N-dimensional second embedding vector to the second prediction result or the loss can be calculated, resulting in N contribution values. Here, the prediction result is obtained by inference from the N-dimensional second embedding vector through the recommender system, and the loss is the difference between the predicted result and the actual result. Subsequently, the target dimension of each input feature data can be determined based on the preset total dimension and the N feature values ​​of each input feature data. Here, the second prediction result is the prediction result obtained by inference from the N-dimensional second embedding vector through the recommender system.

[0088] Since feature attribution is mainly used to explain the model's prediction of a given input and attribute it to each input feature, it can be used to evaluate the contribution of each input feature to the model's prediction result. In this embodiment, the contribution of the first K dimensions of the N-dimensional second embedding vector to the second prediction result can be calculated by feature attribution. Then, based on the contribution value, a basis can be provided for the selection of the embedding vector dimension of the input feature data, thereby determining the target dimension of each input feature data.

[0089] Feature attribution methods include gradient-based attribution algorithms, layer-by-layer backpropagation-based attribution algorithms, and occlusion-based attribution algorithms, which are not specifically limited here. For example, a gradient-based attribution algorithm can be used to calculate the contribution of the first K dimensions of the N-dimensional second embedding vector to the prediction result, obtaining N contribution values. For instance, as... Figure 6 As shown, Figure 6 This represents the contribution of each input feature data point to the prediction result across different dimensions. Figure 5 As can be seen, at this time, the input feature data includes three feature data: secondary classification, gender, and age. For each input feature data, the contribution value of the input feature data to the prediction result under different dimensions is calculated.

[0090] After obtaining N contribution values ​​for each input feature, an integer programming algorithm can be used to determine the target dimension of each input feature data based on the preset total dimension and the N contribution values ​​of each input feature data. The preset total dimension is the total number of dimensions that can be allocated to each input feature data according to user requirements.

[0091] Integer programming algorithms are commonly used to solve combinatorial optimization problems. An objective function is constructed, and an integer programming algorithm is used to find the maximum value of this objective function under constraints. In this embodiment, the objective function can represent the sum of the contributions of each input feature data to the prediction result, and the constraints can be defined as the sum of the dimensions assigned to each input feature data as a preset total dimension. The target dimension of each input feature data satisfying this constraint is then calculated.

[0092] For example, when the preset total dimension is 4, the input feature data are binary classification, age, and gender, respectively. The contribution of different dimensions of the input feature data is as follows: Figure 5 As shown in the diagram, the possible embedding size combinations for the input feature data are: 1-dimensional secondary classification assignment, 1-dimensional age assignment with 2-dimensional gender assignment, 1-dimensional secondary classification assignment with 2-dimensional age assignment with 1-dimensional gender assignment, and 2-dimensional secondary classification assignment with 1-dimensional age assignment with 1-dimensional gender assignment. The sum of contributions for these different embedding size combinations is 1.3, 1.5, and 1.4, respectively. An integer programming algorithm is used to select the embedding size combination with the highest contribution, which is 1-dimensional secondary classification assignment, 2-dimensional age assignment, and 1-dimensional gender assignment, thus obtaining the embedding size for the input feature data.

[0093] Optionally, after obtaining the embedding size of each input feature data, the input feature data can be transformed into an embedding vector with the corresponding embedding size, and the recommendation system can be retrained to improve the accuracy of the recommendation system's prediction results.

[0094] Specifically, after obtaining the target dimension of each input feature data, the input feature data can be transformed into a target embedding vector of the target dimension according to the target dimension, and the target embedding vector of each input feature data can be input into the recommendation system to obtain a first prediction result, and the first prediction result can be recommended to the user.

[0095] In this embodiment, the target embedding vector dimension of each input feature data can be determined by an integer programming algorithm based on the preset total dimension and multiple contribution values ​​of each input feature data. This allows for the determination of the minimum embedding vector dimension of each input feature data without affecting the model prediction results, thereby reducing the computational load in the model inference process and improving model inference efficiency.

[0096] For example, in a mobile personalized advertising scenario, a recommendation model can first analyze a large number of user and item features to obtain recommended personalized ads, which are then delivered. Since ad delivery is a real-time bidding process, the model's computational speed can lead to issues such as lag, resulting in a poor user experience and reduced revenue. In this case, the data processing method proposed in this application's embodiments selects the embedding size for user and item features, allowing the input user and item features to have the smallest possible embedding dimension, thereby reducing the model's computational load and improving its inference efficiency. Furthermore, applying the data processing method proposed in this application's embodiments to shallow and deep conversion scenarios for advertising improves efficiency by 5.6% and 7%, respectively.

[0097] The method flow provided in this application has been described above. The apparatus provided in this application will now be described based on the aforementioned method flow.

[0098] See Figure 7 The present application provides a schematic diagram of the structure of a data processing device, as shown below.

[0099] The transformation module 701 is used to transform the input feature data into an N-dimensional first embedding vector. The input feature data includes user behavior data and / or item feature data, where N is an integer greater than or equal to 2.

[0100] The inference module 702 is used to infer from the N-dimensional first embedding vector through a neural network model to obtain the N-dimensional second embedding vector. The information content of the i-th dimension in the N-dimensional second embedding vector is negatively correlated with the value of i, and i is less than or equal to N.

[0101] The determination module 703 is used to determine the target dimension of each input feature data according to the preset total dimension and the N-dimensional second embedding vector of each input feature data. The preset total dimension is the sum of the vector dimensions allocated to each input feature data. The target dimension is used to obtain the target embedding vector. The target embedding vector is the embedding vector of the input feature data converted into the target dimension. The target embedding vector is used by the recommendation system to predict the first prediction result to be recommended to the user.

[0102] In one possible implementation, before the aforementioned reasoning module 702 infers the N-dimensional first embedding vector through a neural network model to obtain the N-dimensional second embedding vector, the aforementioned reasoning module 702 is also used to randomly truncate the N-dimensional first embedding vector to obtain multiple sub-embedded vectors of different dimensions.

[0103] In one possible implementation, the aforementioned inference module 702 is further configured to transform multiple sub-embedded vectors of different dimensions into multiple M-dimensional third embedded vectors through a mapping matrix, wherein the multiple M-dimensional third embedded vectors are the first embedded vectors.

[0104] In one possible implementation, the aforementioned determining module 703 is specifically used to: calculate the contribution of the first K dimensions of the N-dimensional second embedding vector to the second prediction result, and obtain N contribution values, wherein the second prediction result is obtained by the recommendation system through reasoning on the N-dimensional second embedding vector; and determine the target dimension of each input feature data according to the preset total dimension and the N contribution values ​​of each input feature data.

[0105] In one possible implementation, the aforementioned determining module 703 is specifically used to: calculate the contribution of the first K-dimensional embedding vectors in the N-dimensional second embedding vector to the second prediction result using the feature attribution method, and obtain N contribution values.

[0106] In one possible implementation, the aforementioned determining module 703 is specifically used to: determine the target dimension of each input feature data using an integer programming algorithm based on the preset total dimension and the N contribution values ​​of each input feature data.

[0107] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform a data processing method, or instruct the computing device to perform a data processing method.

[0108] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a data processing method, or a data processing procedure.

[0109] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0110] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process of the systems, devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0111] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0112] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0114] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0115] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0116] Finally, it should be noted that the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: The input feature data is transformed into an N-dimensional first embedding vector, wherein the input feature data includes user behavior data and / or item feature data, and N is an integer greater than or equal to 2; The first N-dimensional embedding vector is inferred through a neural network model to obtain a second N-dimensional embedding vector. The information content of the i-th dimension in the second N-dimensional embedding vector is negatively correlated with the value of i, where i is less than or equal to N. Based on the preset total dimension and the N-dimensional second embedding vector of each input feature data, the target dimension of each input feature data is determined. The preset total dimension is the sum of the vector dimensions allocated to each input feature data. The target dimension is used to obtain the target embedding vector. The target embedding vector is the embedding vector of the input feature data transformed into the target dimension. The target embedding vector is used by the recommendation system to predict the first prediction result to be recommended to the user.

2. The method according to claim 1, characterized in that, Before inferring the N-dimensional first embedding vector through a neural network model to obtain the N-dimensional second embedding vector, the method further includes: The N-dimensional first embedding vector is randomly truncated to obtain multiple sub-embedding vectors of different dimensions.

3. The method according to claim 2, characterized in that, The method further includes: The multiple sub-embedded vectors of different dimensions are transformed into multiple M-dimensional third embedded vectors through a mapping matrix, and the multiple M-dimensional third embedded vectors are the first embedded vector.

4. The method according to any one of claims 1 to 3, characterized in that, The step of determining the target dimension of each input feature data based on the preset total dimension and the N-dimensional second embedding vector of each input feature data includes: The contribution of the first K dimensions of the N-dimensional second embedding vector to the second prediction result is calculated to obtain N contribution values. The second prediction result is obtained by the recommendation system through reasoning on the N-dimensional second embedding vector. The target dimension of each input feature data is determined based on the preset total dimension and the N contribution values ​​of each input feature data.

5. The method according to claim 4, characterized in that, The calculation of the contribution of the first K dimensions of the N-dimensional second embedding vector to the second prediction result yields N contribution values, including: The contribution of the first K-dimensional embedding vectors in the N-dimensional second embedding vector to the second prediction result is calculated using the feature attribution method, resulting in the N contribution values.

6. The method according to any one of claims 4 or 5, characterized in that, The step of determining the target dimension of each input feature data based on the preset total dimension and the N contribution values ​​of each input feature data includes: Based on the preset total dimension and the N contribution values ​​of each input feature data, an integer programming algorithm is used to determine the target dimension of each input feature data.

7. A data processing apparatus, characterized in that, include: The transformation module is used to transform the input feature data into an N-dimensional first embedding vector, wherein the input feature data includes user behavior data and / or item feature data, and N is an integer greater than or equal to 2; The inference module is used to infer the N-dimensional first embedding vector through a neural network model to obtain an N-dimensional second embedding vector, wherein the information content of the i-th dimension in the N-dimensional second embedding vector is negatively correlated with the value of i, and i is less than or equal to N; The determination module is used to determine the target dimension of each input feature data according to the preset total dimension and the N-dimensional second embedding vector of each input feature data. The preset total dimension is the sum of the vector dimensions allocated to each input feature data. The target dimension is used to obtain the target embedding vector. The target embedding vector is the embedding vector of the input feature data transformed into the target dimension. The target embedding vector is used by the recommendation system to predict the first prediction result to be recommended to the user.

8. The apparatus according to claim 7, characterized in that, Before inferring the N-dimensional second embedding vector from the N-dimensional first embedding vector using a neural network model... The inference module is also used to randomly truncate the N-dimensional first embedding vector to obtain multiple sub-embedded vectors of different dimensions.

9. The apparatus according to claim 8, characterized in that, The inference module is further configured to transform the multiple sub-embedded vectors of different dimensions into multiple M-dimensional third embedded vectors through a mapping matrix, wherein the multiple M-dimensional third embedded vectors are the first embedded vector.

10. The apparatus according to any one of claims 7 to 9, characterized in that, The determining module is specifically used for: The contribution of the first K dimensions of the N-dimensional second embedding vector to the second prediction result is calculated to obtain N contribution values. The second prediction result is obtained by the recommendation system through reasoning on the N-dimensional second embedding vector. The target dimension of each input feature data is determined based on the preset total dimension and the N contribution values ​​of each input feature data.

11. The apparatus according to claim 10, characterized in that, The determining module is specifically used for: The contribution of the first K-dimensional embedding vectors in the N-dimensional second embedding vector to the second prediction result is calculated using the feature attribution method, resulting in the N contribution values.

12. The apparatus according to any one of claims 10 or 11, characterized in that, The determining module is specifically used for: Based on the preset total dimension and the N contribution values ​​of each input feature data, an integer programming algorithm is used to determine the target dimension of each input feature data.

13. A data processing apparatus, characterized in that, include: A processor and a memory, wherein the processor is coupled to the memory; The memory is used to store programs; The processor is configured to execute a program in the memory, causing the method as described in any one of claims 1 to 6 to be performed.

14. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method as claimed in any one of claims 1 to 6.

15. A computer program product comprising instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1 to 6.

Images