Data processing method and related apparatus

Abstract

A data processing method, comprising: acquiring a first feature representation, wherein the first feature representation is obtained by performing feature extraction on input data, and the first feature representation comprises a first feature and a second feature; and on the basis of the first feature representation, obtaining a second feature representation by means of a first fully connected layer, wherein the second feature representation comprises a third feature and a fourth feature, the third feature is obtained by means of the first fully connected layer performing feature exchange between sub-features in the first feature, and the fourth feature is obtained by means of the first fully connected layer performing feature exchange between sub-features in the second feature. In the present application, the fully connected layer can perform exchange only on a part of an input feature, thereby reducing the complexity of a model, and further reducing the computational delay and computing power overheads.

Description

Data processing method and related apparatus

[0001] The present application claims priority to the Chinese patent application No. 202410732469.8, filed on June 6, 2024, and entitled "A data processing method and related apparatus", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, and in particular to a data processing method and related apparatus. BACKGROUND

[0003] Artificial intelligence (AI) is the use of digital computers or machine controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is the design principle and implementation method of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0004] Machine learning systems, including personalized recommendation systems, train the parameters of machine learning models based on input data and labels through optimization methods such as gradient descent. When the model parameters converge, the model can be used to predict unknown data. Taking click rate prediction in personalized recommendation system as an example, the input data includes user attributes and product attributes, etc. How to predict personalized recommendation list according to user's preference has an important influence on improving the recommendation accuracy of recommendation system.

[0005] Personalized recommendation systems play an important role in many online service platforms, from online advertising, retail to music and video recommendations. Taking click rate (CTR) prediction as an example, CTR prediction is a key task in modern online personalized platforms, which provides accurate and customized recommendation services to users. The goal of CTR is to estimate the probability of a user clicking a specific item under a given context. In order to maximize the click probability of users, the recommendation system sorts the items according to the estimated CTR and returns them to the user.

[0006] Personalized recommendation technology plays an important role in various service platforms, whether it is video advertising or music media, the accurate estimation of CTR model is needed to meet the customized needs of customers. CTR represents the click-through rate, that is, the click probability of a user on a specified item, the recommendation system trains the collected samples according to the CTR prediction model, and then returns the recommended items to the user according to the estimated CTR. Because MLP can well learn the representation of features, it has been widely used in feature interaction modeling. The characteristics of MLP in capturing high-order feature interaction information make it combined with other networks that capture low-order information to become the paradigm of feature interaction in CTR estimation model. In addition, MLP has a very wide application in user behavior modeling and the final prediction layer, which leads to the main time delay of the CTR estimation model being concentrated in the MLP module.

[0007] With the increase of features and the complexity of model structure, the time delay and computing power overhead of online inference are increasing, therefore, there is an urgent need for a method that can reduce the model inference time delay and computing power overhead. SUMMARY

[0008] In a first aspect, the present application provides a data processing method, when performing data processing in a full connection layer of a machine learning model (for example, the processing object is a first feature representation), the sub-features in the first features included in the first feature representation and the sub-features in the second features can be respectively subjected to feature interaction (that is, the sub-features included in the first features are subjected to feature interaction, and the sub-features included in the second features are subjected to feature interaction), instead of all the sub-features between the input data (for example, all the sub-features included in the first feature representation) being subjected to feature interaction, the present application can obtain a second feature representation through a first full connection layer according to the first feature representation, the second feature representation includes third features and fourth features, the third features are obtained by the first full connection layer performing feature interaction between the sub-features in the first features, and the fourth features are obtained by the first full connection layer performing feature interaction between the sub-features in the second features.

[0009] In the prior art, the full connection layer will perform full connection on all neurons in the network, that is, each neuron will interact with all features to obtain a processing result (new features), which will result in high computing time delay and computing power overhead when the number of features is large.

[0010] In the embodiments of the present application, the full connection layer can only interact with part of the input features, thereby reducing the complexity of the model, and further reducing the calculation delay and the computing power consumption. Specifically, each group of feature representations can include part of the features of the feature representations (for example, the first feature is part of the first feature representation, and the second feature is part of the first feature representation), and the neurons of the full connection layer can only interact with the sub-features in the group, that is, new sub-features are obtained according to the sub-features in the group. For example, one group obtained by grouping can include the first feature, and another group obtained by grouping can include the second feature. The third feature can be obtained by performing feature interaction between the sub-features in the first feature through the first full connection layer (for example, different sub-features in the third feature can be obtained through different neurons), and the fourth feature can be obtained by performing feature interaction between the sub-features in the second feature through the first full connection layer (for example, different sub-features in the fourth feature can be obtained through different neurons). This is equivalent to full connection of the neurons in the group, thereby reducing the complexity of the model.

[0011] In the embodiments of the present application, only part of the sub-features are interacted, which is equivalent to local connection of the full connection layer. However, the local connection ignores the interaction between the neurons in different groups, thereby causing information loss and making it difficult to maintain the original accuracy of the model. Therefore, the embodiments of the present application can further perform feature interaction between groups, thereby alleviating the loss of model information. Specifically, the different groups of the divided groups can be interacted, and in order to ensure low complexity, part of the sub-features (for example, one sub-feature is selected from each group) of different groups can be interacted.

[0012] Among them, the feature interaction between groups can be performed in the network layer where the first full connection layer is located, for example, a network layer (the second full connection layer in the embodiments of the present application) for performing feature interaction between groups can be connected after or before the first full connection layer. Of course, the second full connection layer can not be directly connected with the first full connection layer, or can not belong to the same network layer, and the embodiments of the present application do not limit this.

[0013] In one possible implementation, the first full connection layer includes: a first neuron and a second neuron; and the second feature representation is obtained through the first full connection layer according to the first feature representation, including: a first sub-feature and a second sub-feature in the third feature are obtained through the first neuron and the second neuron, respectively, according to the first feature; and a third sub-feature and a fourth sub-feature in the fourth feature are obtained through the first neuron and the second neuron, respectively, according to the second feature.

[0014] In a possible implementation, a network layer (the second fully connected layer in the embodiment of the present application) for inter-group feature interaction can be connected after the first fully connected layer. Specifically, the fourth feature representation can be obtained through the second fully connected layer according to the third feature representation, and the fourth feature representation includes the fifth feature, which is obtained by performing feature interaction on part of the sub-features in the third feature and part of the sub-features in the fourth feature through the second fully connected layer.

[0015] In a possible implementation, a network layer (the second fully connected layer in the embodiment of the present application) for inter-group feature interaction can be connected before the first fully connected layer. Specifically, the first feature representation can be obtained through the second fully connected layer according to the fourth feature representation, the fourth feature representation is obtained by performing feature extraction on the input data, and the fourth feature representation includes a plurality of sixth features and a plurality of seventh features. The first sub-feature in the first feature is obtained by performing feature interaction on the sub-features in the sixth features through the second fully connected layer, and the second sub-feature in the first feature is obtained by performing feature interaction on the sub-features in the seventh features through the second fully connected layer.

[0016] The embodiment of the present application introduces local connection between groups to maintain the accuracy of the model and keep the interaction of information between groups, and each layer of group interaction maintains the accuracy by using a small network with low complexity.

[0017] In a possible implementation, the method further includes: obtaining a fifth feature representation, the fifth feature representation being a feature representation of the input data obtained through a third fully connected layer; and fusing the fifth feature representation and the second feature representation. The embodiment of the present application can add a residual connection between model layers, directly transmits input information to output, protects the integrity of information, reduces loss and waste, and improves the accuracy of the model without increasing complexity.

[0018] In a possible implementation, the input data is an image, text or audio.

[0019] In a possible implementation, the input data is user attributes and item attributes, and the first fully connected layer belongs to a multi-layer perception (MLP) of a recommendation model.

[0020] In a second aspect, the present application provides a data processing apparatus, the apparatus comprising:

[0021] The obtaining module is configured to obtain a first feature representation, the first feature representation being obtained by performing feature extraction on input data, and the first feature representation including a first feature and a second feature.

[0022] The processing module is configured to obtain a second feature representation from the first feature representation through a first fully connected layer, the second feature representation comprising a third feature and a fourth feature, the third feature being obtained by performing feature interaction between sub-features in the first feature through the first fully connected layer, and the fourth feature being obtained by performing feature interaction between sub-features in the second feature through the first fully connected layer.

[0023] In a possible implementation, the first fully connected layer comprises a first neuron and a second neuron.

[0024] The processing module is specifically configured to:

[0025] obtain a first sub-feature and a second sub-feature in the third feature from the first feature through the first neuron and the second neuron, respectively;

[0026] obtain a third sub-feature and a fourth sub-feature in the fourth feature from the second feature through the first neuron and the second neuron, respectively.

[0027] In a possible implementation, the processing module is further configured to:

[0028] obtain a fourth feature representation from the third feature representation through a second fully connected layer, the fourth feature representation comprising a fifth feature, the fifth feature being obtained by performing feature interaction between part of the sub-features in the third feature and part of the sub-features in the fourth feature through the second fully connected layer.

[0029] In a possible implementation, the obtaining module is specifically configured to:

[0030] obtain the fourth feature representation from the first feature representation through the second fully connected layer, the fourth feature representation being obtained by performing feature extraction on the input data, the fourth feature representation comprising a plurality of sixth features and a plurality of seventh features, a first sub-feature in the first feature being obtained by performing feature interaction between sub-features in the sixth features through the second fully connected layer, and a second sub-feature in the first feature being obtained by performing feature interaction between sub-features in the seventh features through the second fully connected layer.

[0031] In a possible implementation, the processing module is further configured to:

[0032] obtain a fifth feature representation of the input data through a third fully connected layer;

[0033] fuse the fifth feature representation and the second feature representation.

[0034] In a possible implementation, the input data is an image, text, or audio.

[0035] In a possible implementation, the input data are user attributes and item attributes, and the first fully connected layer belongs to a multi-layer perception (MLP) of the recommendation model.

[0036] In a third aspect, an embodiment of the present application provides a data processing apparatus, which can include a memory, a processor and a bus system, wherein the memory is configured to store a program, and the processor is configured to execute the program in the memory to perform any of the methods in the first aspect.

[0037] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed on a computer, the computer is caused to perform the method in the first aspect.

[0038] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes code, and when the code is executed, the code is configured to implement the method in the first aspect.

[0039] In a sixth aspect, the present application provides a chip system, which includes a processor configured to support a device for performing or a device for training to implement the functions involved in the above aspects, for example, to send or process the data involved in the above method; or, information. In a possible design, the chip system further includes a memory, and the memory is configured to store the necessary program instructions and data for the device for performing or the device for training. The chip system can be composed of a chip, or can include the chip and other discrete devices. BRIEF DESCRIPTION OF DRAWINGS

[0040] FIG. 1 is a structural schematic diagram of an artificial intelligence main body framework;

[0041] FIG. 2 is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0042] FIG. 3 is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0043] FIG. 4 is a schematic diagram of a recommendation scenario provided by an embodiment of the present application;

[0044] FIG. 5 is a flow schematic diagram of a data processing method provided by an embodiment of the present application;

[0045] FIG. 6A is a flow schematic diagram of a data processing method provided by an embodiment of the present application;

[0046] FIG. 6B is a flow schematic diagram of a data processing method provided by an embodiment of the present application;

[0047] FIG. 6C is a flow schematic diagram of a data processing method provided by an embodiment of the present application;

[0048] Fig. 7 is a flow diagram of a data processing method according to an embodiment of the present application;

[0049] Fig. 8 is a structural diagram of a data processing apparatus according to an embodiment of the present application;

[0050] Fig. 9 is a schematic diagram of an execution device according to an embodiment of the present application;

[0051] Fig. 10 is a schematic diagram of a training device according to an embodiment of the present application;

[0052] Fig. 11 is a schematic diagram of a chip according to an embodiment of the present application. DETAILED DESCRIPTION

[0053] The embodiments of the present application will be described below in conjunction with the accompanying drawings. The terms used in the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0054] The embodiments of the present application will be described below in conjunction with the accompanying drawings. The terms used in the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0055] The terms "first", "second", etc. in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged, and this is only a distinguishing way used in describing the embodiments of the present application for the same attribute objects in the description. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the processes, methods, systems, products or devices containing a series of units do not have to be limited to those units, but can include other units not clearly listed or inherent to these processes, methods, products or devices.

[0056] First, the overall workflow of the artificial intelligence system is described, please refer to Fig. 1, which shows a structural diagram of an artificial intelligence main framework, and the following describes the above-mentioned artificial intelligence theme framework from two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data has undergone a condensation process of "data-information-knowledge-wisdom". "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the bottom infrastructure of human intelligence, information (provision and processing technology implementation) to the industrial ecological process of the system.

[0057] (1) Infrastructure

[0058] Infrastructure provides computing power support for artificial intelligence systems, enables communication with the outside world, and is supported by a basic platform. Communication with the outside world is achieved through sensors; computing power is provided by intelligent chips (CPU, NPU, GPU, ASIC, FPGA, etc. hardware acceleration chips); the basic platform includes distributed computing framework and network-related platform support and support, which can include cloud storage and computing, interconnection network, etc. For example, sensors and external communication obtain data, which are provided to intelligent chips in the distributed computing system provided by the basic platform for calculation.

[0059] (2) Data

[0060] The data of the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence. Data involves graphics, images, speech, text, and also involves Internet of Things data of traditional devices, including business data of existing systems and sensing data such as force, displacement, liquid level, temperature, and humidity.

[0061] (3) Data processing

[0062] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision making, etc.

[0063] Among them, machine learning and deep learning can model, extract, preprocess, and train symbolic and formalized intelligent information.

[0064] Reasoning refers to the process of simulating human intelligent reasoning methods in a computer or intelligent system, using formalized information to perform machine thinking and solve problems according to reasoning control strategies, and the typical function is search and matching.

[0065] Decision making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0066] (4) General capabilities

[0067] After data is processed as mentioned above, some general capabilities can be formed based on the results of data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0068] (5) Intelligent products and industry applications

[0069] Intelligent product and industry application refers to the product and application of artificial intelligence system in various fields, which is the packaging of the overall solution of artificial intelligence, and realizes the application of intelligent information decision productization. Its application fields mainly include intelligent terminal, intelligent transportation, intelligent medical treatment, automatic driving, smart city, etc.

[0070] Embodiments of the present application can be applied to the field of information recommendation, which includes but is not limited to scenarios involving e-commerce product recommendation, search engine result recommendation, application market recommendation, music recommendation, video recommendation, etc. The recommended items in various application scenarios can also be referred to as "objects" for ease of subsequent description, that is, in different recommendation scenarios, the recommended objects can be APPs, or videos, or music, or a certain product (such as the presentation interface of an online shopping platform, which will display different products for presentation according to different users, which can also be essentially presented through the recommendation results of a recommendation model). These recommendation scenarios usually involve user behavior log collection, log data preprocessing (e.g., quantization, sampling, etc.), sample set training to obtain a recommendation model, analysis and processing of objects (such as APPs, music, etc.) involved in the scene corresponding to the training sample item according to the recommendation model, for example, the samples selected in the recommendation model training link come from the operation behavior of users of a mobile application market on recommended APPs, then the recommendation model trained therefrom is applicable to the above-mentioned mobile APP application market, or can be used for recommending terminal APPs in other types of terminal APP application markets. The recommendation model will finally calculate the recommendation probability or score of each recommended object, and the recommendation system will select the recommendation result according to certain selection rules, for example, according to the recommendation probability or score for sorting, and present it to the user through the corresponding application or terminal device, and the user operates the objects in the recommendation result to generate user behavior logs, etc.

[0071] In the recommendation process, when a user interacts with the recommendation system, a recommendation request is triggered, the recommendation system inputs the request and its related feature information into the deployed recommendation model, and then predicts the click rate of the user on all candidate objects. Subsequently, the candidate objects are arranged in descending order according to the predicted click rate, and the candidate objects are displayed in different positions in order as the recommendation result for the user. The user browses the displayed items and generates user behaviors such as browsing, clicking and downloading, etc. These user behaviors are stored in the log as training data, and the parameters of the recommendation model are updated by the offline training module from time to time to improve the recommendation effect of the model.

[0072] For example, the user opens the mobile application market to trigger the recommendation module of the application market. The recommendation module of the application market predicts the download possibility of the user for each candidate application according to the historical download record, user click record, application characteristics, time, and environmental characteristic information such as location. According to the prediction result, the application market is displayed in descending order of possibility, so as to improve the application download probability. Specifically, the application that is more likely to be downloaded is arranged in the front position, and the application that is less likely to be downloaded is arranged in the rear position. The user behavior is also stored in the log and the parameters of the prediction model are trained and updated through the offline training module.

[0073] For another example, in the application related to lifelong companion, the historical data of the user in the video, music, news, and other fields can be used to build a cognitive brain and establish a user lifelong learning system framework through various models and algorithms, which imitates the human brain mechanism. The lifelong companion can record the past events of the user, understand the current intention of the user, predict the future action or behavior of the user, and finally realize intelligent service according to the system data and application data. In the current first stage, the behavior data of the user (containing end-side short message, photo, email event, and other information) is obtained according to the music APP, video APP, and browser APP. On the one hand, a user portrait system is built, and on the other hand, a learning and memory module based on user information filtering, correlation analysis, cross-domain recommendation, and causal reasoning is realized to build a user personal knowledge graph.

[0074] Next, the application architecture of the embodiments of the present application is introduced.

[0075] Referring to FIG. 2, an embodiment of the present application provides a recommendation system architecture 200. A data collection device 260 is configured to collect samples. One training sample can be composed of multiple feature information (or described as attribute information, such as user attributes and item attributes). The feature information can be various, and can specifically include user feature information and object feature information and label feature. The user feature information is configured to represent the characteristics of the user, such as gender, age, occupation, and hobbies. The object feature information is configured to represent the characteristics of the object pushed to the user. Different recommendation systems correspond to different objects, and the types of features required to be extracted by different objects are also different. For example, the object features extracted in the training sample of the APP market can be the name (identifier) of the APP, the type, the size, etc. The object features extracted in the training sample of the e-commerce APP can be the name of the product, the category to which the product belongs, the price range, etc. The label feature is configured to represent whether the sample is a positive example or a negative example. Generally, the label feature of the sample can be obtained through the operation information of the user on the recommended object. The sample in which the user operates on the recommended object is a positive example, and the sample in which the user does not operate on the recommended object or only browses is a negative example. For example, when the user clicks or downloads or purchases the recommended object, the label feature is 1, indicating that the sample is a positive example. If the user does not operate on the recommended object, the label feature is 0, indicating that the sample is a negative example. The samples can be saved in a database 230 after being collected. Part or all of the feature information in the samples in the database 230 can also be directly obtained from a client device 240, such as user feature information, operation information of the user on the object (used to determine the type identifier), object feature information (such as object identifier), etc. A training device 220 trains to obtain a model parameter matrix for generating a recommendation model 201 (such as the feature extraction network and the neural network in the embodiment of the present application) based on the samples in the database 230. How the training device 220 trains to obtain the model parameter matrix for generating the recommendation model 201 will be described in more detail below. The recommendation model 201 can be used to evaluate a large number of objects to obtain the scores of each object to be recommended. Further, a specified or preset number of objects can be recommended from the evaluation results of a large number of objects. A calculation module 211 obtains a recommendation result based on the evaluation result of the recommendation model 201, and recommends the result to the client device through an I / O interface 212.

[0076] In the embodiment of the present application, the training device 220 can select positive and negative samples from the sample set in the database 230 and add them to the training set. Then, the recommendation model is used to train the samples in the training set to obtain the trained recommendation model. The implementation details of the calculation module 211 can refer to the detailed description of the method embodiment shown in FIG. 5.

[0077] After the training device 220 obtains the model parameter matrix based on the sample training and constructs the recommendation model 201, the recommendation model 201 is sent to the execution device 210, or the model parameter matrix is directly sent to the execution device 210, and the recommendation model is constructed in the execution device 210, which is used for making recommendations of the corresponding system. For example, the recommendation model obtained based on the sample training related to the video can be used for recommending videos to users in the video website or APP, and the recommendation model obtained based on the sample training related to the APP can be used for recommending APPs to users in the application market.

[0078] The execution device 210 is configured with an I / O interface 212 to interact with external devices. The execution device 210 can obtain user feature information such as user identification, user identity, gender, occupation, hobby, etc. from the client device 240 through the I / O interface 212. This part of information can also be obtained from the system database. The recommendation model 201 recommends target recommended objects to users based on user feature information and recommended object feature information. The execution device 210 can be set in a cloud server or a user client.

[0079] The execution device 210 can call data, code, etc. in the data storage system 250, and also can store the output data in the data storage system 250. The data storage system 250 can be set in the execution device 210, or can be independently set, or set in other network entities, and the number can be one or more.

[0080] The calculation module 211 uses the recommendation model 201 to process the user feature information and the recommended object feature information. For example, the calculation module 211 uses the recommendation model 201 to analyze and process the user feature information and the recommended object feature information, so as to obtain the score of the recommended object. The recommended objects are sorted according to the score, and the objects with high ranking are recommended to the client device 240.

[0081] Finally, the I / O interface 212 returns the recommendation result to the client device 240 and presents it to the user.

[0082] More deeply, the training device 220 can generate corresponding recommendation models 201 based on different sample feature information for different targets, so as to provide better results for users.

[0083] It is worth noting that the figure 2 is only a schematic diagram of a system architecture provided by the embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in the figure 2, the data storage system 250 is an external memory relative to the execution device 210, and in other cases, the data storage system 250 can be placed in the execution device 210.

[0084] In the embodiments of the present application, the training device 220, the execution device 210 and the client device 240 can be three different physical devices respectively, or the training device 220 and the execution device 210 can be on the same physical device or a cluster, or the execution device 210 and the client device 240 can be on the same physical device or a cluster.

[0085] Referring to FIG. 3, it is a system architecture 300 according to an embodiment of the present application. In this architecture, the execution device 210 is implemented by one or more servers, which can be optionally combined with other computing devices such as data storage, routers, load balancers, etc. The execution device 210 can be arranged on one physical site or distributed on multiple physical sites. The execution device 210 can use the data in the data storage system 250 or call the program code in the data storage system 250 to implement the function of object recommendation. Specifically, the information of the object to be recommended is input into the recommendation model, the recommendation model generates an estimated score for each object to be recommended, and then the objects are sorted in descending order of the estimated scores, and the user is recommended the objects according to the sorting results. For example, the top 10 objects in the sorting results are recommended to the user.

[0086] The data storage system 250 is used to receive and store the parameters of the recommendation model sent by the training device, and is used to store the data of the recommendation results obtained by the recommendation model, and of course can also include the program code (or instructions) required for the normal operation of the storage system 250. The data storage system 250 can be a distributed storage cluster composed of one device or multiple devices deployed outside the execution device 210. When the execution device 210 needs to use the data on the storage system 250, the storage system 250 can send the data required by the execution device 210 to the execution device 210, and correspondingly, the execution device 210 receives and stores (or caches) the data. Of course, the data storage system 250 can also be deployed in the execution device 210. When deployed in the execution device 210, the distributed storage system can include one or more memories. When there are multiple memories, different memories are used to store different types of data. For example, the model parameters of the recommendation model generated by the training device and the data of the recommendation results obtained by the recommendation model can be stored in two different memories respectively.

[0087] The users can operate their respective user devices (such as local device 301 and local device 302) to interact with the execution device 210. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smartphone, a tablet computer, a smart camera, a smart car or other types of cellular phones, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0088] The local device of each user can interact with the execution device 210 through a communication network of any communication mechanism / communication standard, which can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0089] In another implementation, the execution device 210 can be implemented by the local device, for example, the local device 301 can implement the recommendation function of the execution device 210 to obtain user feature information and feed back the recommendation result to the user, or provide services for the user of the local device 302.

[0090] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the related terms and concepts related to neural networks involved in the embodiments of the present application will be introduced first.

[0091] 1. Click-through rate (CTR)

[0092] Click-through rate, also known as click rate, is the ratio of the number of clicks to the number of exposures of recommended information (e.g., recommended items) on a website or application. Click-through rate is an important indicator for measuring the recommendation system.

[0093] 2. Personalized recommendation system

[0094] The personalized recommendation system refers to a system that uses machine learning algorithms to analyze historical data (e.g., operation information in the embodiments of the present application) of users and make predictions for new requests to give personalized recommendations.

[0095] 3. Offline training

[0096] Offline training refers to a module in the personalized recommendation system that iteratively updates the parameters of the recommendation model according to the historical data (e.g., operation information in the embodiments of the present application) of the user according to the learning algorithm until the set requirements are met.

[0097] 4. Online inference

[0098] Online inference refers to predicting the user's preference for recommended items in the current context based on the user, item, and context features using the offline trained model, and predicting the probability of the user selecting the recommended item.

[0099] For example, FIG. 4 is a schematic diagram of a recommendation system according to an embodiment of the present application. As shown in FIG. 4, when a user enters the system, a request for recommendation is triggered, the recommendation system inputs the request and its related information (e.g., operation information in the embodiment of the present application) into a recommendation model, and then predicts the selection rate of the user for the items in the system. Further, the items are arranged in descending order according to the predicted selection rate or a function based on the selection rate, i.e., the recommendation system can display the items in different positions in order as the recommendation result for the user. The user browses the items in different positions and user behaviors occur, such as browsing, selecting, and downloading, etc. Meanwhile, the actual behaviors of the user are stored in the log as training data, the parameters of the recommendation model are constantly updated by the offline training module, and the prediction effect of the model is improved.

[0100] For example, the user opens the application market in the smart terminal (e.g., mobile phone) to trigger the recommendation system in the application market. The recommendation system in the application market predicts the probability of the user downloading each candidate APP according to the historical behavior log of the user, e.g., the historical download record of the user, the selection record of the user, the environmental characteristic information of the application market itself, such as time, location, etc. According to the calculation result, the recommendation system in the application market can display the candidate APPs in descending order according to the predicted probability value, so as to improve the download probability of the candidate APPs.

[0101] For example, the APPs with high predicted user selection rate can be displayed in the front recommendation position, and the APPs with low predicted user selection rate can be displayed in the rear recommendation position.

[0102] The recommendation model described above can be a neural network model, and the related terms and concepts of the neural network that can be involved in the embodiments of the present application are introduced as follows.

[0103] (1) Neural network

[0104] The neural network can be composed of neural units, and the neural unit can be an operation unit with xs (i.e., input data) and intercept 1 as input. The output of the operation unit can be:

[0105] where s = 1, 2, … n, n is a natural number greater than 1, Ws is the weight of xs, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce non-linear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. The neural network is a network formed by connecting a plurality of the above single neural units, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of a plurality of neural units.

[0106] (2) Deep neural network

[0107] Deep neural network (DNN), also known as multi-layer neural network, can be understood as a neural network with many layers of hidden layers, where "many" has no special measurement standard. From the position of DNN according to different layers, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, and the last layer is the output layer, and the number of layers in between is the hidden layer. The layers are fully connected, that is, any neuron in the i-th layer is connected to any neuron in the i+1-th layer. Although DNN looks very complex, in terms of the work of each layer, it is actually not complex, which can be simply expressed as the following linear relationship expression: where, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer only performs the following simple operation on the input vector to obtain the output vector Since DNN has many layers, the number of coefficients W and offset vectors is also large. These parameters in DNN are defined as follows: taking the coefficient W as an example: assuming in a three-layer DNN, the linear coefficient of the fourth neuron in the second layer to the second neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, and the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary: the coefficient of the k-th neuron in the L-1-th layer to the j-th neuron in the L-th layer is defined as It is noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better capture the complexity of real-world situations. In theory, the more parameters a model has, the higher its complexity and the greater its "capacity" to perform more complex learning tasks. Training a deep neural network is essentially a process of learning the weight matrices, and the ultimate goal is to obtain the weight matrices of all layers of the trained deep neural network (weight matrices formed by vectors W of many layers).

[0108] (3) Loss function

[0109] In the process of training a deep neural network, because the output of the deep neural network is expected to be as close as possible to the value that is truly intended to be predicted, the weight vectors of each layer of the neural network can be updated based on the difference between the predicted value of the current network and the target value that is truly intended to be predicted. (Of course, before the first update, there is usually an initialization process, i.e., pre-configuring parameters for each layer in the deep neural network.) For example, if the predicted value of the network is too high, the weight vectors are adjusted to make it predict lower values. This process is repeated until the deep neural network can predict the target value that is truly intended to be predicted or a value very close to it. Therefore, it is necessary to define "how to compare the difference between the predicted value and the target value" in advance, which is the loss function or the objective function, which is an important equation for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and the training of the deep neural network becomes a process of trying to minimize this loss.

[0110] (4) Back propagation algorithm

[0111] The back propagation (BP) algorithm can be used to correct the size of the parameters in the initial model during training, so that the error loss of the model becomes smaller and smaller. Specifically, the error loss is generated by forward propagating the input signal until the output, and the parameters in the initial model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain optimal model parameters such as weight matrices.

[0112] (5) Machine learning system

[0113] Based on the input data and labels, the parameters of the machine learning model are trained by optimization methods such as gradient descent, and the trained model is finally used to complete the prediction of unknown data.

[0114] (6) Personalized recommendation system

[0115] According to historical data of users, a system is provided for analyzing and modeling by using a machine learning algorithm, and making predictions on new user requests to give personalized recommendation results.

[0116] A machine learning system, including a personalized recommendation system, trains parameters of a machine learning model based on input data and labels by using an optimization method such as gradient descent, and when the model parameters converge, the model can be used to complete prediction of unknown data. Taking click rate prediction in a personalized recommendation system as an example, the input data includes user attributes and product attributes. How to predict a personalized recommendation list according to user preferences has an important influence on improving the recommendation accuracy of the recommendation system.

[0117] A personalized recommendation system plays an important role in many online service platforms, from online advertising, retail to music and video recommendation. Taking click through rate (CTR) prediction as an example, CTR prediction is a key task in modern online personalized platforms, which provides accurate and customized recommendation services for users. The goal of CTR is to estimate the probability of a user clicking a specific item in a given context. In order to maximize the click probability of users, the recommendation system sorts the items according to the estimated CTR and returns them to the user.

[0118] Personalized recommendation technology plays an important role in various service platforms, whether it is video advertising or music media, and accurate estimation of the CTR model is needed to meet the customization needs of customers. CTR represents the click-through rate, i.e., the probability of a user clicking on a specified item, and the recommendation system trains the collected samples according to the CTR prediction model to sort and return the recommended items to the user according to the estimated CTR. Due to its ability to learn feature representations, MLP has been widely used in feature interaction modeling. The ability of MLP to capture high-order feature interaction information makes it a paradigm for feature interaction in CTR estimation models. In addition, MLP has a very wide range of applications in user behavior modeling and the final prediction layer, which results in the main delay of the CTR estimation model being concentrated in the MLP module.

[0119] With the increase of features and the complexity of model structure, the online inference time delay and computing power overhead are increasing, therefore, there is an urgent need for a method that can reduce the model inference time delay and computing power overhead.

[0120] To solve the above problems, the present application provides a data processing method, which can be used for the inference process of the model or the feedforward process of the model training.

[0121] With reference to FIG. 5, FIG. 5 is an embodiment of a data processing method provided by the present application. As shown in FIG. 5, the data processing method provided by the present application includes the following steps.

[0122] 501. Obtain a first feature representation, the first feature representation being obtained by performing feature extraction on input data, the first feature representation including a first feature and a second feature.

[0123] In a possible implementation, the input data is an image, text or audio.

[0124] In a possible implementation, the input data is user attributes and item attributes, and the first fully connected layer belongs to a multilayer perceptron (MLP) of a recommendation model.

[0125] The item can be a real item or a virtual item, for example, an application (APP), audio / video, a webpage, news information or the like. The attribute information of the item can be at least one of an item name, a developer, an installation package size, a category and a degree of praise. For example, when the item is an application, the category of the item can be a chatting application, a running game, an office application or the like, and the degree of praise can be a score or a comment for the item. The present application does not limit the specific type of the attribute information of the item.

[0126] The attribute information of the user can be at least one of a gender, an age, an occupation, an income, a hobby and an education level. The gender can be male or female, the age can be a number between 0 and 100, the occupation can be a teacher, a programmer, a chef or the like, the hobby can be basketball, tennis, running or the like, and the education level can be primary school, junior high school, high school, college or the like. The present application does not limit the specific type of the attribute information of the user.

[0127] In a possible implementation, the first feature representation is obtained by performing feature extraction on the input data, and the first feature representation includes the first feature and the second feature. The input data can be input of a machine learning model (for example, a recommendation model), the first feature representation can be intermediate output of feature extraction of a feature extraction network (for example, a multilayer perceptron (MLP)) in the machine learning model, and the first feature representation can be input of a network layer (for example, a first fully connected layer) of the machine learning model.

[0128] Optionally, the input data can be embedded by an embedding layer of the machine learning model to obtain an embedded representation, and the embedded representation can include a plurality of features. Then, the features can be interacted by a feature interaction network (for example, an MLP).

[0129] 502、According to the first feature representation, a second feature representation is obtained through a first full connection layer, the second feature representation comprising a third feature and a fourth feature, the third feature being obtained by performing feature interaction between sub-features in the first feature through the first full connection layer, and the fourth feature being obtained by performing feature interaction between sub-features in the second feature through the first full connection layer.

[0130] The first full connection layer can belong to an MLP, and the second feature representation can be obtained through the first full connection layer according to the first feature representation. The first feature representation can comprise a plurality of sub-features, wherein the first feature comprises part of the sub-features of the first feature representation, and the second feature comprises part of the sub-features of the first feature representation. The first full connection layer can process the first feature representation.

[0131] In the prior art, a full connection layer connects all neurons in a network, that is, each neuron interacts with all features to obtain a processing result (new feature), which can result in high computational latency and high computational resource consumption when the number of features is large.

[0132] In the embodiments of the present application, the full connection layer can only interact with part of the input features, thereby reducing the complexity of the model and further reducing the computational latency and computational resource consumption. Specifically, each group of feature representations can comprise part of the features of the feature representation (for example, the first feature is part of the first feature representation, and the second feature is part of the first feature representation), and the neurons of the full connection layer can only interact with the sub-features within the group, that is, new sub-features are obtained according to the sub-features within the group. For example, one group obtained by grouping can comprise a first feature, and another group obtained by grouping can comprise a second feature. The third feature can be obtained by performing feature interaction between sub-features in the first feature through the first full connection layer (for example, different sub-features in the third feature can be obtained through different neurons), and the fourth feature can be obtained by performing feature interaction between sub-features in the second feature through the first full connection layer (for example, different sub-features in the fourth feature can be obtained through different neurons). This is equivalent to only performing full connection on the neurons within the group, thereby reducing the complexity of the model.

[0133] In one possible implementation, the first full connection layer comprises a first neuron and a second neuron. The first sub-feature and the second sub-feature in the third feature can be obtained through the first neuron and the second neuron, respectively, according to the first feature. The third sub-feature and the fourth sub-feature in the fourth feature can be obtained through the first neuron and the second neuron, respectively, according to the second feature.

[0134] The first fully connected layer can be one network layer in the machine learning model, and embodiments of the present application do not limit the position of the first fully connected layer in the machine learning model. For example, the first fully connected layer can be one network layer of the MLP, or one network layer in the DNN.

[0135] For example, referring to FIG. 6A, FIG. 6A shows an example of dividing the features into five groups, where the fully connected layer can interact within the group.

[0136] In embodiments of the present application, only part of the sub-features are interacted, which is equivalent to local connection of the fully connected layer. However, the local connection ignores the interaction of neurons between different groups, thereby causing information loss and making it difficult to maintain the original accuracy of the model. Therefore, embodiments of the present application can further interact the features between groups to alleviate the loss of model information. Specifically, the features between different groups of divided groups can be interacted, and in order to ensure low complexity, part of the sub-features (for example, one sub-feature is selected from each group) of different groups can be interacted.

[0137] The feature interaction between groups can be performed in the network layer where the first fully connected layer is located, for example, a network layer (second fully connected layer in embodiments of the present application) for interacting features between groups can be connected after or before the first fully connected layer. Of course, the second fully connected layer can not be directly connected with the first fully connected layer, or not belong to the same network layer, and embodiments of the present application do not limit this.

[0138] In a possible implementation, a network layer (second fully connected layer in embodiments of the present application) for interacting features between groups can be connected after the first fully connected layer. Specifically, the fourth feature representation can be obtained through the second fully connected layer according to the third feature representation, and the fourth feature representation includes a fifth feature obtained by interacting part of the sub-features in the third feature and part of the sub-features in the fourth feature through the second fully connected layer.

[0139] In a possible implementation, a network layer (second fully connected layer in embodiments of the present application) for interacting features between groups can be connected before the first fully connected layer. Specifically, the first feature representation can be obtained through the second fully connected layer according to the fourth feature representation, and the fourth feature representation is obtained by feature extraction on the input data. The fourth feature representation includes a plurality of sixth features and a plurality of seventh features. The first sub-feature in the first feature is obtained by interacting the sub-features in the sixth feature through the second fully connected layer, and the second sub-feature in the first feature is obtained by interacting the sub-features in the seventh feature through the second fully connected layer.

[0140] The embodiments of the present application introduce local connection between groups to keep the accuracy of the model and keep the interaction of information between groups, and each layer group interacts to keep the accuracy by using a small network with low complexity.

[0141] For example, referring to FIG. 6B, FIG. 6B shows an example of dividing features into five groups, and the full connection layer can interact between groups.

[0142] In addition, residual connection can be added between model layers to protect the integrity of information, reduce loss and waste, and improve the accuracy of the model without increasing complexity.

[0143] In a possible implementation, a fifth feature representation can be obtained, the fifth feature representation being a feature representation of the input data obtained through a third full connection layer (i.e., a network layer other than the first full connection layer); and the fifth feature representation and the second feature representation are fused.

[0144] For example, referring to FIG. 6C, FIG. 6C shows an example of dividing features into five groups, and residual connection between network layers can be performed.

[0145] The machine learning model in the embodiments of the present application can be a recommendation model. Taking the recommendation model as an example, a system architecture of an embodiment of the present application is shown in FIG. 7. Different CTR estimation models use different feature interaction designs to realize the mining of shallow interaction information and deep interaction information, which usually combines MLP network and other network designs. The wide use of MLP in the CTR estimation model makes the core operator Matmul of MLP the main bottleneck of model latency, so the embodiment designs a pluggable lightweight SMLP network to replace the original MLP network to reduce the complexity of the model and optimize the system latency bottleneck.

[0146] In a large number of CTR estimation scenarios, the recommendation model provides a recommendation list to the user by modeling the interaction history of the user and the goods, and each scenario may correspond to a different CTR estimation model. For example, news, video and music, different CTR estimation models can be adapted in different scenarios, and these models can use the lightweight network optimization of the present application. Taking two mainstream CTR estimation models DCN and Fibinet shown in FIG. 7 as examples, the DNN part of DCN and the MLP estimation part of Fibinet can be replaced by the simple recommendation network structure based on sparse connection provided by the embodiments of the present application. The embodiments of the present application can be used as a replaceable plug-in in each CTR estimation model in all recommendation scenarios to generally improve the efficiency of the model and optimize the system bottleneck.

[0147] Next, from the perspective of the device, introduce a data processing device provided by the embodiment of the application, referring to figure 8, figure 8 is a data processing device provided by the embodiment of the application, as shown in figure 8, a data processing device 800 provided by the embodiment of the application includes:

[0148] The acquisition module 801 is configured to acquire a first feature representation, the first feature representation being obtained by performing feature extraction on the input data, and the first feature representation including a first feature and a second feature.

[0149] Specifically, the acquisition module 801 can be implemented by referring to the description of step 501 in the above embodiment, and details are not described here.

[0150] The processing module 802 is configured to obtain a second feature representation by a first full connection layer according to the first feature representation, the second feature representation including a third feature and a fourth feature, the third feature being obtained by performing feature interaction between sub-features in the first feature by the first full connection layer, and the fourth feature being obtained by performing feature interaction between sub-features in the second feature by the first full connection layer.

[0151] Specifically, the processing module 802 can be implemented by referring to the description of step 502 in the above embodiment, and details are not described here.

[0152] In a possible implementation, the first full connection layer includes a first neuron and a second neuron.

[0153] The processing module is specifically configured to:

[0154] According to the first feature, the first sub-feature and the second sub-feature in the third feature are obtained by the first neuron and the second neuron, respectively.

[0155] According to the second feature, the third sub-feature and the fourth sub-feature in the fourth feature are obtained by the first neuron and the second neuron, respectively.

[0156] In a possible implementation, the processing module is further configured to:

[0157] According to the third feature representation, the fourth feature representation is obtained by a second full connection layer, the fourth feature representation including a fifth feature, the fifth feature being obtained by performing feature interaction between part of the sub-features in the third feature and part of the sub-features in the fourth feature by the second full connection layer.

[0158] In a possible implementation, the acquisition module is specifically configured to:

[0159] According to a fourth feature representation, the first feature representation is obtained through a second fully connected layer, the fourth feature representation is obtained by feature extraction on the input data, the fourth feature representation includes a plurality of sixth features and a plurality of seventh features, a first sub-feature in the first feature is obtained by feature interaction between sub-features in the sixth features through the second fully connected layer, and a second sub-feature in the first feature is obtained by feature interaction between sub-features in the seventh features through the second fully connected layer.

[0160] In a possible implementation, the processing module is further configured to:

[0161] obtain a fifth feature representation, the fifth feature representation being a feature representation of the input data obtained through a third fully connected layer;

[0162] fuse the fifth feature representation and the second feature representation.

[0163] In a possible implementation, the input data is an image, text, or audio.

[0164] In a possible implementation, the input data is a user attribute and an item attribute, and the first fully connected layer is a multi-layer perception (MLP) of a recommendation model.

[0165] Next, a terminal device provided by an embodiment of the present application is introduced. Referring to FIG. 9, FIG. 9 is a structural schematic diagram of a terminal device provided by an embodiment of the present application. The terminal device 900 can specifically be a mobile phone, a tablet, a notebook computer, a smart wearable device, etc., which is not limited herein. The terminal device 900 implements the function of the data processing method in the embodiment corresponding to FIG. 5. Specifically, the terminal device 900 includes a receiver 901, a transmitter 902, a processor 903, and a memory 904 (wherein the number of processors 903 in the terminal device 900 can be one or more), wherein the processor 903 can include an application processor 9031 and a communication processor 9032. In some embodiments of the present application, the receiver 901, the transmitter 902, the processor 903, and the memory 904 can be connected through a bus or other means.

[0166] The memory 904 can include a read-only memory and a random access memory, and provide instructions and data for the processor 903. A part of the memory 904 can also include a non-volatile random access memory (NVRAM). The memory 904 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or expanded sets thereof, wherein the operation instructions can include various operation instructions for implementing various operations.

[0167] The processor 903 controls the operation of the terminal device. In a specific application, various components of the terminal device are coupled together by a bus system, which can include a data bus, a power bus, a control bus, and a state signal bus, etc. However, for the sake of clarity, various buses are referred to as a bus system in the figure.

[0168] The method disclosed in the embodiments of the present application can be applied to or implemented by the processor 903. The processor 903 can be an integrated circuit chip having a signal processing capability. In the implementation process, the steps of the above method can be completed by hardware integrated logic circuits in the processor 903 or by instructions in the form of software. The processor 903 described above can be a general processor, a digital signal processor (DSP), a microprocessor or a microcontroller, a vision processing unit (VPU), a tensor processing unit (TPU), and other processors suitable for AI operation, and can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 903 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, and other mature storage media in the art. The storage medium is located in the memory 904, and the processor 903 reads the information in the memory 904 and combines the hardware to complete the steps 501 to 503 in the above embodiments.

[0169] The receiver 901 can be used to receive input digital or character information, and to generate signal input related to the relevant settings and function control of the terminal device. The transmitter 902 can be used to output digital or character information through the first interface; the transmitter 902 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 902 can also include a display device such as a display screen.

[0170] The embodiment of the present application further provides a server. Please refer to FIG. 10, which is a structural schematic diagram of the server according to the embodiment of the present application. Specifically, the server 1000 is implemented by one or more servers. The server 1000 can be quite different due to different configurations or performances. The server 1000 can include one or more central processing units (CPUs) 1010 (for example, one or more processors) and a memory 1032, and one or more storage media 1030 (for example, one or more mass storage devices) for storing application programs 1042 or data 1044. The memory 1032 and the storage media 1030 can be temporary storage or persistent storage. The programs stored in the storage media 1030 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the server. Furthermore, the central processing unit 1010 can be configured to communicate with the storage media 1030 and execute the series of instruction operations in the storage media 1030 on the server 1000.

[0171] The server 1000 can further include one or more power supplies 1026, one or more wired or wireless network interfaces 1050, one or more input and output interfaces 1058, or one or more operating systems 1041, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0172] Specifically, the server can perform the steps 501 to 503 in the above embodiment.

[0173] The embodiment of the present application further provides a computer program product, which, when running on a computer, causes the computer to perform the steps performed by the above execution device, or causes the computer to perform the steps performed by the above training device.

[0174] The embodiment of the present application further provides a computer readable storage medium, which stores a program for signal processing. When the program runs on a computer, the computer performs the steps performed by the above execution device, or the computer performs the steps performed by the above training device.

[0175] The execution device, the training device or the terminal device provided by the embodiments of the present application can specifically be a chip, which comprises a processing unit, for example, a processor, and a communication unit, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute computer execution instructions stored in a storage unit, so that the chip in the execution device executes the data processing method described in the above embodiments, or so that the chip in the training device executes the data processing method described in the above embodiments. Alternatively, the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit can also be a storage unit outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0176] Specifically, please refer to FIG. 11, which is a structural schematic diagram of a chip provided by the embodiments of the present application. The chip can be represented as a neural network processor NPU 1100, which is mounted on a host CPU (Host CPU) as a coprocessor and is assigned tasks by the Host CPU. The core part of the NPU is an operation circuit 1103, which extracts matrix data in a memory and performs multiplication operation through a controller 1104.

[0177] The NPU 1100 can realize the data processing method provided in the embodiments described in FIG. 5 through mutual cooperation between various devices inside the NPU 1100.

[0178] More specifically, in some implementations, the operation circuit 1103 in the NPU 1100 internally includes a plurality of processing units (PEs). In some implementations, the operation circuit 1103 is a two-dimensional systolic array. The operation circuit 1103 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1103 is a general-purpose matrix processor.

[0179] For example, it is assumed that there are an input matrix A, a weight matrix B and an output matrix C. The operation circuit takes corresponding data of the matrix B from the weight memory 1102 and buffers it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1101 and performs matrix operation with the matrix B, and the partial result or final result of the obtained matrix is saved in an accumulator 1108.

[0180] The unified memory 1106 is used to store input data and output data. The weight data is transferred to the weight memory 1102 through the Direct Memory Access Controller (DMAC) 1105. The input data is also transferred to the unified memory 1106 through the DMAC.

[0181] The BIU is the Bus Interface Unit 1110, which is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 1109.

[0182] The BIU 1110 is used for the instruction fetch buffer 1109 to fetch instructions from the external memory, and is also used for the memory access controller 1105 to fetch the original data of the input matrix A or the weight matrix B from the external memory.

[0183] The DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 1106, or to transfer the weight data to the weight memory 1102, or to transfer the input data to the input memory 1101.

[0184] The vector calculation unit 1107 includes a plurality of operation processing units, which further process the output of the operation circuit 1103 as needed, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / full connection layer network calculation in neural networks, such as batch normalization, pixel-level summation, upsampling of feature planes, etc.

[0185] In some implementations, the vector calculation unit 1107 can store the processed output vector to the unified memory 1106. For example, the vector calculation unit 1107 can apply a linear function; or, a nonlinear function to the output of the operation circuit 1103, such as linear interpolation on the feature planes extracted by the convolutional layer, and further, a vector of accumulated values to generate activation values. In some implementations, the vector calculation unit 1107 generates normalized values, pixel-level summation values, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1103, such as for use in subsequent layers in a neural network.

[0186] The controller 1104 is connected to the instruction fetch buffer 1109, which is used to store instructions used by the controller 1104;

[0187] The unified memory 1106, the input memory 1101, the weight memory 1102, and the instruction memory 1109 are on-chip memories. The external memory is private to the NPU hardware architecture.

[0188] The processor mentioned in any of the above can be a general central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling execution of the above programs.

[0189] It should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. In addition, the connection relationship between the modules in the apparatus embodiment provided in the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0190] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and necessary general hardware, and of course can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including a plurality of instructions for making a computer device (which can be a personal computer, a training device, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0191] In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product in whole or in part.

[0192] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be stored by the computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

Claims

1. A data processing method, characterized by, The method comprises: obtaining a first feature representation, the first feature representation being obtained by feature extraction on input data, the first feature representation comprising a first feature and a second feature; obtaining a second feature representation from the first feature representation through a first fully connected layer, the second feature representation comprising a third feature and a fourth feature, the third feature being obtained by feature interaction between sub-features in the first feature through the first fully connected layer, and the fourth feature being obtained by feature interaction between sub-features in the second feature through the first fully connected layer.

2. The method of claim 1, wherein, The first fully connected layer comprises a first neuron and a second neuron. The first fully connected layer comprises a first neuron and a second neuron. The method comprises: obtaining a fourth feature representation from the third feature representation through a second fully connected layer, the fourth feature representation comprising a fifth feature, the fifth feature being obtained by feature interaction between part of the sub-features in the third feature and part of the sub-features in the fourth feature through the second fully connected layer.

3. The method according to claim 1 or 2, characterized in that, The method comprises: obtaining a fourth feature representation from the third feature representation through a second fully connected layer, the fourth feature representation comprising a fifth feature, the fifth feature being obtained by feature interaction between part of the sub-features in the third feature and part of the sub-features in the fourth feature through the second fully connected layer.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: obtaining a fifth feature representation of the input data through a third fully connected layer; 5. The method according to any one of claims 1 to 4, characterized in that, fusing the fifth feature representation and the second feature representation. The input data is an image, text or audio. The input data is user attributes and item attributes, and the first fully connected layer belongs to a multi-layer perceptron (MLP) of a recommendation model.

6. The method according to any one of claims 1 to 5, characterized in that, The device comprises:

7. The method according to any one of claims 1 to 6, characterized in that, an obtaining module configured to obtain a first feature representation, the first feature representation being obtained by feature extraction on input data, the first feature representation comprising a first feature and a second feature; 8. A data processing apparatus, characterized by, a processing module configured to obtain a second feature representation from the first feature representation through a first fully connected layer, the second feature representation comprising a third feature and a fourth feature, the third feature being obtained by feature interaction between sub-features in the first feature through the first fully connected layer, and the fourth feature being obtained by feature interaction between sub-features in the second feature through the first fully connected layer. ​ ​ 9. The apparatus of claim 8, wherein, The first full connection layer includes a first neuron and a second neuron. The processing module is specifically configured to: According to the first feature, a first sub-feature and a second sub-feature in the third feature are obtained through the first neuron and the second neuron, respectively. According to the second feature, a third sub-feature and a fourth sub-feature in the fourth feature are obtained through the first neuron and the second neuron, respectively.

10. The apparatus of claim 8 or 9, wherein, The processing module is further configured to: According to the third feature representation, a fourth feature representation is obtained through a second full connection layer, the fourth feature representation including a fifth feature obtained by performing feature interaction on part of the sub-features in the third feature and part of the sub-features in the fourth feature through the second full connection layer.

11. The apparatus of any one of claims 8 to 10, wherein, The obtaining module is specifically configured to: According to the fourth feature representation, the first feature representation is obtained through a second full connection layer, the fourth feature representation being obtained by performing feature extraction on the input data, the fourth feature representation including a plurality of sixth features and a plurality of seventh features, the first sub-feature in the first feature being obtained by performing feature interaction between the sub-features in the sixth features through the second full connection layer, and the second sub-feature in the first feature being obtained by performing feature interaction between the sub-features in the seventh features through the second full connection layer.

12. The apparatus of any one of claims 8 to 11, wherein, The processing module is further configured to: Obtain a fifth feature representation, the fifth feature representation being a feature representation of the input data obtained through a third full connection layer; Fuse the fifth feature representation and the second feature representation.

13. The apparatus of any one of claims 8 to 12, wherein, The input data is an image, a text, or an audio.

14. The apparatus of any one of claims 8 to 13, wherein, The input data is a user attribute and an item attribute, and the first full connection layer belongs to a multi-layer perception (MLP) of a recommendation model.

15. A computing device, comprising: The computing device includes a memory and a processor; the memory stores code, and the processor is configured to obtain the code and execute the method of any one of claims 1 to 7.

16. A computer storage medium, comprising, The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to implement the method of any one of claims 1 to 7.

17. A computer program product comprising code, characterized in that, The code is configured to implement the method of any one of claims 1 to 7 when the code is executed.

18. A chip comprising a processor, characterized in that The processor is configured to support a data processing apparatus to implement the method of any one of claims 1 to 7. The processor is configured to support a data processing apparatus to implement the method of any one of claims 1 to 7.

Images