Feature interaction through edge search

Abstract

An interactive feature generation system can receive multiple distinct features associated with an application and associate multiple nodes in a first-order feature map with these features. The system can iteratively generate higher-order interactive features from lower-order interactive features to form multiple feature maps of different orders. Then, the system can propagate the corresponding interactive features from these different-order feature maps to a neural network to determine the number of interactive features of one or more orders. This determined number of interactive features of one or more orders is used to train a predictive model for inference about the application.

Description

Background Technology

[0001] Feature interaction is a crucial step in feature engineering. Feature interaction occurs when the behavior of one feature is influenced by the presence of another feature, and this interaction is often not easily inferred from the expected behavior of the individual features involved. By combining features in a certain way, it is possible to generate multiple higher-order interaction features (i.e., cross features) to better represent data and improve learning performance in machine learning. For example, in ad recommendation applications, the third-order interaction feature "gender"... age "Revenue" can be used as a strong feature to determine the type of ads to recommend to users.

[0002] Typically, interaction feature generation methods rely heavily on expert experience and knowledge, which is not only time-consuming but also task-specific. While automated interaction feature generation methods have been developed (primarily divided into two categories: search-based and deep learning-based methods), these methods face challenges such as excessively large search spaces (e.g., due to the trial-and-error approach used in search-based methods) or a lack of interpretability (e.g., due to the implicit nature of feature interactions in deep learning-based methods). In other words, these existing automated interaction feature generation methods cannot generate useful and explicit interaction features with simple and efficient training methods. Summary of the Invention

[0003] This invention presents a simplified concept of an interactive feature generation system, which will be further described in the detailed embodiments below. This invention is not intended to identify essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0004] This disclosure describes exemplary implementations of an interaction feature generation system. In some implementations, the interaction feature generation system may receive multiple distinct features associated with an application and associate multiple nodes in a first-order feature map with the multiple distinct features. The interaction feature generation system may iteratively generate higher-order interaction features from lower-order interaction features to form multiple feature maps of different orders. In some implementations, the interaction feature generation system may propagate the corresponding interaction features of the multiple feature maps of different orders to a neural network to determine the number of interaction features of one or more orders, the determined number of interaction features of one or more orders being used to train a predictive model to infer the application. Attached Figure Description

[0005] The specific embodiments are explained with reference to the accompanying drawings. In the drawings, the leftmost number of the reference numerals indicates the drawing in which the reference numeral first appears. The same reference numerals are used in different drawings to indicate similar or identical items.

[0006] Figure 1 An exemplary environment for generating a system using exemplary interactive features is shown.

[0007] Figure 2 An exemplary interactive feature generation system is shown in more detail.

[0008] Figure 3 The processing stages of an exemplary method for generating interactive features are shown.

[0009] Figure 4 An exemplary instance of adjacency matrix generation is shown.

[0010] Figure 5 An exemplary method for generating interactive features is shown. Detailed Implementation

[0011] Overview

[0012] As mentioned above, interaction feature generation is an important task in feature engineering. However, existing interaction feature generation methods face challenges such as excessively large search spaces or difficulties in interpreting general feature interaction rules, which stem from the implicit nature of feature interactions. In other words, these existing methods cannot generate useful and explicit interaction features in a simple and effective training manner.

[0013] This disclosure describes an exemplary interactive feature generation system. The interactive feature generation system can find interactive features of various orders (i.e., combinations of different features), thereby improving the performance of predictive models built on the interactive feature generation system. In some implementations, the interactive feature generation system may employ a feature map that models each feature as a node and represents the interaction between two nodes as an edge.

[0014] In some implementations, the interaction feature generation system can generate K feature maps to represent interaction features of order 2 to (K+1), where the feature maps have a hierarchical relationship with each other. In some implementations, the interaction feature generation system can generate interaction features in a continuous or iterative manner. For example, the interaction feature generation system can generate higher-order interaction features from lower-order interaction features and their corresponding feature maps.

[0015] In some implementations, to find useful interaction features for a predictive model from a large number of potential interaction features, the interaction feature generation system can perform edge search, such as a Markov Decision Process (MDP), to generate candidate interaction features. Exemplarily, and not limitingly, given the interactions of multiple k-order interaction features in the current state, the interaction feature generation system can determine the optimal cross-actions to efficiently generate (k+1)-order interaction features (e.g., the performance of a predictive model trained on the selected interaction features). Furthermore, to effectively and efficiently optimize the edge search, in some implementations, the interaction feature generation system can perform the edge search within a neural network architecture and can learn the edge parameters of the feature map in a differentiable manner.

[0016] Furthermore, the interactive feature generation system can optimize the parameters used to control the edge search process based on the predictions obtained as feedback during the training process. In some implementations, to make the optimization differentiable, the interactive feature generation system can further relax the hard binary representation of feature map edges (which characterizes the probability of connections between corresponding nodes in the feature map), that is, allow edges to take any value in the range [0, 1].

[0017] After training, the interaction feature generation system can reconstruct useful interaction features from the K feature maps. In some implementations, the interaction feature generation system can use these interaction features to train lightweight or less complex models (such as logistic regression models), which can be used by real-time inference systems.

[0018] In some implementations, the functions to be performed by the interaction feature generation system described herein may be performed by multiple separate units or services. Furthermore, although in the examples described herein the interaction feature generation system may be implemented by a combination of software and hardware installed in a single device, in other examples the interaction feature generation system may be distributed across multiple devices and implemented by multiple devices, or provided as a service in one or more computing devices within a network and / or cloud computing architecture.

[0019] This application describes several different embodiments and implementations. The following sections describe exemplary frameworks suitable for implementing various implementations. Next, this application describes exemplary systems, devices, and processes for implementing an interactive feature generation system.

[0020] Exemplary Environment

[0021] Figure 1An exemplary environment 100 is shown that can be used to implement an interactive feature generation system. Environment 100 may include an interactive feature generation system 102. In this example, the interactive feature generation system 102 is described as the presence of various entities or devices. In some instances, the interactive feature generation system 102 may be included in one or more servers 104, such as one or more computing devices or nodes in a cloud architecture. In other instances, the interactive feature generation system 102 may be included in a client device 106. For example, some or all of the functionality of the interactive feature generation system 102 may be included in, or provided by, one or more servers 104 and / or client devices 106, which are connected and communicate via a network 108.

[0022] In some implementations, the client device 106 may be considered any of a variety of computing devices, including but not limited to desktop computers, laptops or portable computers, handheld devices, netbooks, Internet devices, tablets or tablet computers, mobile devices (e.g., mobile phones, personal digital assistants, smartphones, etc.), server computers, or combinations thereof.

[0023] Network 108 can be a wireless or wired network, or a combination of both. Network 108 can be a collection of interconnected single large networks (e.g., the Internet or an intranet). Single large networks described in the example include, but are not limited to, telephone networks, cable networks, local area networks (LANs), wide area networks (WANs), and metropolitan area networks (MANs). Furthermore, individual networks can be wireless or wired networks, or a combination of both. Wired networks can include electrical carrier connections (such as communication cables) and / or optical carrier connections (such as fiber optic connections), or connections of both. Wireless networks can include, for example, WiFi networks, other radio frequency networks (e.g., ZigBee, etc.

[0024] In some implementations, the interaction feature generation system 102 may receive a request from a user's client device (e.g., client device 106) to generate or select interaction features for a specific application (such as an advertising recommendation application, a product recommendation application, etc.). In some implementations, the interaction feature generation system 102 may also receive additional information from the client device 106. This additional information may include, but is not limited to, initial or raw feature information for generating interaction or combined features, and training data information for training and generating interaction or combined features. Upon receiving a request, the interaction feature generation system 102 may execute the interaction feature generation method described below to generate or select multiple interaction features for that specific application. In some implementations, the interaction feature generation system 102 may return the number of interaction features to the client device 106 for the user of the client device 106 to display and / or manipulate. In some implementations, the interaction feature generation system 102 may also provide these interaction features for training a lightweight or less complex model (such as a linear regression model) and return the trained model to the client device 106 so that the client device 106 can make inferences on the specific application in real time.

[0025] Exemplary interactive feature generation system

[0026] Figure 2 The interactive feature generation system 102 is shown in more detail. In some implementations, the interactive feature generation system 102 may include, but is not limited to, one or more processors 202, memory 204, and program data 206. In some implementations, the interactive feature generation system 102 may also include an input / output (I / O) interface 208 and / or a network interface 210. In some implementations, some functions of the interactive feature generation system 102 may be implemented using hardware, including, for example, an ASIC (i.e., application-specific integrated circuit), an FPGA (i.e., field-programmable gate array), and / or other hardware.

[0027] In some implementations, processor 202 may be configured to execute instructions stored in memory 204 and / or received from input / output interface 208 and / or network interface 210. In some implementations, processor 202 may be implemented by one or more hardware processors, including, for example, microprocessors, dedicated instruction set processors, physical processing units (PPUs), central processing units (CPUs), graphics processing units, digital signal processors, tensor processing units, etc. Additionally or alternatively, the functions described herein may be performed at least in part by one or more hardware logic components. Examples of such hardware logic components, but not limited to, include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc.

[0028] Memory 204 may include computer-readable media in the form of volatile memory, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory 204 is an example of computer-readable media.

[0029] Computer-readable media can be of volatile or non-volatile type, removable or non-removable, and can be used to store information using any method or technology. Information can include computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other internal storage technologies, read-only optical disc memory (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include any medium that stores transient values, such as modulated digital signals and carrier waves.

[0030] Although only hardware components are described in this example of the interactive feature generation system 102, in other instances, the interactive feature generation system 102 may also include other hardware components and / or other software components, such as program units that perform various operations by executing instructions stored in memory 204. For example, the interactive feature generation system 102 may also include one or more databases 212 configured to store training data, parameters of the prediction model, information associated with feature maps, (initial, intermediate, or final) information associated with interactive features, etc.

[0031] Exemplary interactive feature generation algorithm

[0032] Figure 3 A schematic diagram illustrating the processing stages of an exemplary method for generating interactive features is shown. In some implementations, the exemplary method 300 may include at least four stages: a transformation stage 302, an edge search stage 304, a propagation stage 306, and a training stage 308.

[0033] In some implementations, during the transformation phase 302, the interactive feature generation system 102 can construct feature maps. This represents each row of input data associated with a specific application or application model. In some implementations, the input data can be stored or presented in tabular form and can include multiple fields, each storing different features. In some implementations, feature maps... Each node n i Different features of the input data can be represented, and the two nodes of the feature map (e.g., n) i and n j The edges between two nodes can represent the interaction between them.

[0034] In some implementations, the interactive feature generation system 102 can use one-hot encoding to represent the features of the input data and map the features of the input data to distributed feature embedding vectors. In some implementations, these feature embedding vectors can be defined as feature maps. The nodes. For example, given input data F = [f1, f2, ..., f m ], where m is the number of features, feature map Nodes can be defined or marked as

[0035] N = [n1, n2, ..., nn] m (1)

[0036] Each element N = [n1, n2, ..., nn] m ] and h are the dimensions of the feature embedding vector.

[0037] In some implementations, feature maps The nodes (i.e., features) can interact with each other through edges, and the adjacency matrix It can be used to represent feature maps The adjacency matrix is ​​a connection of (k+1) order interaction features. Exemplary and not restrictive, the adjacency matrix can be a binary matrix if the number of nodes n... i to node n i If the edge exists, then its elements The value is 1 if the condition is met, and 0 otherwise. In some implementations, the interaction feature generation system 102 can construct an adjacency tensor. The k-th slice A of the adjacent tensor A k These can be called adjacency matrices to record the connections between interaction features of different orders. These K adjacency matrices, or adjacency tensors, can be considered as feature maps. The architecture can be determined via edge search in the edge search stage 304.

[0038] For example, the k-th order interactive feature f k It can be defined as the cross product of the selected k distinct features, as shown below:

[0039]

[0040] in This represents the cross-product operation (e.g., Cartesian product), and each selected feature...

[0041] In the edge search phase 304, the interaction feature generation system 102 may employ edge states. To represent feature maps The probability of interaction between nodes in the matrix, where K is the highest order of feature crossover or feature interaction. For example, for the k-th matrix... When k-order interaction features are generated, each element This represents the corresponding pair of nodes (i.e., n). i and n j The probability of interaction between the adjacency matrix and the adjacency tensor A. In some implementations, the adjacency matrix or adjacency tensor A can be considered as a Bernoulli random variable parameterized by the edge states H.

[0042] In some implementations, the interaction feature generation system 102 may determine the adjacency tensor A via edge search. Exemplarily, and not limitingly, the interaction feature generation system 102 may employ a Markov decision process (MDP) to model the process of determining the adjacency matrix A via edge search. For example, the interaction feature generation system 102 may generate a k-th order interaction feature f k The generation process is divided into k consecutive decision steps. In each decision step, the interaction feature generation system 102 can choose to receive some first-order features (i.e., original features) from the input data, which are then crossed with higher-order interaction features to generate higher-order interaction features.

[0043] Exemplary, and not restrictive, given an interaction feature f of order (k-1). k-1 As the current state, the interactive feature generation system 102 can make a strategic decision to select a certain first-order feature and combine the selected first-order feature with the (k-1)-order interactive feature f. k-1 Cross-interaction to generate k-th order interactive features f k In some implementations, the edge state H represents the probability of an interaction between two nodes. This edge state can be used to guide the cross-interaction decision-making of the interaction feature generation system 102. For example, a high probability of an interaction between two nodes (i.e., two features) implies a high probability that these two nodes (i.e., these two features) will be selected for the cross-interaction.

[0044] In some implementations, each matrix H k ∈H represents the interaction of first-order features (i.e., the original features), rather than the interaction of k-order features, and edge search can be set up in an MDP-like manner. In some implementations, the edge search process can be represented recursively as follows:

[0045]

[0046] A 0 =I (4)

[0047] in This is a binary function, where I is the identity matrix and D is the normalized matrix, defined as follows:

[0048]

[0049] D i ,:=∑ j A i，j (6)

[0050] Where α is an adjustable threshold.

[0051] Figure 4 An exemplary instance 400 of adjacency matrix generation is shown. In some implementations, such as Figure 4 As shown, matrix multiplication A k-1 H k This can be considered as information compression by connecting two hops to the adjacency matrix. The calculated result x can represent a sequence starting at n. i And ends at n j The probability of multi-hop connections, the calculation result corresponds to the interaction feature. Therefore, the obtained adjacency matrix A k (k-1)-order interaction features and their corresponding cross probabilities can be aggregated and used to represent the interactions of k-order interaction features. In some implementations, each (k+1)-order interaction feature, for example, It can be considered as starting from node n i Jump to n j The k-hop path, and A k It can be viewed as a binary sample extracted from the k-hop transition matrix (or the k-hop transition matrix), where Atmosphere from n i to n j The k-hop visibility (or accessibility). In some implementations, the k-hop transition matrix can be calculated by multiplying the (k-1)-hop transition matrix by the corresponding adjacency matrix. Since the topologies of different layers are different, H... k It can be designed as a layer-by-layer transition matrix.

[0052] In some implementations, during propagation phase 306, given the feature map... This feature map Given a node vector N and a corresponding adjacency matrix or adjacency tensor A, the propagation process of vector-based feature crossings based on graph neural networks (GNNs) can be defined. For example, in a k-order feature crossing, each node can aggregate information from its corresponding one-hop neighbors to form an aggregated node vector, which is the sum of the initial node vectors (i.e., feature embedding vectors) of its neighbors:

[0053]

[0054]

[0055] Among them W j n is the transformation matrix of the node vectors j .

[0056] After K aggregations, the interactive feature generation system 102 can obtain the following node vectors:

[0057]

[0058] In some implementations, node vectors can include k-order interaction or cross features. Because node vectors It interacts with the corresponding K-order neighbor nodes, so it is possible to model the K-order interaction or cross features.

[0059] In some implementations, during the training phase 308, the interactive feature generation system 102 can train a lightweight prediction model, such as a nonlinear projection, and apply the lightweight prediction model to the following node vectors:

[0060]

[0061] Among them W p It is a projection matrix that can linearly combine and connect features, and σ(x)=1 / (1+e -x Values ​​can be converted into probabilities.

[0062] In some implementations, the interaction feature generation system 102 can also optimize the generated interaction features. In some implementations, as described above, the edge state H can guide the interaction feature generation system 102 to make feature crossover decisions, which can be considered a strategy and optimized for high rewards. In some implementations, the interaction feature generation system 102 can construct or employ a reward function to guide it to take actions that produce high or maximum rewards. In some implementations, the reward function can include, but is not limited to, the negation of logarithmic loss as shown below:

[0063]

[0064]

[0065] Among them, y i and These refer to the basic facts and the estimated probabilities, respectively, and D is the total number of training samples.

[0066] In some implementations, at the k-th order, the value function Q k This may include both immediate and long-term returns:

[0067]

[0068] Where K is the highest order and γ∈[0,1] is the discount factor. The inspiration behind this value function is to request an agent (e.g., interaction feature generation system 102) to consider the usefulness of features, which includes generating low-order interaction or cross features (i.e., immediate rewards) and subsequently generating or higher-order (i.e., long-term rewards) high-order interaction features. Since high-order interaction features can depend on low-order interaction features under the propagation phase 306 described above, the objective function that interaction feature generation system 102 can use may include, but is not limited to:

[0069]

[0070] In some implementations, since the edge state matrix can be binary-coded by the interaction feature generation system 102 to obtain the adjacency matrix, for example, using equation (5) as described above, the edge state matrix can be obtained without minimizing the loss as defined by the backpropagation (BP) method as in equation (14). Let's optimize it directly.

[0071] In some implementations, to make optimization more efficient and effective, the interaction feature generation system 102 can optimize the edge states H using Neural Architecture Search (NAS). As an example, and not a limitation, the interaction feature generation system 102 can relax the hard binary representation of the adjacency matrix to the probabilities of node interactions. The k-order adjacency matrix, depending on the edge state matrix and the binary representation of the previous order adjacency matrix (i.e., the current state), can be formally given as:

[0072]

[0073] In some implementations, a gap may exist between training using soft probabilities and testing using hard binary adjacency matrices due to the use of differentiable optimization techniques. In some implementations, the interactive feature generation system 102 may employ a continuous distribution that approximates samples from the classification distribution and works in conjunction with backpropagation. For example, the interactive feature generation system 102 may apply a variant of the Gumbel normalization function to A. k Each element, to reduce performance penalty:

[0074]

[0075] Where σ(x) = 1 / 1 + e -x a k It is A k σ is an element of σ, and τ is a temperature parameter. When τ approaches 0, σ becomes binary (i.e., close to 0 or 1).

[0076] In some implementations, as described above, after enabling relaxation from hard binary representation, the edge search task can involve optimizing continuous parameters via a backpropagation algorithm. If the edge state parameters are represented as W... e And the model parameters are represented as W o Then the dataset D is divided into training set D. train and verification set D val Example training algorithms may include the following:

[0077]

[0078] In some implementations, at the end of the edge search phase 304, the interaction feature generation system 102 can apply a binaryization function with an adjustable threshold to the adjacency matrix to directly obtain a binary adjacency matrix, which can be obtained in the edge search phase 304, and reconstruct useful interaction features of various orders based on this binary adjacency matrix. In some implementations, the interaction feature generation system 102 can also use interaction features of various orders to train lightweight or less complex models (such as linear regression models), thereby enabling real-time inference. Furthermore, in some implementations, the interaction feature generation system 102 can specify layer-wise thresholds used to binaryize the learned A and inductively derive useful k-order (1 < k < K) interaction features.

[0079] Exemplary methods

[0080] Figure 5 A schematic diagram illustrating an exemplary method for generating interactive features is shown. Figure 5 The method is possible but not required Figure 1 It can be used in an environment where it is not required. Figure 2 The system shown is implemented to achieve this. Figure 2 The system shown has Figure 3 Processing stages and Figure 4 Examples. For ease of explanation, see [reference]. Figure 1-4 Method 400 is described. However, method 500 can be alternatively implemented in other environments and / or using other systems.

[0081] Method 500 is described under the general premise of computer-executable instructions. Generally, computer-executable instructions can include routines, programs, objects, components, data structures, procedures, modules, functions, etc., that perform specific functions or implement specific abstract data types. Furthermore, each method in the exemplary methods is described as a set of boxes in a logic flowchart representing a sequence of operations that can be implemented in hardware, software, firmware, or a combination thereof. The order in which the method is described is not restrictive, and any number of method boxes described can be combined in any order to implement or replace the method. Additionally, individual boxes may be omitted from the method without departing from the subject matter and scope described herein. In a software environment, a box represents a computer instruction when the operation is performed by one or more processors. In a hardware environment, some or all of the boxes may represent an application-specific integrated circuit (ASIC) or other physical components that perform the operation.

[0082] Referring back to Figure 500, in box 502, the interaction feature generation system 102 can receive a request from a client device that can generate or determine interaction features for a specific application.

[0083] In some implementations, the interaction feature generation system 102 may receive a request from a user's client device (e.g., client device 106) for generating or determining interaction features for a specific application (such as an advertising recommendation application, a product recommendation application, etc.). In some implementations, the interaction feature generation system 102 may also receive additional data from the client device 106. In some implementations, the additional data may be included in the request or may be sent by the client device as separate information from the request. In some implementations, the additional data may be data stored in a storage device accessible to the interaction feature generation system 102, and the interaction feature generation system 102 may retrieve the additional data from the storage device upon receiving additional information, including address information included in the request or separate information, from the client device. In some implementations, the additional data may include, but is not limited to, information about initial or original features for generating interaction or combined features, and information about training data for training and generating interaction or combined features.

[0084] In some implementations, the initial or original features that generate interactive or combined features can be stored or input in tabular form, such as tabular data.

[0085] In box 504, the interactive feature generation system 102 can create a first-order feature map and associate multiple nodes of the feature map with multiple different features associated with a specific application.

[0086] In some implementations, upon receiving a request from a client device, the interaction feature generation system 102 can further obtain data on multiple different features associated with a specific application, and said data can be selectively or strategically combined into interaction features of various orders. In some implementations, the interaction feature generation system 102 can transform or convert the data on multiple different features into feature vector representations (e.g., vector representations) to obtain feature embedding vectors.

[0087] In some implementations, the interactive feature generation system 102 can use one-hot encoding to convert multiple different features into feature representations and map these feature representations into feature embedding vectors.

[0088] By way of example, and not limitation, the interactive feature generation system 102 may use one-hot encoding to modify or transform data with multiple different features. One-hot encoding identifies categorical variables (including label values ​​rather than numerical values) as binary vectors and includes mapping label values ​​to integer values. Each integer value is represented as a binary vector whose elements are zero except for the index of the integer marked as 1. For example, if the "color" variable includes three categories, namely red, green, and blue, one-hot encoding can represent these three label values ​​(i.e., red, green, and blue) as three different binary vectors, namely [1, 0, 0], [0, 1, 0], and [0, 0, 1], respectively. In some implementations, after transforming data with multiple different features, the interactive feature generation system 102 may obtain multiple feature embedding vectors.

[0089] In some implementations, the interactive feature generation system 102 can also create feature maps (e.g., the feature maps described above). The system associates multiple nodes in the feature map with multiple feature embedding vectors. In some implementations, the interactive feature generation system 102 can model each different feature among multiple different features as a corresponding node among multiple nodes in the feature map, and model the interaction between two different features among multiple different features as an edge between corresponding nodes among multiple nodes in the feature map. For example, given data with multiple different features F = [f1, f2, ..., f...], ... m ], where m is the number of features, and the nodes in the feature graph can be defined or labeled as N = [n1, n2, ..., n]. m ], where each element And h is the dimension of the feature embedding vector described above.

[0090] In some implementations, feature maps The nodes (i.e., features) can interact with each other through edges, and the adjacency matrix It can be used to represent feature maps The connections in the matrix. For example, and not limitingly, the adjacency matrix can be a binary matrix, if from node n... i to node n j If the edge exists, then element A i，j The value is 1 if the condition is met, and 0 otherwise. In some implementations, the interaction feature generation system 102 can construct K adjacency matrices to record the connections of interaction features of different orders.

[0091] In box 506, the interaction feature generation system 102 can iteratively generate higher-order interaction features from lower-order interaction features to form multiple feature maps of different orders.

[0092] In some implementations, such as the feature maps described above It can include multiple feature maps of different orders. For example, a first-order feature map (or a first-order feature map) can include the raw features input from the client device as described above, and a k-order feature map (or a k-th order feature map) can include interactive or combined features of k-th and lower orders. In some implementations, the k-th order interactive features can include the cross product of k distinct features, where k is an integer greater than or equal to 1.

[0093] In some implementations, the interaction feature generation system 102 can generate higher-order interaction features by interleaving low-order interaction features with features in a first-order feature map using edge search. In some implementations, edge search may include, but is not limited to, edge search via a Markov decision process as described above. In some implementations, the interaction feature generation system 102 can determine whether to connect two low-order interaction features to form a higher-order interaction feature, based at least in part on a reward function as described above. For example, the reward function may include an immediate reward component related to the usefulness of generating the low-order interaction feature and a long-term reward component related to the usefulness of generating the higher-order interaction feature.

[0094] In some implementations, the interactive feature generation system 102 can use edge states H = [H 1 H 2 H K ] represents the feature map The probability of interaction between nodes is as described above, where K is the highest order of feature crossover or feature interaction. The interaction feature generation system 102 can then determine the adjacency matrix A via edge search. Exemplarily, and not restrictively, the interaction feature generation system 102 can employ a Markov decision process (MDP) to model the process of determining the adjacency matrix A via edge search. For example, the interaction feature generation system 102 can generate k-th order interaction features f... kThe generation process is divided into k consecutive decision steps. In each decision step, the interaction feature generation system 102 can select some of the first-order features (i.e., the original features) received from the input data to cross with the higher-order interaction features to generate higher-order interaction features.

[0095] Exemplary, and not restrictive, given an interaction feature f of order (k-1). k-1 As the current state, the interaction feature generation system 102 can make a strategic decision to select a first-order feature and combine the selected first-order feature with the (k-1)th-order interaction feature f. k-1 Cross-interaction to generate k-th order interactive features f k In some implementations, the edge state H, representing the probability of an interaction between two nodes, can be used to guide the cross-interaction decision of the interaction feature generation system 102. For example, a high probability of an interaction between two nodes (i.e., two features) implies a high probability that these two nodes (i.e., these two features) will be selected for cross-interaction. Therefore, further details regarding the cross-interaction of low-order interaction features with features in a first-order feature map via edge search to generate higher-order interaction features can be found in the preceding description of the exemplary interaction feature generation algorithm, and will not be repeated here.

[0096] In box 508, the interaction feature generation system 102 can propagate the corresponding interaction features of multiple feature maps of different orders to a neural network to determine the number of interaction features of one or more orders. The determined number of interaction features of one or more orders is used to train a prediction model to make inferences for a specific application.

[0097] In some implementations, the interaction feature generation system 102 can propagate the corresponding interaction features of multiple feature maps of different orders to a neural network to determine the number of interaction features of one or more orders. In some implementations, the neural network may include, but is limited to, graph-based neural architectures, such as GNNs (Graph Neural Networks). For example, given a feature map with a node vector N and a corresponding adjacency matrix A obtained from the above operations... The interactive feature generation system 102 can aggregate information from the corresponding one-hop neighbor nodes for each node to form an aggregated node vector. This aggregated node vector is the sum of the initial node vectors (i.e., feature embedding vectors) of the neighbor nodes in the k-order feature cross. After K aggregations, the interactive feature generation system 102 obtains the node vector. It includes k-order interaction or cross features as described above. Because the node vectors Having already interacted with the corresponding K-order neighbor nodes, the interaction feature generation system 102 can model the K-order interaction or cross features accordingly. Therefore, for details regarding propagating the corresponding interaction features of multiple feature maps of different orders to the neural network to further determine the number of interaction features of one or more orders, please refer to the preceding description of the exemplary interaction feature generation algorithm; its details will not be repeated here.

[0098] In box 510, the interaction feature generation system 102 can collect data on a determined number of interaction features of one or more orders and use at least some of the collected data to train a prediction model.

[0099] In some implementations, after determining the number of interaction features to use for training the predictive model, the interaction feature generation system 102 may collect data on one or more orders of the determined number of interaction features and use at least some of the collected data to train the predictive model. For example, the interaction feature generation system 102 may use some of the collected data as training data and the remainder as test data. In some implementations, the interaction feature generation system 102 may collect data on the determined number of interaction features from a database associated with a specific application. For example, if the specific application is a product recommendation application for a shopping website, the interaction feature generation system 102 may collect data on the determined number of interaction features from a database associated with the shopping website, and this database may include data on customers visiting the website.

[0100] In some implementations, the predictive model may include a lightweight model that is simpler than a neural network. In some implementations, the predictive model may include, but is not limited to, linear regression models, decision trees, support vector machines, simplified neural networks, etc. In some implementations, the interaction feature generation system 102 can routinely train and test predictive models such as linear regression models, decision trees, and support vector machines using data of a determined number of interaction features of one or more orders collected.

[0101] For example, if the specific application is a product recommendation application, and the predictive model is used to recommend products to users, the multiple different features can include a variety of different features, which may include, but are not limited to, gender, age, income, geographic location, occupation, number of past purchases, and total amount of past purchases. Among the large number of potentially available different features, some features may be useful for inference or prediction, while others may be useless. Therefore, the interaction feature generation system 102 can select one or more orders of interaction features as determined in box 508 above, and use one or more orders of interaction features to train one or more predictive models. Continuing with the specific application of the product recommendation application described in the above example, the determined number of interaction features of one or more orders may include, for example, "gender". income "Age", "Gender" Location, Number of past purchases Total amount of past purchases income "Geographic location", etc. The interaction feature generation system 102 can use one or more of these interaction features of different orders to train the prediction model. In addition, or alternatively, the interaction feature generation system 102 can use a determined number of interaction features of one or more orders to train multiple prediction models, for example, training each prediction model based on one or more interaction features of the same or different orders.

[0102] In block 512, the interaction feature generation system 102 can receive new data of a determined number of interaction features of one or more orders and use a prediction model to infer specific applications based on the received data.

[0103] In some implementations, after obtaining the prediction model, the interaction feature generation system 102 can use the prediction model to infer a specific application based on newly received data of a determined number of interaction features of one or more orders.

[0104] Although the above method boxes are described as executing in a specific order, in some implementations, some or all of the method boxes may execute in a different order or in parallel.

[0105] in conclusion

[0106] Although the implementation of the structural features and / or methodological behavior has been described in specific language, it should be understood that the claims are not necessarily limited to the specific features or actions described. Rather, the specific features and actions are disclosed as exemplary forms of implementing the claimed subject matter. Furthermore, or alternatively, some or all of the operation may be implemented by one or more ASICs, FPGAs, or other hardware.

[0107] The following terms can be used to further understand this disclosure.

[0108] Clause 1: A method implemented by one or more computing devices, the method comprising: associating a plurality of nodes in a first-order feature map with a plurality of different features associated with an application; iteratively generating higher-order interaction features from lower-order interaction features to form a plurality of feature maps of different orders; and propagating the corresponding interaction features of the plurality of feature maps of different orders to a neural network to determine the number of interaction features of one or more orders, the determined number of interaction features of one or more orders being used to train a prediction model to infer the application.

[0109] Clause 2: The method according to Clause 1, wherein iteratively generating the higher-order interaction features from the lower-order interaction features to form the plurality of feature maps of different orders includes: determining, at least in part, based on a reward function, whether to connect the two lower-order interaction features to form the higher-order interaction features.

[0110] Clause 3: The method according to Clause 1, wherein the reward function includes an immediate reward component related to the usefulness of generating low-order interaction features and a long-term reward component related to the usefulness of generating high-order interaction features.

[0111] Clause 4: The method described in Clause 1 further includes receiving the plurality of different features in a tabular format.

[0112] Clause 5: The method according to Clause 1, wherein associating the plurality of nodes in the first-order feature map with the plurality of distinct features associated with the inference application includes: converting the plurality of distinct features into feature representations using one-hot encoding; and mapping the feature representations to feature embedding vectors, the feature embedding vectors being regarded as the plurality of nodes in the first-order feature map.

[0113] Clause 6: The method according to Clause 1, wherein associating the plurality of nodes in the first-order feature graph with the plurality of distinct features associated with the inference application comprises: modeling each of the plurality of distinct features as a corresponding node in the plurality of nodes in the feature graph, and modeling the interaction between two distinct features as an edge between corresponding nodes in the plurality of nodes in the feature graph.

[0114] Clause 7: The method according to Clause 1, wherein generating the higher-order interaction features iteratively from the lower-order interaction features to form the plurality of feature maps of different orders includes: generating the higher-order interaction features by intersecting the lower-order interaction features with features in the first-order feature map through edge search.

[0115] Clause 8: The method according to Clause 7, wherein the edge search includes edge search via a Markov decision process.

[0116] Clause 9: The method according to Clause 1, wherein the k-th order interaction features comprise the cross product of k distinct features, where k is an integer greater than or equal to 1.

[0117] Clause 10: The method according to Clause 1 further includes: collecting data on a determined number of interaction features for the one or more orders; and using the prediction model to make new inferences about the application based on the collected data.

[0118] Clause 11: One or more computer-readable media storing executable instructions that, when executed by one or more processors, cause the processors to perform actions, the actions comprising: associating a plurality of nodes in a first-order feature map with a plurality of distinct features associated with an application; iteratively generating higher-order interaction features from lower-order interaction features to form a plurality of feature maps of different orders; and propagating the corresponding interaction features of the plurality of feature maps of different orders to a neural network to determine a number of interaction features of one or more orders, the determined number of interaction features of one or more orders being used to train a predictive model to infer the application.

[0119] Clause 12: One or more computer-readable media according to Clause 11, wherein associating the plurality of nodes in the first-order feature map with the plurality of distinct features associated with the inference application includes: converting the plurality of distinct features into feature representations using one-hot encoding; and mapping the feature representations to feature embedding vectors, the feature embedding vectors being regarded as the plurality of nodes in the first-order feature map.

[0120] Clause 13: One or more computer-readable media as described in Clause 11, wherein associating the plurality of nodes in the first-order feature graph with the plurality of distinct features associated with the inference application includes: modeling each of the plurality of distinct features as a corresponding node among the plurality of nodes in the feature graph, and modeling the interaction between two distinct features as an edge between corresponding nodes among the plurality of nodes in the feature graph.

[0121] Clause 14: One or more computer-readable media according to Clause 11, wherein iteratively generating the higher-order interaction features from the lower-order interaction features to form the plurality of feature maps of different orders includes: generating the higher-order interaction features by intersecting the lower-order interaction features with features in the first-order feature map through edge search.

[0122] Clause 15: One or more computer-readable media as described in Clause 11, wherein the action further comprises: collecting data on a determined number of interaction features of the one or more orders; and using the prediction model to make new inferences about the application based on the collected data.

[0123] Clause 16: A system comprising: one or more processors; and a memory storing executable instructions that, when executed by the one or more processors, cause the one or more processors to perform actions, the actions including: associating a plurality of nodes in a first-order feature map with a plurality of different features associated with an application; iteratively generating higher-order interaction features from lower-order interaction features to form a plurality of feature maps of different orders; and propagating the corresponding interaction features of the plurality of feature maps of different orders to a neural network to determine a number of interaction features of one or more orders, the determined number of interaction features of one or more orders being used to train a predictive model to infer the application.

[0124] Clause 17: The system according to Clause 16, wherein associating the plurality of nodes in the first-order feature map with the plurality of distinct features associated with the inference application includes: converting the plurality of distinct features into feature representations using one-hot encoding; and mapping the feature representations to feature embedding vectors, the feature embedding vectors being regarded as the plurality of nodes in the first-order feature map.

[0125] Clause 18: The system according to Clause 16, wherein associating the plurality of nodes in the first-order feature graph with the plurality of distinct features associated with the inference application includes: modeling each of the plurality of distinct features as a corresponding node in the plurality of nodes in the feature graph, and modeling the interaction between two distinct features as an edge between corresponding nodes in the plurality of nodes in the feature graph.

[0126] Clause 19: The system according to Clause 16, wherein iteratively generating the higher-order interaction features from the lower-order interaction features to form the plurality of feature maps of different orders includes: generating the higher-order interaction features by intersecting the lower-order interaction features with features in the first-order feature map through edge search.

[0127] Clause 20: The system according to Clause 16, wherein the action further includes: collecting data on a determined number of interaction features of the one or more orders; and using the prediction model to make new inferences about the application based on the collected data.

Claims

1. A method implemented by one or more computing devices, the method comprising: Associate multiple nodes in the first-order feature map with multiple different features associated with the application; Higher-order interaction features are iteratively generated from lower-order interaction features to form multiple feature maps of different orders; wherein, at least in part, the determination of whether to connect two lower-order interaction features to form the higher-order interaction feature is based on a reward function, each of the multiple different features is modeled as a corresponding node among the multiple nodes in the feature map, the interaction between two different features is modeled as an edge between corresponding nodes among the multiple nodes in the feature map, and the connection relationship of features of different orders is recorded by an adjacency matrix; The corresponding interaction features of the multiple feature maps of different orders are propagated to the neural network to determine the number of interaction features of one or more orders; and A prediction model is trained using a determined number of interaction features of at least one or more orders, and the trained prediction model is used to infer the application. The process of iteratively generating higher-order interaction features from lower-order interaction features to form the multiple feature maps of different orders includes: generating higher-order interaction features by crossing the lower-order interaction features with features in the first-order feature map through edge search. The application is a product recommendation application, and the prediction model is used to recommend products to users. The different features include, but are not limited to, gender, age, income, geographical location, occupation, number of past purchases, and total amount of past purchases.

2. The method according to claim 1, wherein, The reward function includes an immediate reward component related to the usefulness of generating low-order interaction features and a long-term reward component related to the usefulness of generating high-order interaction features.

3. The method of claim 1, further comprising receiving the plurality of different features in a tabular format.

4. The method according to claim 1, wherein, Associating the plurality of nodes in the first-order feature map with the plurality of different features associated with the inference application includes: One-hot encoding is used to convert the multiple different features into feature representations; and The feature representation is mapped to a feature embedding vector, which is regarded as the plurality of nodes in the first-order feature map.

5. The method according to claim 1, wherein, The edge search includes edge search through Markov decision processes.

6. The method according to claim 1, wherein, The k-th order interaction features consist of the cross product of k distinct features, where k is an integer greater than or equal to 1.

7. The method according to claim 1, further comprising: Collect data on a defined number of interaction features for one or more of the aforementioned orders; as well as The predictive model is used to make new inferences about the application based on the collected data.

8. One or more computer-readable media storing executable instructions that, when executed by one or more processors, cause the one or more processors to perform an action, the action comprising: Associate multiple nodes in the first-order feature map with multiple different features associated with the application; Higher-order interaction features are iteratively generated from lower-order interaction features to form multiple feature maps of different orders; wherein, at least in part, the determination of whether to connect two lower-order interaction features to form the higher-order interaction feature is based on a reward function, each of the multiple different features is modeled as a corresponding node among the multiple nodes in the feature map, the interaction between two different features is modeled as an edge between corresponding nodes among the multiple nodes in the feature map, and the connection relationship of features of different orders is recorded by an adjacency matrix; The corresponding interaction features of the multiple feature maps of different orders are propagated to the neural network to determine the number of interaction features of one or more orders; and A prediction model is trained using a determined number of interaction features of at least one or more orders, and the trained prediction model is used to infer the application. The process of iteratively generating higher-order interaction features from lower-order interaction features to form the multiple feature maps of different orders includes: generating higher-order interaction features by crossing the lower-order interaction features with features in the first-order feature map through edge search. The application is a product recommendation application, and the prediction model is used to recommend products to users. The different features include, but are not limited to, gender, age, income, geographical location, occupation, number of past purchases, and total amount of past purchases.

9. One or more computer-readable media according to claim 8, wherein, Associating the plurality of nodes in the first-order feature map with the plurality of different features associated with the inference application includes: One-hot encoding is used to convert the multiple different features into feature representations; and The feature representation is mapped to a feature embedding vector, which is regarded as the plurality of nodes in the first-order feature map.

10. One or more computer-readable media according to claim 8, wherein, The action also includes: Collect data on a defined number of interaction features of one or more orders; and The predictive model is used to make new inferences about the application based on the collected data.

11. A system comprising: One or more processors; A memory storing executable instructions that, when executed by one or more processors, cause the one or more processors to perform actions, including: Associate multiple nodes in the first-order feature map with multiple different features associated with the application; Higher-order interaction features are iteratively generated from lower-order interaction features to form multiple feature maps of different orders; wherein, at least in part, the determination of whether to connect two lower-order interaction features to form the higher-order interaction feature is based on a reward function, each of the multiple different features is modeled as a corresponding node among the multiple nodes in the feature map, the interaction between two different features is modeled as an edge between corresponding nodes among the multiple nodes in the feature map, and the connection relationship of features of different orders is recorded by an adjacency matrix; The corresponding interaction features of the multiple feature maps of different orders are propagated to the neural network to determine the number of interaction features of one or more orders; and A prediction model is trained using a determined number of interaction features of at least one or more orders, and the trained prediction model is used to infer the application. The process of iteratively generating higher-order interaction features from lower-order interaction features to form the multiple feature maps of different orders includes: generating higher-order interaction features by crossing the lower-order interaction features with features in the first-order feature map through edge search. The application is a product recommendation application, and the prediction model is used to recommend products to users. The different features include, but are not limited to, gender, age, income, geographical location, occupation, number of past purchases, and total amount of past purchases.

12. The system according to claim 11, wherein, Associating the plurality of nodes in the first-order feature map with the plurality of different features associated with the inference application includes: One-hot encoding is used to convert the multiple different features into feature representations; and The feature representation is mapped to a feature embedding vector, which is regarded as the plurality of nodes in the first-order feature map.

13. The system according to claim 12, wherein, The action also includes: Collect data on a defined number of interaction features of one or more orders; and The predictive model is used to make new inferences about the application based on the collected data.

Images