Method and apparatus for data processing

Abstract

The application provides a data processing method and device. It relates to the field of artificial intelligence, and specifically relates to the field of computer vision. The method comprises: adding an architecture parameter to each feature combination item in a first model to obtain a second model, wherein the first model is an FM-based model, and the architecture parameter is used to represent the importance of the corresponding feature combination item; optimizing and learning the architecture parameter in the second model to obtain an optimized architecture parameter; and obtaining a third model after feature combination item reduction based on the first model or the second model according to the optimized architecture parameter. Through the optimization and learning of the architecture parameter, the selection of the feature combination item can be realized, and the multiple-pass training for multiple candidate subsets in the prior art is not required, so that the calculation amount of feature combination selection can be effectively reduced, thereby saving computing power and improving the efficiency of feature combination selection.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, specifically to a data processing method and apparatus. Background Technology

[0002] With the rapid development of internet technology, the problem of information overload has emerged. To address this issue, recommender systems (RS) have been developed. Click-through rate (CTR) prediction is a crucial step in recommender systems; the predicted CTR is used to determine whether to recommend a product. CTR prediction considers not only individual features but also combinations of features. To represent these combinations, factorization machine (FM) models have been proposed. FM models include feature combination terms representing all possible combinations of individual features. Current technologies typically employ FM-based models for CTR prediction.

[0003] The number of feature interaction terms in a Model-Factor (FM) increases exponentially with the order of the feature combination. Therefore, as the order of the feature combination increases, the number of feature interaction terms becomes enormous, leading to extremely high computational costs for training the FM model. To address this problem, feature interaction selection (FIS) was proposed. Since manual feature interaction selection is time-consuming and labor-intensive, automatic feature interaction selection (AutoFIS) was developed.

[0004] Existing automatic feature combination screening schemes achieve feature combination screening by searching for the optimal subset in a search space consisting of all possible feature combination subsets. The search process is energy-intensive and consumes a lot of computing power. Summary of the Invention

[0005] This application provides a data processing method and apparatus that can reduce the computational load and computing power consumption of feature combination screening.

[0006] In a first aspect, a data processing method is provided, comprising: adding architecture parameters to each feature combination item in a first model to obtain a second model, wherein the first model is a factorization machine (FM) based model, and the architecture parameters are used to characterize the importance of the corresponding feature combination item; performing optimization learning on the architecture parameters in the second model to obtain optimized architecture parameters; and obtaining a third model after feature combination item reduction based on the optimized architecture parameters, either the first model or the second model.

[0007] FM-based models refer to models built on the principles of FM, including any of the following: FM model, DeepFM model, IPNN model, AFM model, and NFM model.

[0008] The third model can be obtained by reducing the feature combination terms based on the first model.

[0009] Alternatively, the third model can be obtained by reducing the feature combination terms based on the second model.

[0010] There are several ways to determine which feature combinations to remove, or in other words, which feature combinations to retain (i.e., filter).

[0011] Alternatively, as an implementation method, feature combination items corresponding to architecture parameters whose values ​​are less than a threshold can be deleted from the optimized architecture parameters.

[0012] The threshold is used to characterize the criterion for whether to retain a feature combination item. For example, if the value of the optimized architecture parameter of a feature combination item is less than the threshold, it indicates that the feature combination item is deleted. If the value of the optimized architecture parameter of a feature combination item reaches the threshold, it indicates that the feature combination item is retained (i.e., filtered).

[0013] This threshold can be determined based on actual application requirements. For example, the value of this threshold can be obtained through model training. This application does not limit the method of obtaining this threshold.

[0014] Alternatively, as another implementation, if some architectural parameters become zero after optimization, the feature combination terms corresponding to the architectural parameters whose values ​​are not zero after optimization can be directly used as the retained feature combination terms to obtain the third model.

[0015] Alternatively, as another implementation, if some architecture parameters become zero after optimization, further, among the feature combination items corresponding to architecture parameters whose values ​​are not zero after optimization, the feature combination items corresponding to architecture parameters whose values ​​are less than a threshold are deleted to obtain the third model.

[0016] In existing automatic feature combination selection schemes, all possible feature combination subsets are treated as a search space, and a discrete algorithm is used to select the best candidate subset from a randomly selected set of n candidate subsets as the selected feature combination. However, evaluating each candidate subset requires performing a training operation, resulting in high computational cost and high power consumption.

[0017] In this application, by introducing architecture parameters into the FM-based model and optimizing these parameters, feature combination selection can be achieved. In other words, this application only requires optimizing the architecture parameters; a single model training process is sufficient to select feature combinations, eliminating the need for multiple training iterations for multiple candidate subsets as in existing technologies. Therefore, the computational cost of feature combination selection can be effectively reduced, saving computing power and improving efficiency.

[0018] Furthermore, existing automatic feature combination and selection schemes are computationally intensive and consume a lot of computing power, making them unsuitable for deep learning models with long training times.

[0019] In this application, feature combination filtering can be achieved simply through the optimization learning process of architecture parameters. In other words, feature combination filtering can be completed in just one end-to-end model training process, making the time for feature combination filtering (or searching) equivalent to the time for one model training. Therefore, it can be applied to deep learning models with long training times.

[0020] Existing FM models are difficult to extend to higher orders because they require enumerating all combinations of features.

[0021] In this application, by introducing architecture parameters into the FM-based model and optimizing the learning of architecture parameters, feature combination filtering can be achieved. Therefore, the solution of this application can support the extension of feature combination terms in the FM-based model to higher orders.

[0022] Various optimization algorithms (or optimizers) can be used to optimize the architecture parameters in the second model.

[0023] In conjunction with the first aspect, in one possible implementation of the first aspect, the optimization learning makes the optimized architecture parameters exhibit sparsity.

[0024] In this application, by optimizing the learning of the architecture parameters, the architecture parameters are made sparse, which is beneficial for the subsequent reduction of feature combination items.

[0025] Optionally, in the implementation of the optimization learning that makes the optimized architecture parameters sparse, obtaining a third model after feature combination term reduction based on the optimized architecture parameters and the first model or the second model includes: based on the first model or the second model, reducing the feature combination terms corresponding to the architecture parameters whose values ​​are less than a threshold in the optimized architecture parameters to obtain the third model.

[0026] As one implementation method, in the first model, the feature combination items corresponding to the architecture parameters whose values ​​are less than a threshold in the optimized architecture parameters are removed to obtain the third model.

[0027] As another implementation, in the second model, the feature combination items corresponding to the architecture parameters whose values ​​are less than the threshold in the optimized architecture parameters are removed to obtain the third model.

[0028] It should be understood that by reducing the feature combination terms based on the second model, a third model is obtained. This third model can have optimized architectural parameters that represent the importance of the feature combination terms. Subsequently, by training the third model, the importance of the feature combination terms can be further learned.

[0029] In conjunction with the first aspect, in one possible implementation of the first aspect, the optimization learning makes the value of the architecture parameter of at least one feature combination term equal to zero after optimization.

[0030] If we consider the feature combination terms corresponding to the architecture parameters that take a value of zero after optimization as unimportant feature combination terms, then the optimization learning that makes the architecture parameter of at least one feature combination term equal to zero after optimization can be regarded as making the architecture parameter of unimportant feature combination terms equal to zero after optimization.

[0031] Optionally, the optimization learning uses a GRDA optimizer to optimize the architecture parameters in the second model, wherein the GRDA optimizer causes the values ​​of the architecture parameters of at least one feature combination term to tend to zero during the optimization process.

[0032] In this application, by optimizing the architecture parameters, some architecture parameters are made to approach zero. This is equivalent to eliminating some unimportant feature combinations during the optimization process of the architecture parameters. In other words, by optimizing the architecture parameters, both the optimization of the architecture parameters and the screening of feature combinations are achieved, which can effectively improve the efficiency of feature combination screening and further reduce the amount of computation and computing power consumption.

[0033] Furthermore, during the optimization of architectural parameters, removing some unimportant feature combinations can avoid the noise generated by these unimportant feature combinations, thereby allowing the model to gradually approach the ideal model during the optimization process. This also makes the estimation of other parameters in the model (e.g., architectural parameters and model parameters of the feature combinations that have not been removed) more accurate.

[0034] Optionally, in the implementation where the optimization learning makes the optimized architecture parameters sparse and the value of the architecture parameter of at least one feature combination term is equal to zero after optimization, the step of obtaining a third model after feature combination term reduction based on the optimized architecture parameters and the first model or the second model includes: deleting feature combination terms other than those corresponding to the optimized architecture parameters to obtain the third model.

[0035] Optionally, in the first model, feature combination terms other than those corresponding to the optimized architecture parameters are deleted to obtain the third model. That is, the third model is obtained by reducing the feature combination terms based on the first model.

[0036] Optionally, the second model after architectural parameter optimization can be used as the third model. That is, the third model is obtained by reducing the feature combination terms based on the second model.

[0037] It should be understood that by reducing the feature combination terms based on the second model, a third model is obtained. This third model can have optimized architectural parameters that represent the importance of the feature combination terms. Subsequently, by training the third model, the importance of the feature combination terms can be further learned.

[0038] Optionally, in the implementation where the optimization learning makes the optimized architecture parameters sparse and the value of the architecture parameter of at least one feature combination term is equal to zero after optimization, the step of obtaining a third model after feature combination term reduction based on the optimized architecture parameters and the first model or the second model includes: deleting feature combination terms other than those corresponding to the optimized architecture parameters, and then deleting feature combination terms corresponding to architecture parameters whose values ​​are less than a threshold in the optimized architecture parameters to obtain the third model.

[0039] Optionally, in the first model, feature combination terms other than those corresponding to the optimized architecture parameters are deleted, and feature combination terms corresponding to architecture parameters whose values ​​are less than the threshold are reduced to obtain the third model.

[0040] Optionally, in the second model after optimizing the architecture parameters, the feature combination terms corresponding to the architecture parameters whose values ​​are less than the threshold are removed to obtain the third model.

[0041] It should be understood that by reducing the feature combination terms based on the second model, a third model is obtained. This third model can have optimized architectural parameters that represent the importance of the feature combination terms. Subsequently, by training the third model, the importance of the feature combination terms can be further learned.

[0042] In conjunction with the first aspect, in one possible implementation of the first aspect, the method further includes optimizing the model parameters in the second model, wherein the optimization learning includes scalarizing the model parameters in the second model.

[0043] Model parameters refer to the weight parameters of the feature combination terms themselves in the second model, excluding the architecture parameters. Alternatively, model parameters refer to the original parameters in the first model.

[0044] As one implementation, the optimization learning includes performing batch normalization (BN) on the model parameters in the second model.

[0045] It should be understood that by scalarizing the model parameters of the feature combination, the model parameters and architecture parameters of the feature combination can be decoupled, allowing the architecture parameters to more accurately reflect the importance of the feature combination, thereby improving the optimization accuracy of the architecture parameters.

[0046] In conjunction with the first aspect, in one possible implementation of the first aspect, optimizing the architecture parameters in the second model and optimizing the model parameters in the second model includes: using the same training data, simultaneously optimizing the architecture parameters and model parameters in the second model to obtain optimized architecture parameters.

[0047] In other words, in each round of training during the optimization process, the architecture parameters and model parameters are simultaneously optimized and learned based on the same batch of training data.

[0048] Alternatively, the architecture parameters and model parameters in the second model can be treated as decision variables at the same level, and the architecture parameters and model parameters in the second model can be optimized and learned simultaneously to obtain optimized architecture parameters.

[0049] In this application, by performing a layer of optimization processing on the architecture parameters and model parameters in the second model, the architecture parameters in the second model can be optimized simultaneously. This reduces the time consumed in the optimization learning process of the architecture parameters in the second model, thereby helping to improve the efficiency of feature combination item selection.

[0050] In conjunction with the first aspect, in one possible implementation of the first aspect, the method further includes: training the third model to obtain a click-through rate (CTR) prediction model or a conversion rate (CVR) prediction model.

[0051] Secondly, a data processing method is provided, which includes: inputting the data of a target object into a click-through rate (CTR) prediction model or a conversion rate (CVR) prediction model to obtain the prediction result of the target object; and determining the recommendation status of the target object based on the prediction result of the target object.

[0052] The CTR prediction model or the CVR prediction model is obtained through the method of the first aspect.

[0053] The steps for training the third model include: using the training samples of the target object to train the third model to obtain the CTR prediction model or the CVR prediction model.

[0054] Optionally, the step of optimizing the architecture parameters includes: using the same training data from the training samples of the target object, simultaneously optimizing the architecture parameters and model parameters in the second model to obtain optimized architecture parameters.

[0055] Thirdly, a data processing apparatus is provided, the apparatus comprising the following units.

[0056] The first processing unit is used to add architecture parameters to each feature combination in the first model to obtain the second model, wherein the first model is a factorization machine (FM) based model, and the architecture parameters are used to characterize the importance of the corresponding feature combination.

[0057] The second processing unit is used to perform optimization learning on the architecture parameters in the second model to obtain optimized architecture parameters.

[0058] The third processing unit is used to obtain a third model after feature combination item reduction based on the first model or the second model, according to the optimized architecture parameters.

[0059] In conjunction with the third aspect, in one possible implementation of the third aspect, the second processing unit performs optimization learning on the architecture parameters such that the optimized architecture parameters exhibit sparsity.

[0060] In conjunction with the third aspect, in one possible implementation of the third aspect, the third processing unit is used to, based on the first model or the second model, delete the feature combination items corresponding to the architecture parameters whose values ​​are less than a threshold in the optimized architecture parameters to obtain the third model.

[0061] In conjunction with the third aspect, in one possible implementation of the third aspect, the second processing unit performs optimization learning on the architecture parameters such that the value of the architecture parameter of at least one feature combination item is equal to zero after optimization.

[0062] In conjunction with the third aspect, in one possible implementation of the third aspect, the third processing unit is used to optimize the architecture parameters in the second model using a generalized normal dual average (GRDA) optimizer, wherein the GRDA optimizer causes the architecture parameters of at least one feature combination term to tend to zero during the optimization process.

[0063] In conjunction with the third aspect, in one possible implementation of the third aspect, the second processing unit is further configured to perform optimization learning on the model parameters in the second model, wherein the optimization learning includes scalarizing the model parameters in the second model.

[0064] In conjunction with the third aspect, in one possible implementation of the third aspect, the second processing unit is used to perform batch normalization (BN) processing on the model parameters in the second model.

[0065] In conjunction with the third aspect, in one possible implementation of the third aspect, the second processing unit is used to simultaneously optimize and learn the architecture parameters and model parameters in the second model using the same training data to obtain optimized architecture parameters.

[0066] In conjunction with the third aspect, in one possible implementation of the third aspect, the apparatus further includes: a training unit for training the third model to obtain a CTR prediction model or a CVR prediction model.

[0067] Fourthly, a data processing apparatus is provided, the apparatus comprising the following units.

[0068] The first processing unit is used to input the data of the target object into the click-through rate (CTR) prediction model or the conversion rate (CVR) prediction model to obtain the prediction result of the target object.

[0069] The first processing unit is used to determine the recommendation status of the target object based on the prediction result of the target object.

[0070] The CTR prediction model or the CVR prediction model is obtained through the method of the first aspect.

[0071] The steps for training the third model include: using the training samples of the target object to train the third model to obtain the CTR prediction model or the CVR prediction model.

[0072] Optionally, the step of optimizing the architecture parameters includes: using the same training data from the training samples of the target object, simultaneously optimizing the architecture parameters and model parameters in the second model to obtain optimized architecture parameters.

[0073] Fifthly, an image processing apparatus is provided, the apparatus comprising: a memory for storing a program; and a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor performs the method described in the first or second aspect.

[0074] A sixth aspect provides a computer-readable medium storing program code for execution by a device, the program code including methods for performing the first or second aspect described above.

[0075] In a seventh aspect, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the methods described in the first or second aspect above.

[0076] Eighthly, a chip is provided, the chip including a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes the methods in the first or second aspect described above.

[0077] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to perform the method in the first or second aspect described above.

[0078] Ninth aspect, an electronic device is provided, which includes the means provided in the third, fourth, fifth or sixth aspects described above.

[0079] As described above, this application introduces architecture parameters into an FM-based model and achieves feature combination selection through optimization learning of these parameters. In other words, this application can achieve feature combination selection through optimization learning of architecture parameters without requiring multiple training iterations for multiple candidate subsets as in existing technologies. Therefore, it can effectively reduce the computational load of feature combination selection, thereby saving computing power and improving the efficiency of feature combination selection.

[0080] Furthermore, the scheme provided in this application can support the extension of feature combination terms in FM-based models to higher orders. Attached Figure Description

[0081] Figure 1 This is a schematic diagram of the FM model architecture.

[0082] Figure 2 This is a schematic diagram of the training of the FM model.

[0083] Figure 3 This is a schematic flowchart of the data processing method provided in the embodiments of this application.

[0084] Figure 4 This is a schematic diagram of the architecture of the FM model in the embodiments of this application.

[0085] Figure 5 This is another illustrative flowchart of the data processing method provided in the embodiments of this application.

[0086] Figure 6 This is another illustrative flowchart of the data processing method provided in the embodiments of this application.

[0087] Figure 7 This is a schematic block diagram of a data processing apparatus provided in an embodiment of this application.

[0088] Figure 8 This is another schematic block diagram of the data processing apparatus provided in the embodiments of this application.

[0089] Figure 9 This is another schematic block diagram of the data processing apparatus provided in the embodiments of this application.

[0090] Figure 10 This is a schematic diagram of a chip hardware structure provided in an embodiment of this application. Detailed Implementation

[0091] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0092] With the rapid development of technology and the ever-increasing volume of data, recommender systems (RS) have been proposed to address the problem of information overload. Recommender systems work by feeding users' historical behavior, interests, or demographic characteristics into a recommendation algorithm, which then generates a list of items the user might be interested in.

[0093] In recommender systems, click-through rate (CTR) prediction (and conversion rate (CVR) prediction) is a crucial step. Determining whether to recommend a product depends on the predicted CTR. When predicting CTR, in addition to considering individual features, combinations of features are often also considered. Feature combinations are very important for recommendation ranking, and factorization machines (FM) can represent these combinations. Factorization machines can be simply referred to as FM models.

[0094] Based on the maximum order of the feature combination terms, an FM model can be called a *-th order FM model. For example, an FM model with a maximum order of second-order feature combination terms can be called a second-order FM model, and an FM model with a maximum order of third-order feature combination terms can be called a third-order FM model.

[0095] The order of a feature combination term indicates that the feature combination term is an interaction term of several features. For example, an interaction term of 2 features can be called a second-order feature combination term, and an interaction term of 3 features can be called a third-order feature combination term.

[0096] As an example, the second-order FM model is shown in the following formula (1).

[0097]

[0098] x represents the eigenvector, x i Let x represent the i-th feature. j Let represent the j-th feature. m represents the feature dimension, also known as the feature field. w0 represents the global bias, w0∈R. i Let w represent the intensity of the i-th feature, where w ∈ R. m v i Represents the i-th feature x i Auxiliary vector, v j Represents the j-th feature x j The auxiliary vector. k is v i With v j The dimension of a two-dimensional matrix v∈R. m×k .

[0099] Where, x i x j Represents the i-th feature x i With the j-th feature x j The combination of .

[0100] <v i ,v j > indicates v i With v jThe inner product of x represents the i-th feature x. i With the j-th feature x j The interaction between them. <v i ,v j This can also be understood as the feature combination term x. i x j Weight parameters, for example, <v i ,v j >Can be denoted as w ij .

[0101] In this article, <v i ,v j Let x be the characteristic combination term. i x j The weight parameters.

[0102] Alternatively, formula (1) can also be expressed as the following formula (2):

[0103]

[0104] Among them, in formula (2) <e i ,e j > indicates that in formula (1) <v i ,v j >x i x j In formula (2)<w,x> In formula (1)

[0105] As another example, the third-order FM model is shown in the following formula (3).

[0106]

[0107] The FM model includes feature combination terms for all combinations of a single feature. For example, in the second-order FM model shown in formula (1) or formula (2), there are combination feature terms for all combinations of second-order features of a single feature. As another example, in the third-order FM model shown in formula (3), there are combination feature terms for all combinations of second-order features of a single feature, as well as combination feature terms for all combinations of third-order features.

[0108] For example, the industry will acquire feature x i Auxiliary vector v i The operation is called embedding, which involves modifying the data based on feature x. i and its auxiliary vector v i The operation of constructing feature interaction items is called interaction. Figure 1 This is a schematic diagram of the FM model architecture. Figure 1As shown, the FM model can be viewed as a neural network model, including an input layer, an embedding layer, an interaction layer, and an output layer. The input layer generates features, where domain 1, domain 2, ..., domain m represent the feature dimensions. The embedding layer generates auxiliary vectors for the features. The interaction layer generates feature combinations based on the features and their auxiliary vectors. The output layer outputs the results of the FM model.

[0109] Existing technologies typically use FM (Fast Moving Average) to predict CTR (Cross-Trip Transmission) or CVR (Cross-Rating Transmission).

[0110] Current FM-based models include: FM model, DeepFM model, IPNN model, AFM model, and NFM model.

[0111] As an example, and not a limitation, the process of building an FM model is as follows: Figure 2 As shown.

[0112] S210, enumerate feature combination terms and enter them into the FM model.

[0113] For example, use formula (1) or formula (3) to construct the FM model.

[0114] S220, train the FM model until it converges, that is, obtain an FM model that can be put into application.

[0115] After training the FM model, you can use the trained FM model for online inference, such as... Figure 2 The step S230 shown.

[0116] As described earlier, the FM model includes feature combinations for all possible combinations of a single feature. Therefore, training the FM model is computationally intensive and time-consuming.

[0117] Furthermore, it can be seen from formulas (1) and (3) that the number of feature combination terms in the FM model increases dramatically with the increase of feature dimension and the order of feature combination.

[0118] For example, in formula (1), as the feature dimension m increases, the number of feature combination terms increases exponentially. Furthermore, from a second-order FM model to a third-order FM model, the increase in the number of feature combination terms in the FM model is enormous as the order of the feature combination increases.

[0119] Therefore, increasing the feature dimension and the order of feature combinations places a significant burden on the inference latency and computational cost of FM models, thus limiting the maximum feature dimension and the order of feature combinations that FM models can accommodate. For example, current FM models are difficult to extend to higher orders.

[0120] To address this problem, feature interaction selection (FIS) was proposed.

[0121] Some existing technologies use manual screening to select feature combinations. Finding suitable feature combinations may require several years of exploration by engineers. This manual screening method not only consumes a huge amount of manpower but may also miss important feature combinations.

[0122] To address the drawbacks of manual screening, the industry has proposed an automatic feature interaction selection (AutoFIS) solution. Compared to manual screening, AutoFIS can identify valuable feature combinations in a shorter time.

[0123] Current technology proposes an automatic feature combination selection scheme. This scheme treats all possible feature combination subsets as the search space and uses a discrete algorithm to select the best candidate subset from a randomly selected set of n candidate subsets as the selected feature combination. However, evaluating each candidate subset requires performing a training operation, resulting in high computational cost and power consumption. Furthermore, while fully training the entire model improves evaluation accuracy, it leads to enormous search costs; using mini-batches for approximation results in inaccurate evaluations. Additionally, the search space grows exponentially with the order of the feature combination, further exacerbating the energy consumption of the search process.

[0124] Therefore, existing automatic feature combination screening schemes involve large computational loads, high computing power consumption, and high energy consumption during the search process.

[0125] To address the aforementioned issues, this application proposes an automatic feature combination screening scheme, which, compared to existing technologies, can reduce the computational power consumption of automatic feature combination screening and improve its efficiency.

[0126] Figure 3 This is a schematic flowchart of a data processing method 300 provided in an embodiment of this application. The method 300 includes the following steps S310, S320 and S330.

[0127] S310, add architecture parameters to each feature combination item in the first model to obtain the second model.

[0128] The first model is based on the factorization machine (FM) model. In other words, the first model includes feature combination terms for all combinations of a single feature, or enumerates feature combination terms for all combinations.

[0129] For example, the first model can be any of the following FM-based models: FM model, DeepFM model, IPNN model, AFM model, and NFM model.

[0130] As an example, the first model is a second-order FM model as shown in formula (1) or formula (2), or the first model is a third-order FM model as shown in formula (3).

[0131] This application aims to achieve the filtering of feature combination items. The first model can be regarded as a model for reducing feature combination items.

[0132] Adding an architecture parameter to each feature combination term in the first model means adding a coefficient to each feature combination term in the first model; in this application, this coefficient is referred to as an architecture parameter. The architecture parameter is used to characterize the importance of the corresponding feature combination term. The model obtained after adding the architecture parameter to each feature combination term in the first model is denoted as the second model.

[0133] As an example, assuming the first model is a second-order FM model as shown in formula (1), the second model can be as shown in formula (4).

[0134]

[0135] x represents the eigenvector, x i Let x represent the i-th feature. j Let represent the j-th feature. m represents the feature dimension, also known as the feature field. w0 represents the global bias, w0∈R. i Let w represent the intensity of the i-th feature, where w ∈ R. m v i Represents the i-th feature x i Auxiliary vector, v j Represents the j-th feature x j The auxiliary vector. k is v i With v j The dimension of a two-dimensional matrix v∈R. m×k .

[0136] Where, x i x j Represents the i-th feature x i With the j-th feature x j The combination of .

[0137] <v i ,v j > Represents the feature combination term x i x j The weighting parameter, α( i,j ) represents the characteristic combination term x i x j Architecture parameters.

[0138] <v i ,v j > indicates v i With v j The inner product of x represents the i-th feature x. i With the j-th feature x j The interaction between them. <v i ,v j This can also be understood as the feature combination term x. i x j Weight parameters, for example, <v i ,v j >Can be denoted as w ij .

[0139] Assuming the first model is expressed as the second-order FM model shown in formula (2), the second model can be expressed as the following formula (5).

[0140]

[0141] Where, α (i,j) The architecture parameters represent the feature combination items.

[0142] As another example, assuming the first model is a third-order FM model as shown in formula (3), the second model can be as shown in formula (6).

[0143]

[0144] Where, α (i,j) With α (i,j,t) The architecture parameters represent the feature combination items.

[0145] For ease of understanding and description, the following conventions are made in this paper: the original weight parameters of the feature combination terms in the first model (such as those in formula (4)) will be used instead of the weight parameters of the feature combination terms in the first model. <v i ,v j >) are called model parameters.

[0146] In other words, in the second model, each feature combination term has two types of coefficient parameters: model parameters and architecture parameters.

[0147] Figure 4 This is a schematic diagram illustrating the feature combination item filtering provided in the embodiments of this application. Figure 4 The meaning of the embedded layer and the interaction layer in the text Figure 1 The embedded layer and the interaction layer in this context have the same meaning. For example... Figure 4 As shown in this embodiment, an architecture parameter α is added to each feature combination item in the interaction layer. (i,j) (like Figure 4 The α shown (1,2) α (1,m) α (m-1,m) ). can Figure 4 The middle interaction layer is considered as the first model, and an architectural parameter α is added to the feature combination term. (i,j) The subsequent interaction layer can be regarded as a second model.

[0148] S320 performs optimization learning on the architecture parameters in the second model to obtain optimized architecture parameters.

[0149] For example, using training data, the architecture parameters in the second model are optimized to obtain optimized architecture parameters.

[0150] For example, the optimized architecture parameters can be considered as the optimal value α of the architecture parameters in the second model. * .

[0151] In this embodiment, the architecture parameters characterize the importance of the corresponding feature combinations. Therefore, optimizing the architecture parameters is equivalent to learning the importance of each feature combination or the contribution of each feature combination to the model's prediction. In other words, the optimized architecture parameters characterize the importance of the learned feature combinations.

[0152] In other words, in the embodiments of this application, the contribution (or importance) of each feature combination can be learned by the architecture parameters in an end-to-end manner.

[0153] S330, based on the optimized architecture parameters, obtains a third model after feature combination term reduction based on the first or second model.

[0154] The third model can be obtained by reducing the feature combination terms based on the first model.

[0155] Alternatively, the third model can be obtained by reducing the feature combination terms based on the second model.

[0156] There are several ways to determine which feature combinations to remove, or in other words, which feature combinations to retain (i.e., filter).

[0157] Alternatively, as an implementation method, feature combination items corresponding to architecture parameters whose values ​​are less than a threshold can be deleted from the optimized architecture parameters.

[0158] The threshold is used to characterize the criterion for whether to retain a feature combination item. For example, if the value of the optimized architecture parameter of a feature combination item is less than the threshold, it indicates that the feature combination item is deleted. If the value of the optimized architecture parameter of a feature combination item reaches the threshold, it indicates that the feature combination item is retained (i.e., filtered).

[0159] This threshold can be determined based on actual application requirements. For example, the value of this threshold can be obtained through model training. This application does not limit the method of obtaining this threshold.

[0160] See also Figure 4 Assuming the optimized architecture parameter α (1,2) If it is less than the threshold, then the architecture parameter α can be... (1,2) The corresponding feature combination terms are deleted, assuming the optimized architecture parameter α. (1,m) Once the threshold is reached, the architecture parameter α can be adjusted. (1,m) The corresponding feature combination terms are retained. After reducing the feature combination terms based on the optimized architecture parameters, the next layer model is obtained, such as... Figure 4 As shown. Figure 3 The third model in the embodiment is, for example, Figure 4 The next layer model is shown below.

[0161] As an example rather than a limitation, such as Figure 4 As shown, the operation of deciding whether to delete a corresponding feature combination item based on the optimized architecture parameters can be denoted as a selection gate.

[0162] It should be noted that, Figure 4 This is for illustrative purposes only and not as a limitation.

[0163] Alternatively, as another implementation, if some architectural parameters become zero after optimization, the feature combination terms corresponding to the architectural parameters whose values ​​are not zero after optimization can be directly used as the retained feature combination terms to obtain the third model.

[0164] Alternatively, as another implementation, if some architecture parameters become zero after optimization, further, among the feature combination items corresponding to architecture parameters whose values ​​are not zero after optimization, the feature combination items corresponding to architecture parameters whose values ​​are less than a threshold are deleted to obtain the third model.

[0165] The phrase "model after feature combination term reduction" in this article can be replaced with "model after feature combination term filtering".

[0166] As described above, existing automatic feature combination screening schemes treat all possible feature combination subsets as the search space, and use discrete algorithms to select the best candidate subset from a randomly selected set of n candidate subsets as the filtered feature combination. However, evaluating each candidate subset requires performing a training operation, resulting in high computational cost and high power consumption.

[0167] In this embodiment, by introducing architecture parameters into the FM-based model and optimizing these parameters, feature combination selection can be achieved. In other words, this application only requires optimizing the architecture parameters; a single model training process is sufficient to select feature combinations, eliminating the need for multiple training iterations for multiple candidate subsets as in existing technologies. Therefore, the computational load for feature combination selection can be effectively reduced, saving computing power and improving efficiency.

[0168] Furthermore, existing automatic feature combination screening schemes achieve feature combination screening by searching for a subset of candidates from a search space. It can be understood that existing technologies treat feature combination screening as a discrete problem, i.e., searching for a discrete set of candidate feature combinations.

[0169] In this embodiment, feature combination filtering is achieved through optimized learning of the architecture parameters introduced into the FM-based model. It can be understood that this embodiment makes the existing problem of searching discrete feature combination candidate sets continuous, or in other words, it solves feature combination filtering as a continuous problem. For example, the automatic feature combination filtering scheme provided in this application can be described as a combination feature search scheme based on a continuous search space. In other words, in this embodiment, the operation of introducing architecture parameters into the FM-based model can be regarded as a continuous modeling of the problem of automatic feature combination filtering.

[0170] Furthermore, existing automatic feature combination and selection schemes are computationally intensive and consume a lot of computing power, making them unsuitable for deep learning models with long training times.

[0171] In the embodiments of this application, feature combination filtering can be achieved simply through the optimization learning process of architecture parameters. In other words, feature combination filtering can be completed in just one end-to-end model training process, so that the time for feature combination filtering (or searching) can be equivalent to the time for one model training. Therefore, it can be applied to deep learning models with long training times.

[0172] As described above, the existing FM model is difficult to extend to higher orders because it requires enumerating all combinations of features.

[0173] In the embodiments of this application, by introducing architectural parameters into the FM-based model and optimizing the learning of architectural parameters, feature combination filtering can be achieved. Therefore, the solution provided in the embodiments of this application can support the extension of feature combination items in the FM-based model to higher orders.

[0174] For example, the FM model constructed using the scheme provided in the embodiments of this application can be extended to third order or higher.

[0175] For example, the DeepFM model constructed using the scheme provided in the embodiments of this application can be extended to third order or higher.

[0176] In this embodiment, feature combination filtering can be achieved by introducing architectural parameters into a traditional FM-based model and optimizing these parameters. Alternatively, this embodiment constructs an FM-based model incorporating architectural parameters and optimizes these parameters to achieve feature combination filtering. Specifically, this method involves adding architectural parameters before each feature combination item in a traditional FM-based model.

[0177] like Figure 3 As shown, the method 300 may further include step S340.

[0178] S340, Train the third model.

[0179] Step S340 can also be understood as retraining the model. It can be understood that steps S310, S320, and S330 reduced the number of feature combination terms. In step S340, the model after reducing the number of feature combination terms is retrained.

[0180] In step S340, the third model can be trained directly; or, L1 regularization terms and / or L2 regularization terms can be added to the third model before training.

[0181] For example, the training objective of training the third model can be determined based on the application requirements.

[0182] For example, if we want to obtain a click-through rate (CTR) prediction model, we would train a third model with the goal of obtaining a CTR prediction model, in order to obtain the CTR prediction model.

[0183] For example, if we want to obtain a conversion rate (CVR) prediction model, we would train a third model with the goal of obtaining the CVR prediction model, in order to obtain the CVR prediction model.

[0184] In step S320, for example, various optimization algorithms (or optimizers) can be used to optimize and learn the architecture parameters in the second model.

[0185] The first optimization algorithm.

[0186] Optionally, in step S320, the architecture parameters are optimized and learned so that the optimized architecture parameters exhibit sparsity.

[0187] For example, in step S320, the architecture parameters in the second model are optimized using Lasso (least absolute shrinkage and selection operator) regularization. Lasso can also be called the minimum absolute shrinkage and selection operator.

[0188] Taking the second model as an example (equation (5), in step S320, the following formula (7) can be used to optimize the architecture parameters in the second model:

[0189]

[0190] in, Let y represent the loss function and y represent the model observations. This represents the model's predicted value. λ represents a constant, which can be assigned a value according to specific needs.

[0191] It should be understood that formula (7) represents the constraints for optimizing the architecture parameters.

[0192] By making the optimized architecture parameters sparse, it is beneficial to reduce the number of feature combination items in the future.

[0193] Optionally, as an embodiment, in step S320, the architecture parameters are optimized and learned to make the optimized architecture parameters sparse; in step S330, based on the first model or the second model, the feature combination terms corresponding to the architecture parameters whose values ​​are less than a threshold in the optimized architecture parameters are deleted to obtain the third model.

[0194] The threshold is used to characterize the criterion for whether to retain a feature combination item. For example, if the value of the optimized architecture parameter of a feature combination item is less than the threshold, it indicates that the feature combination item is deleted. If the value of the optimized architecture parameter of a feature combination item reaches the threshold, it indicates that the feature combination item is retained (i.e., filtered).

[0195] This threshold can be determined based on actual application requirements. For example, the value of this threshold can be obtained through model training. This application does not limit the method of obtaining this threshold.

[0196] In this embodiment of the application, by optimizing the learning of the architecture parameters, the architecture parameters are made sparse, which is beneficial for the selection of feature combination items.

[0197] It should be understood that the architecture parameters in the second model represent the importance or contribution of the corresponding feature combination. If the value of a certain architecture parameter after optimization is less than the threshold, for example, close to zero, it indicates that the feature combination corresponding to the architecture parameter is not important or has a very small contribution. Deleting (or removing or pruning) such feature combination can not only remove the noise it introduces, but also reduce energy consumption and improve the inference speed of the model.

[0198] Therefore, removing feature combination items corresponding to architecture parameters whose values ​​are less than the threshold from the optimized architecture parameters is a reasonable feature combination filtering operation.

[0199] The second optimization algorithm.

[0200] Optionally, in step S320, the architecture parameters are optimized and learned so that the optimized architecture parameters are sparse, and the value of the architecture parameter of at least one feature combination term is equal to zero after optimization.

[0201] If we consider the feature combination items corresponding to the architecture parameters that take a value of zero after optimization as unimportant feature combination items, then the optimization learning of the architecture parameters in step S320 can be regarded as making the architecture parameters of unimportant feature combination items equal to zero after optimization.

[0202] In other words, the optimization learning of the architecture parameters makes the value of the architecture parameter of at least one feature combination term tend to zero during the optimization process.

[0203] For example, in step S320, the architecture parameters in the second model are optimized using a generalized regularized dual averaging (GRDA) optimizer. The GRDA optimizer not only achieves sparsity of architecture parameters, but also allows the values ​​of the architecture parameters of at least one feature combination term to gradually approach zero during the optimization process.

[0204] As an example, in step S320, the architecture parameters in the second model are optimized using the following formula (8):

[0205]

[0206] Where γ represents the learning rate. i+1 Represents the model observations. g(t,γ)=cγ 1 / 2 (tγ)μ c and μ represent adjustable hyperparameters. The goal of adjusting c and μ is to find a balance between model accuracy and the sparsity of the architecture parameter α.

[0207] It should be understood that formula (8) represents the constraints for optimizing the architecture parameters.

[0208] It should also be understood that in this embodiment, in step S320, the second model after the architecture parameter optimization is already a model that has been filtered by feature combination items.

[0209] In this embodiment of the application, by optimizing the learning of architectural parameters, some architectural parameters are made to approach zero. This is equivalent to eliminating some unimportant feature combinations during the optimization process of architectural parameters. In other words, by optimizing the learning of architectural parameters, both the optimization of architectural parameters and the screening of feature combinations are achieved, which can effectively improve the efficiency of feature combination screening and further reduce the amount of computation and computing power consumption.

[0210] Furthermore, during the optimization of architectural parameters, removing some unimportant feature combinations can avoid the noise generated by these unimportant feature combinations, thereby allowing the model to gradually approach the ideal model during the optimization process. This also makes the estimation of other parameters in the model (e.g., architectural parameters and model parameters of the feature combinations that have not been removed) more accurate.

[0211] Optionally, as an embodiment, in step S320, the optimization learning of the architecture parameters makes the optimized architecture parameters sparse, and makes the value of the architecture parameter of at least one feature combination item equal to zero after optimization; in step S330, the third model can be obtained in a variety of ways.

[0212] Method (1)

[0213] In step S330, the feature combination items corresponding to the optimized architecture parameters can be directly used as the selected feature combination items, and the third model can be obtained based on the selected feature combination items.

[0214] For example, in the first model, the feature combination items corresponding to the optimized architecture parameters are used as the filtered feature combination items, and the remaining feature combination items are deleted to obtain the third model.

[0215] For example, the second model, which has undergone architecture parameter optimization, can be used as the third model.

[0216] Method (2)

[0217] In step S330, among the feature combination items corresponding to the optimized architecture parameters, the feature combination items corresponding to the architecture parameters whose values ​​are less than the threshold are deleted to obtain the third model.

[0218] This threshold can be determined based on actual application requirements. For example, the value of this threshold can be obtained through model training. This application does not limit the method of obtaining this threshold.

[0219] For example, in the first model, feature combination terms other than those corresponding to the optimized architecture parameters are deleted, and feature combination terms corresponding to architecture parameters whose values ​​are less than the threshold are reduced to obtain the third model.

[0220] For example, in the second model after optimizing the architecture parameters, the feature combination items corresponding to the architecture parameters whose values ​​are less than the threshold are removed to obtain the third model.

[0221] In this embodiment of the application, by optimizing the learning of architectural parameters, some architectural parameters are made to approach zero. This is equivalent to eliminating some unimportant feature combinations during the optimization process of architectural parameters. In other words, by optimizing the learning of architectural parameters, both the optimization of architectural parameters and the screening of feature combinations are achieved, which can effectively improve the efficiency of feature combination screening and further reduce the amount of computation and computing power consumption.

[0222] Based on the above description of step S320, it can be seen that the implementation method of obtaining the third model after feature combination item filtering in step S330 can be determined according to the optimization method of architecture parameters in step S320. The implementation method of obtaining the third model is described below in the following two cases.

[0223] In the first case, in step S320, the architecture parameters are optimized and learned, so that the optimized architecture parameters exhibit sparsity.

[0224] In step S330, feature combination terms corresponding to architecture parameters whose values ​​are less than a threshold are removed from the optimized architecture parameters to obtain the third model. This threshold is described above and will not be repeated here.

[0225] As an example, and not a limitation, the optimized architecture parameters obtained after the optimization learning of the architecture parameters is completed (i.e., the optimal training converges) are denoted as the optimal value α of the architecture parameters. * According to the optimal value α * This allows us to determine which feature combinations to retain or remove. For example, if the optimal value of the architecture parameter α for a feature interaction term is... * If (i,j) reaches the threshold, then the feature interaction term should be retained; if the optimal value α of the architectural parameters of a feature interaction term is...* If (i,j) is less than the threshold, then the feature interaction item should be deleted.

[0226] As an example, in the second model, for each feature interaction term, a switch term ψ is set to characterize whether the feature interaction term is retained in the model. (i.j) The second model can be represented by the following formula (9):

[0227]

[0228] Among them, the switching term ψ (i.j) The value of can be expressed by the following formula (10):

[0229]

[0230] Where thr represents the threshold.

[0231] Switch item ψ (i.j The feature combination terms with a value of 0 are removed from the second model to obtain the third model after feature combination term filtering.

[0232] In this embodiment, the switching term ψ (i.j) The setting can be regarded as the criterion for whether to retain feature combination items.

[0233] Optionally, the third model can be a model obtained by reducing the feature combination terms based on the first model.

[0234] For example, the switch term ψ (i.j) Feature combination terms that are zero are removed from the first model, resulting in the third model after feature combination term filtering.

[0235] Optionally, the third model can be a model obtained by reducing the feature combination terms based on the second model.

[0236] For example, the switch term ψ (i.j) Feature combination terms that are zero are removed from the second model, resulting in the third model after feature combination term filtering.

[0237] It should be understood that in this embodiment, the third model has optimized architectural parameters representing the importance of feature combination terms. The importance of feature combination terms can be further learned through subsequent training of the third model.

[0238] In the second case, in step S320, the architecture parameters are optimized and learned so that the optimized architecture parameters are sparse, and the value of the architecture parameter of at least one feature combination item is equal to zero after the optimization is completed.

[0239] Optionally, in step S330, feature combination terms other than those corresponding to the optimized architecture parameters are deleted to obtain the third model.

[0240] As an example, in step S330, in the first model, feature combination terms other than those corresponding to the optimized architecture parameters are deleted to obtain the third model. That is, the third model is obtained by reducing feature combination terms based on the first model.

[0241] As another example, in step S330, the second model after the architecture parameter optimization is used as the third model. That is, the third model is obtained by reducing the feature combination terms based on the second model.

[0242] Optionally, in step S330, feature combination items other than those corresponding to the optimized architecture parameters are deleted, and feature combination items corresponding to architecture parameters whose values ​​are less than the threshold are reduced to obtain the third model.

[0243] As an example, in step S330, in the first model, feature combination terms other than those corresponding to the optimized architecture parameters are deleted, and then feature combination terms corresponding to architecture parameters whose values ​​are less than a threshold are reduced to obtain the third model. That is, the third model is obtained by reducing feature combination terms based on the first model.

[0244] As another example, in step S330, in the second model after optimizing the architecture parameters, the feature combination terms corresponding to the architecture parameters whose values ​​are less than a threshold are removed to obtain the third model. That is, the third model is obtained by removing feature combination terms based on the second model.

[0245] It should be understood that in the embodiment where the third model is obtained by reducing the feature combination terms based on the second model, the third model has optimized architectural parameters that represent the importance of the feature combination terms. Through subsequent training of the third model, the importance of the feature combination terms can be further learned.

[0246] From formulas (4), (5), or (6), it can be understood that the second model includes two types of parameters: architecture parameters and model parameters. Model parameters represent the weight parameters of the feature combination terms themselves, in addition to the architecture parameters. For example, in the example of the second model as shown in formula (4), α... (i,j) The architecture parameters representing the feature combination terms are: <v i ,v j > represents the model parameters of the feature combination term. For example, in the second model as shown in formula (5), α (i,j) The architecture parameters representing the feature combination terms are: <ei ,e j > can represent model parameters for feature combination terms.

[0247] It can be understood that the optimization learning process for the architecture parameters involves not only the training of the architecture parameters but also the training of the model parameters. In other words, the optimization learning of the architecture parameters in the second model in step S320 is accompanied by the optimization learning of the model parameters in the second model.

[0248] For example, in Figure 3 In the illustrated embodiment, the method 300 further includes: optimization learning of model parameters in the second model, the optimization learning including scalarization processing of the model parameters.

[0249] In each training round of the model parameter optimization learning process, the model parameters in the second model are scalarized.

[0250] For example, by performing batch normalization (BN) on the model parameters in the second model, the model parameters in the second model can be scalarized.

[0251] As an example, taking the second model as shown in formula (5), the model parameters in the second model can be scalarized using the following formula (11):

[0252]

[0253] in, <e i ,e j > BN express <e i ,e j > Batch normalization.

[0254] <e i ,e j > B express <e i ,e j >Min batch data.

[0255] μ B ( <e i ,e j > B )express <e i ,e j The average value of the minimum batch of data.

[0256] express <e i ,e j> The variance of the min batch data.

[0257] θ represents the disturbance.

[0258] See also Figure 4 , Figure 4 The batch normalization (BN) shown indicates that the model parameters in the second model are subjected to BN processing.

[0259] By scalarizing the model parameters of feature combinations, the model parameters and architecture parameters of these feature combinations can be decoupled. This allows the architecture parameters to more accurately reflect the importance of the feature combinations, thereby improving the optimization accuracy of the architecture parameters. The reasons are as follows.

[0260] It should be understood that e i It is constantly updated and changed during the model training process. i With e j After taking the inner product, that is... <e i ,e j Its scale is also constantly updated and changed. Assume α... (i,j) <e i ,e j >It can be done Obtain, of which, the first item With the second term (η· <e i ,e j >) has coupling. If the second term (η· <e i ,e j >) If no scalarization is performed, then the first term This cannot be used to absolutely represent the importance of the second item, as doing so would introduce significant instability into the system.

[0261] In this embodiment, by scalarizing the model parameters of the feature combination terms, α can be avoided. (i,j) <e i ,e j >It can be done The observed phenomenon is that the model parameters and architecture parameters of this feature combination can be decoupled.

[0262] By decoupling the model parameters and architecture parameters of feature combination items, the architecture parameters can more accurately reflect the importance of the feature combination items, thereby improving the optimization accuracy of the architecture parameters.

[0263] In other words, by scalarizing the model parameters of the feature combination, the model parameters and architecture parameters of the feature combination can be decoupled, thus avoiding the significant instability caused by the coupling effect between the model parameters and architecture parameters of the feature combination.

[0264] As described above, the second model includes two types of parameters: architecture parameters and model parameters. The optimization learning process for the architecture parameters involves not only training the architecture parameters but also training the model parameters. In other words, the optimization learning of the architecture parameters in the second model in step S320 is accompanied by the optimization learning of the model parameters within the second model.

[0265] For ease of understanding and description, in the following description, the architecture parameter in the second model will be denoted as α, and the model parameter in the second model will be denoted as w (corresponding to the parameter in formula (4) above). <v i ,v j >).

[0266] Optionally, in Figure 3 In the embodiment shown, the optimization processing of the architecture parameter α and the model parameter w in the second model includes: performing two-level optimization processing on the architecture parameter α and the model parameter w in the second model.

[0267] In other words, in step S320, the optimized architecture parameter α is obtained by performing two-level optimization on the architecture parameter α and the model parameter w in the second model. * .

[0268] In this embodiment, the architecture parameter α in the second model is used as a model hyperparameter for optimization learning, and the model parameter w in the second model is used as a model parameter for optimization learning. That is, the architecture parameter α is used as a high-level decision variable, and the model parameter w is used as a low-level decision variable. Each value of the high-level decision variable α corresponds to a different model.

[0269] Optionally, for any value of the high-level decision variable α corresponding to a model, the optimal model parameters w are obtained by fully training the model when evaluating it. optimal In other words, for each candidate value of an architectural parameter α evaluated, the model corresponding to that candidate value is trained completely once.

[0270] Optionally, for a model corresponding to any value of the high-level decision variable α, when evaluating the model, the w obtained in one step of the model is updated with mini-batches of data. t+1 Replace the optimal model parameters w optimal .

[0271] Optionally, in Figure 3 In the embodiment shown, the optimization processing of the architecture parameter α and the model parameter w in the second model includes: using the same training data, simultaneously optimizing and learning the architecture parameter α and the model parameter w in the second model.

[0272] In other words, in step S320, by using the same training data, the architecture parameters α and model parameters w in the second model are simultaneously optimized to obtain the optimized architecture parameters α. * .

[0273] In this embodiment, during each training round of the optimization process, the architecture parameter α and the model parameter w are simultaneously optimized based on the same batch of training data. Alternatively, the architecture parameter and model parameter in the second model are treated as decision variables at the same level, and the architecture parameter α and model parameter w in the second model are simultaneously optimized to obtain the optimized architecture parameter α. * .

[0274] In this embodiment, the optimization of the architecture parameter α and the model parameter w in the second model can be referred to as one-level optimization.

[0275] For example, in the second model, the architecture parameter α and the model parameter w freely explore their feasible region until convergence in stochastic gradient descent (SGD) optimization.

[0276] As an example, the following formula (12) is used to achieve the optimization learning of the architecture parameter α and the model parameter w in the second model:

[0277]

[0278] Where, α t This represents the architecture parameters after the t-th optimization step. α t-1 This represents the architecture parameters after optimization in step t-1. t This represents the model parameters after the t-th optimization step. t-1 Let η represent the model parameters after optimization in step t-1. t δ represents the optimization learning rate of the architecture parameters during the t-th optimization step. t L represents the learning rate of the model parameters during the t-th optimization step. train (w t-1 ,α t-1 ) represents the value of the loss function on the test set at step t during optimization. This represents the gradient of the loss function with respect to the architecture parameter α at the t-th step of optimization on the test set. Let represent the gradient of the loss function with respect to the model parameters w on the test set at step t.

[0279] In this embodiment, by performing a layer of optimization processing on the architecture parameters and model parameters in the second model, the architecture parameters in the second model can be optimized simultaneously. This reduces the time consumed in the optimization learning process of the architecture parameters in the second model, thereby helping to improve the efficiency of feature combination item selection.

[0280] After completing step S330, the selection of feature combination items is completed, and the third model is the model after the feature combination item selection is completed.

[0281] In step S340, the third model is trained.

[0282] The third model can be trained, or an L1 regularization term and / or an L2 regularization term can be added to the third model before training.

[0283] The training objective for training the third model can be determined based on the application requirements.

[0284] For example, if we want to obtain a click-through rate (CTR) prediction model, we would train a third model with the goal of obtaining a CTR prediction model, in order to obtain the CTR prediction model.

[0285] For example, if we want to obtain a conversion rate (CVR) prediction model, we would train a third model with the goal of obtaining the CVR prediction model, in order to obtain the CVR prediction model.

[0286] Optionally, the third model is obtained by reducing the feature combination terms based on the first model. See the description of step S330 above for details, which will not be repeated here.

[0287] Optionally, the third model is obtained by reducing the feature combination terms based on the second model. See the description of step S330 above for details, which will not be repeated here.

[0288] It should be understood that after completing the reduction (or filtering) of feature combinations, retaining the architectural parameters in the model and training the model can further learn the importance of feature combinations.

[0289] As described above, in this embodiment, by introducing architecture parameters into the FM-based model and optimizing these parameters, feature combination selection can be achieved. In other words, this application can achieve feature combination selection through optimized learning of architecture parameters, without requiring multiple training iterations for multiple candidate subsets as in the prior art. Therefore, it can effectively reduce the computational load of feature combination selection, thereby saving computing power and improving the efficiency of feature combination selection.

[0290] Furthermore, the solution provided in the embodiments of this application can support the extension of feature combination terms in FM-based models to higher orders.

[0291] Figure 5 Another schematic flowchart of the automatic feature combination screening method 500 provided in the embodiments of this application.

[0292] First, obtain the training data.

[0293] For example, assuming the feature dimension is m, then training data is obtained for features in m domains.

[0294] S510, enumerate feature combination terms and enter them into the FM-based model.

[0295] The FM-based model can be the FM model shown in formula (1) or formula (2) above, or any of the following FM-based models: DeepFM model, IPNN model, AFM model and NFM model.

[0296] Enumerating feature combinations for an FM-based model means constructing feature combinations for the FM-based model based on all possible combinations of m features.

[0297] It should be understood that constructing the feature combination term involves auxiliary vectors for m features.

[0298] For example, it can be adopted Figure 1 or Figure 3 The embedding layer shown obtains auxiliary vectors for m features. The technique of obtaining auxiliary vectors for m features through the embedding layer is existing technology and will not be elaborated upon in this paper.

[0299] S520 introduces architecture parameters into FM-based models. Specifically, it adds a coefficient parameter to each feature combination in the FM-based model; this coefficient parameter is called the architecture parameter.

[0300] Step S520 corresponds to step S310 in the above embodiment, and the details are described above.

[0301] Figure 5The FM-based model in the illustrated embodiment corresponds to Figure 3 The first model in the illustrated embodiment, Figure 5 The model shown in the embodiment is the FM-based model after adding architectural parameters. Figure 3 The second model in the illustrated embodiment.

[0302] The S530 performs optimization learning on the architecture parameters until convergence, obtaining the optimized architecture parameters.

[0303] Step S530 corresponds to step S320 in the above embodiment, and the details are described above.

[0304] S540, based on the optimized architecture parameters, reduces the feature combination items to obtain the model after feature combination item reduction.

[0305] Step S540 corresponds to step S330 in the above embodiment, and the details are described above.

[0306] Figure 5 The model corresponding to the feature combination term reduction in the illustrated embodiment Figure 3 The third model in the illustrated embodiment.

[0307] S550 trains the model after feature combination term reduction until convergence, obtaining the CTR prediction model.

[0308] Step S550 corresponds to step S340 in the above embodiment, and the details are described above.

[0309] After training the CTR prediction model, it can be used for online inference.

[0310] For example, by inputting the data of a target object into a CTR prediction model, the output of the CTR prediction model is the click-through rate of that target object. Based on this click-through rate, it can be determined whether to recommend the target object.

[0311] The automatic feature combination screening scheme provided in this application embodiment can be applied to any factorization machine (FM) based model, such as FM model, DeepFM model, IPNN model, AFM model and NFM model, etc.

[0312] As an example, the automatic feature combination screening scheme provided in this application embodiment can be applied to existing FM models.

[0313] For example, an architecture parameter can be introduced into the existing FM model. Through the optimization learning of the architecture parameter, the importance of each feature combination can be obtained. Then, based on the importance of each feature combination, the feature combination can be filtered, and finally the FM model after feature combination filtering is obtained.

[0314] It should be understood that by applying this application to the FM model, feature combination selection of the FM model can be efficiently achieved, thereby supporting the extension of feature combination of the FM model to higher orders.

[0315] As another example, the automatic feature combination screening scheme provided in this application embodiment can be applied to existing deep-FM models.

[0316] For example, an architecture parameter can be introduced into the existing DeepFM model. By optimizing the architecture parameter, the importance of each feature combination can be obtained. Then, based on the importance of each feature combination, the feature combination can be filtered, and finally the DeepFM model after feature combination filtering is obtained.

[0317] It should be understood that by applying this application to the DeepFM model, feature combination selection of the DeepFM model can be achieved efficiently.

[0318] like Figure 6 As shown in the figure, this application embodiment also provides a data processing method 600. The method 600 includes the following steps S610 and S620.

[0319] S610: Input the target object's data into the CTR prediction model or CVR prediction model to obtain the target object's prediction results.

[0320] For example, the target object is a product.

[0321] S620, based on the estimated results of the target object, determine the recommendation status of the target object.

[0322] The CTR prediction model or CVR prediction model is obtained through method 300 provided in the above embodiments, that is, the CTR prediction model or CVR prediction model is obtained through steps S310 to S340 in the above embodiments. See the above description for details, which will not be repeated here.

[0323] In step S340, the training samples of the target object are used to train the third model to obtain the CTR prediction model or the CVR prediction model.

[0324] Optionally, in step S320, the same training data from the training samples of the target object is used to simultaneously optimize the architecture parameters and model parameters in the second model to obtain the optimized architecture parameters.

[0325] Alternatively, the architecture parameters and model parameters in the second model can be treated as decision variables at the same level. Using the training samples of the target object, the architecture parameters and model parameters in the second model can be optimized and learned simultaneously to obtain the optimized architecture parameters.

[0326] Simulation test: The online A / B test showed a significant improvement in CTR prediction accuracy and a significant reduction in inference energy consumption.

[0327] As an example, simulation experiments show that applying the feature combination screening scheme provided in this application to the DeepFM model of the recommendation system and conducting A / B testing online can improve the game download rate by 20%, the CTR prediction accuracy by 20.3%, and the conversion rate (CVR) by 20.1%. Therefore, it can effectively accelerate the inference speed of the model.

[0328] As an example, the FM model and DeepFM model were obtained using the scheme provided in this application on the public dataset Avazu. The performance comparison results of the FM model and DeepFM model obtained using the scheme of this application with the performance of other industry models are shown in Tables 1 and 2. Table 1 shows the comparison of second-order models, and Table 2 shows the comparison of third-order models. The second-order mode indicates that the highest order of the model's feature combination terms is second-order. The third-order mode indicates that the highest order of the model's feature combination terms is third-order.

[0329] Table 1

[0330]

[0331]

[0332] Table 2

[0333]

[0334] The meanings of the horizontal headers in Tables 1 and 2 are as follows.

[0335] AUC (area under curve) represents the area under the curve. Log loss represents the logarithm of the loss value. Top represents the proportion of feature combinations retained after feature combination selection. Time represents the time required for the model to infer two million samples. Search + re-train cost represents the search time plus the retraining time, where the search time represents the time spent in steps S320 and S330 of the above embodiment, and the retraining time represents the time spent in step S340 of the above embodiment. Rel.Impr represents the relative improvement value.

[0336] In Table 1, the meanings of the vertical headers are as follows.

[0337] FM, FwFM, AFM, FFM, and DeepFM represent existing FM-based models. GBDT+LR and GBDT+FFM represent existing models that use manual feature combination selection.

[0338] AutoFM(2nd) represents a second-order FM model obtained using the solution provided in the embodiments of this application. AutoDeepFM(2nd) represents a third-order DeepFM model obtained using the solution provided in the embodiments of this application.

[0339] In Table 2, the meanings of the vertical column headers are as follows.

[0340] FM(3rd) represents the third-order FM model in the prior art. DeepFM(3rd) represents the third-order DeepFM model in the prior art.

[0341] AutoFM(3rd) represents a third-order FM model obtained using the solution provided in the embodiments of this application. AutoDeepFM(3rd) represents a third-order DeepFM model obtained using the solution provided in the embodiments of this application.

[0342] As can be seen from Tables 1 and 2, compared with the prior art, the FM model or DeepFM model obtained by the scheme provided in the embodiments of this application can significantly improve the CTR prediction accuracy and effectively reduce inference time and energy consumption.

[0343] As described above, in this embodiment, by introducing architecture parameters into the FM-based model and optimizing these parameters, feature combination selection can be achieved. In other words, this application only requires optimizing the architecture parameters; a single model training process is sufficient to select feature combinations, eliminating the need for multiple training iterations for multiple candidate subsets as in existing technologies. Therefore, the computational load for feature combination selection can be effectively reduced, saving computing power and improving efficiency.

[0344] Furthermore, the solution provided in the embodiments of this application can support the extension of feature combination terms in FM-based models to higher orders.

[0345] The various embodiments described herein can be independent solutions or combinations thereof based on their inherent logic, and all such solutions fall within the protection scope of this application.

[0346] The method embodiments provided in this application have been described above. The apparatus embodiments provided in this application will be described below. It should be understood that the description of the apparatus embodiments corresponds to the description of the method embodiments. Therefore, any content not described in detail can be referred to the method embodiments above. For the sake of brevity, it will not be repeated here.

[0347] like Figure 7 As shown in the figure, this application embodiment also provides a data processing apparatus 700. The apparatus 700 includes the following units.

[0348] The first processing unit 710 is used to add architecture parameters to each feature combination in the first model to obtain the second model, wherein the first model is a model based on factorization machine (FM), and the architecture parameters are used to characterize the importance of the corresponding feature combination.

[0349] The second processing unit 720 is used to optimize and learn the architecture parameters in the second model to obtain the optimized architecture parameters.

[0350] The third processing unit 730 is used to obtain a third model after feature combination term reduction based on the first model or the second model, according to the optimized architecture parameters.

[0351] Optionally, the optimization learning performed by the second processing unit 720 on the architecture parameters results in the optimized architecture parameters exhibiting sparsity.

[0352] In this embodiment, the third processing unit 730 is used to, based on the first model or the second model, delete the feature combination items corresponding to the architecture parameters whose values ​​are less than a threshold in the optimized architecture parameters to obtain the third model.

[0353] Optionally, the second processing unit 720 performs optimization learning on the architecture parameters such that the value of the architecture parameter of at least one feature combination item is equal to zero after optimization.

[0354] For example, the third processing unit 730 is used to optimize the architecture parameters in the second model using a GRDA optimizer, which causes the value of the architecture parameter of at least one feature combination term to tend to zero during the optimization process.

[0355] Optionally, the second processing unit 720 is further configured to perform optimization learning on the model parameters in the second model, the optimization learning including scalarization processing on the model parameters in the second model.

[0356] For example, the second processing unit 720 is used to perform batch normalization (BN) processing on the model parameters in the second model.

[0357] Optionally, the second processing unit 720 is used to simultaneously optimize the architecture parameters and model parameters in the second model using the same training data to obtain optimized architecture parameters.

[0358] Optionally, the device 700 also includes a training unit 740 for training a third model.

[0359] Optionally, training unit 740 is used to train a third model to obtain a CTR prediction model or a CVR prediction model.

[0360] The device 700 can be integrated into terminal equipment, network equipment, or chips.

[0361] The device 700 can be deployed on the computing nodes of relevant devices.

[0362] like Figure 8 As shown, this application embodiment also provides an image processing apparatus 800. The apparatus 800 includes the following units.

[0363] The first processing unit 810 is used to input the data of the target object into the click-through rate (CTR) prediction model or the conversion rate (CVR) prediction model to obtain the prediction result of the target object.

[0364] The first processing unit 820 is used to determine the recommendation status of the target object based on the prediction result of the target object.

[0365] The CTR prediction model or the CVR prediction model is obtained through method 300 or 500 in the above embodiments.

[0366] The steps for training the third model include: using the training samples of the target object to train the third model to obtain the CTR prediction model or the CVR prediction model.

[0367] Optionally, the step of optimizing the architecture parameters includes: using the same training data from the training samples of the target object, simultaneously optimizing the architecture parameters and model parameters in the second model to obtain optimized architecture parameters.

[0368] The device 800 can be integrated into terminal equipment, network equipment, or chips.

[0369] The device 800 can be deployed on the computing nodes of relevant equipment.

[0370] like Figure 9As shown, this application embodiment also provides an image processing apparatus 900. The apparatus 900 includes a processor 910 coupled to a memory 920, the memory 920 for storing computer programs or instructions, and the processor 910 for executing the computer programs or instructions stored in the memory 920, causing the methods in the above method embodiments to be executed.

[0371] Optionally, such as Figure 9 As shown, the device 900 may also include a memory 920.

[0372] Optionally, such as Figure 9 As shown, the device 900 may also include a data interface 930, which is used for data transmission with the outside world.

[0373] Alternatively, as one approach, the device 900 is used to implement the method 300 in the above embodiments.

[0374] Alternatively, as another option, the device 900 is used to implement the method 500 in the above embodiments.

[0375] Alternatively, as another option, the device 900 is used to implement the method 600 in the above embodiments.

[0376] This application also provides a computer-readable medium storing program code for execution by a device, the program code including methods for performing the above embodiments.

[0377] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the methods described in the above embodiments.

[0378] This application also provides a chip, which includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface and executes the method described in the above embodiments.

[0379] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to perform the methods described in the above embodiments.

[0380] This application also provides an electronic device, which includes one or more of the devices 700, 800 or 900 in the above embodiments.

[0381] Figure 10A chip hardware structure provided in this application embodiment includes a neural network processor 1000. This chip can be disposed in any one or more of the following devices or systems:

[0382] like Figure 7 The device 700 shown, such as Figure 8 The device 800 shown, such as Figure 9 The device 900 shown is shown.

[0383] Methods 300, 500, or 600 in the above method embodiments can all be implemented as follows: Figure 10 This is achieved in the chip shown.

[0384] The neural network processor 1000 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core of the neural network processor 1000 is the arithmetic circuit 1003, and the controller 1004 controls the arithmetic circuit 1003 to obtain data from the memory (weight memory 1002 or input memory 1001) and perform calculations.

[0385] In some implementations, the arithmetic circuit 1003 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 1003 is a two-dimensional pulsating array. The arithmetic circuit 1003 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1003 is a general-purpose matrix processor.

[0386] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 1003 retrieves the corresponding data for matrix B from the weight memory 1002 and caches it in each PE of the arithmetic circuit 1003. The arithmetic circuit 1003 retrieves the data for matrix A from the input memory 1001 and performs matrix operations with matrix B. The partial or final result of the obtained matrix is ​​stored in the accumulator 1008.

[0387] The vector computation unit 1007 can further process the output of the arithmetic circuit 1003, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector computation unit 1007 can be used for network computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, local response normalization, etc.

[0388] In some implementations, the vector computation unit 1007 can store the processed output vector into a unified memory (also called a unified buffer) 1006. For example, the vector computation unit 1007 can apply a nonlinear function to the output of the arithmetic circuit 1003, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 1007 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as an activation input to the arithmetic circuit 1003, for example, for use in subsequent layers of a neural network.

[0389] Methods 300, 500, or 600 in the above method embodiments can be executed by 1003 or 1007.

[0390] The unified memory 1006 is used to store input data and output data.

[0391] The weight data is directly transferred from the external memory to the input memory 1001 and / or the unified memory 1006 through the direct memory access controller 1005 (DMAC), the weight data in the external memory is stored in the weight memory 1002, and the data in the unified memory 1006 is stored in the external memory.

[0392] The bus interface unit (BIU) 1010 is used to enable interaction between the main CPU, DMAC and instruction fetch memory 1009 via the bus.

[0393] The instruction fetch buffer 1009, which is connected to the controller 1004, is used to store the instructions used by the controller 1004.

[0394] The controller 1004 is used to call the instructions cached in the instruction memory 1009 to control the operation of the computing accelerator.

[0395] In this embodiment of the application, the data here may be image data to be processed.

[0396] Generally, the unified memory 1006, input memory 1001, weight memory 1002, and instruction fetch memory 1009 are all on-chip memories, while the external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0397] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0398] It should be noted that the various numerical designations such as first, second, third, or fourth used in this document are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0399] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0400] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0401] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0402] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0403] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0404] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: Universal Serial Bus flash disks (USB flash disks, UFDs) (UFDs can also be simply referred to as USB flash drives or USB flash drives), portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0405] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of data processing, characterized by, The method comprises: adding an architecture parameter to each feature combination item in a first model to obtain a second model, wherein the first model is a model based on a factorization machine (FM), and the architecture parameter is used to represent the importance of the corresponding feature combination item; simultaneously optimizing and learning the architecture parameter in the second model and a model parameter in the second model using the same training data to obtain an optimized architecture parameter, wherein the optimization and learning causes the optimized architecture parameter to present sparsity, the optimization and learning uses a generalized regular dual averaging (GRDA) optimizer to optimize the architecture parameter in the second model, and the GRDA optimizer causes the value of the architecture parameter of at least one feature combination item to tend to zero during the optimization process; pruning the feature combination item in the first model or the second model according to the optimized architecture parameter to obtain a third model, wherein the third model is used to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model, and the CTR estimation model or the CVR estimation model and data of a target object are used to determine whether to recommend the target object.

2. The method of claim 1, wherein, The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model.

3. The method according to claim 1 or 2, characterized in that, The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model.

5. The method of claim 4, wherein, The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model.

7. A method of data processing, characterized by, The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model to obtain a click-through rate (CTR) estimation model or a conversion rate (CVR) estimation model. The method further comprises training the third model According to the optimized architecture parameter, a third model is obtained by pruning a feature combination item in the first model or the second model; The third model is trained using the training sample of the target object to obtain the CTR estimation model or the CVR estimation model.

8. The method of claim 7, wherein, The third model is obtained by pruning the feature combination item corresponding to the architecture parameter with a value less than a threshold in the optimized architecture parameter. The third model is obtained by pruning the feature combination item corresponding to the architecture parameter with a value less than a threshold in the optimized architecture parameter.

9. The method according to claim 7 or 8, characterized in that, The optimization learning makes the value of the architecture parameter of at least one feature combination item equal to zero after the optimization is completed.

10. The method according to any one of claims 7 to 9, characterized in that, The optimization learning of the model parameter in the second model includes a quantization processing of the model parameter.

11. The method of claim 10, wherein, The optimization learning includes a batch normalization (BN) processing of the model parameter in the second model.

12. An apparatus for image processing, characterized by The computer readable medium stores program codes for execution by a device, and the program codes, when executed, cause the device to perform the method in any one of claims 1-11. The computer readable medium stores program codes for execution by a device, and the program codes, when executed, cause the device to perform the method in any one of claims 1-11. The device includes at least one processor and a data interface.

13. A computer-readable storage medium, characterized in that, The at least one processor is configured to call and run a computer program stored on a memory through the data interface, so that the chip performs the method in any one of claims 1-11.

14. A chip, characterized by ​ ​

Images