Automatic integrated architecture search system and method for click-through rate prediction model

Abstract

The present application proposes an automatic integrated architecture search system for a click-through rate prediction model, including a search space for a feature embedding module, which is used to search the dimension of the embedding vector of a feature and find a suitable embedding vector dimension for each feature; a search space for a feature interaction module, which is used to search for a sub-network of feature interaction according to a feature combination and simultaneously search for a feature interaction combination to obtain a feature interaction result; and a search space for a deep network module, which is used to incorporate the feature interaction result into an implicit high-order interaction search space, and then perform implicit high-order interaction in deep networks with different numbers of layers to obtain a click-through rate prediction value. The present application realizes the automatic integration of the design of the network of the click-through rate prediction model, obtains a model architecture with better performance from a complete search space, improves the integrity and expressive ability of the click-through rate prediction model, and at the same time greatly reduces the time, resources, and human and technical costs for designing a neural network, and improves the efficiency and effect of the design process.

Description

Technical Field

[0001] The present application relates to the technical field of click-through rate prediction model architecture, and in particular to an automatic integrated architecture search system and method for click-through rate prediction models. Background Art

[0002] The structure of a click-through rate prediction model has a crucial impact on its performance. However, in production practice, the model architecture needs to be manually designed for each different recommendation task or dataset, which consumes a lot of manpower and material resources for trial and error and verification. In addition, since click-through rate prediction models are usually composed of several modules, ordinary architecture search algorithms cannot search the entire space in an integrated manner.

[0003] Existing related technologies are mainly classified into two categories:

[0004] The first category of related technologies involves network architecture design for click-through rate prediction models. These technologies manually design embedding vector dimensions, feature interaction methods and combinations, and deep network modules tailored to specific tasks and datasets. Earlier linear models modeled click-through rate prediction as a linear classification problem. Later, factorization machines, using factorization to enable feature interaction, achieved significant progress. Later models, such as Wide & Deep, DeepFM, and PIN, combined deep networks with traditional low-level interactions to deeply explore relationships between features.

[0005] The second type of related technology is the application of neural network architecture search in click-through rate prediction models. This type of technology searches each module in the model separately by directly applying existing neural network architecture search algorithms. Existing work can be divided according to the modules targeted: feature embedding module, feature interaction module, and deep network module. For the feature embedding module, AutoEmb and AutoDim use differentiable architecture search and weight sharing technology to search the dimensions of the feature embedding vector respectively; for the feature interaction module, AutoFeature searches for sub-networks of feature interactions and searches for feature interaction combinations at the same time, while AutoFIS makes the search process of feature interaction combinations differentiable to improve search efficiency; for the deep network module, no work has yet proposed a specific search space and algorithm for search.

[0006] The existing methods for designing click-through rate prediction models mainly rely on expert experience, manual debugging and design, with low efficiency, high resource, manpower and technical costs, and are prone to introducing design biases and sub-optimal designs. The existing neural network architecture search technologies do not propose an integrated search framework for multiple modules of the entire model, which will cause the search results to fall into local optima. Moreover, if the relationships between multiple modules are not modeled, the expressive capabilities of each module will be inconsistent, affecting the overall performance. And there are significant differences in the structures of each module, and the existing search spaces and algorithms cannot be directly applied to the holistic search of the model, and cannot solve the problems of adaptability and consistency between modules in the design of click-through rate prediction models. Summary of the Invention

[0007] This application aims to solve at least one of the technical problems in the related art to some extent.

[0008] To this end, the first object of this application is to propose an automatic integrated architecture search system for click-through rate prediction models, which solves the problems that the existing network design methods for click-through rate prediction models rely on experts' design experience for target tasks, data and domains, as well as a large amount of tuning and trial-and-error, with low design efficiency and being prone to obtaining sub-optimal solutions. At the same time, it solves the problem that the search spaces and search algorithms of the existing neural network architecture search methods are only limited to a specific module, and there is no integrated search for the entire system, and the optimal solution cannot be obtained. For the first time, a complete model search space including all modules is designed, and an automatic integrated search algorithm is proposed, which can generate a better-quality model architecture, greatly improving the network construction efficiency of click-through rate prediction models, as well as the integrity and expressive capabilities of click-through rate prediction models. At the same time, it greatly reduces the time, resources and manpower and technical costs of designing neural networks, and improves the efficiency and effect of the design process.

[0009] The second object of this application is to propose an automatic integrated architecture search method for click-through rate prediction models.

[0010] The third object of this application is to propose a non-transitory computer-readable storage medium.

[0011] To achieve the above object, an automatic integrated architecture search system for a click-through rate prediction model according to an embodiment of the first aspect of the present application includes a search space for a feature embedding module, a search space for a feature interaction module, and a search space for a deep network module. Among them, the search space for the feature embedding module is used to search for the dimension of the embedding vector of the feature and find a suitable embedding vector dimension for each feature; the search space for the feature interaction module is used to search for a sub-network of feature interaction according to the feature combination and simultaneously search for the feature interaction combination to obtain a feature interaction result; the search space for the deep network module is used to incorporate the feature interaction result into the search space of implicit high-order interaction, and then perform implicit high-order interaction in deep networks of different layers to finally obtain a click-through rate prediction value.

[0012] Optionally, in an embodiment of the present application, it further includes a mapping space, and the mapping space is used to map feature combinations of different dimensions into a unified dimension space where interaction can be performed.

[0013] Optionally, in an embodiment of the present application, the search space of the feature interaction module is correlated with the selection of the feature embedding layer structure.

[0014] Optionally, in an embodiment of the present application, it further includes an architecture generator, and the architecture generator is used to consider the structure selection of all previous search spaces in the automatic integrated architecture search system for the click-through rate prediction model, generate the architecture of the current module, and then generate a complete integrated architecture of the click-through rate prediction model.

[0015] Optionally, in an embodiment of the present application, joint optimization and update are performed by reducing all sub-networks in the search space of the automatic integrated architecture search system for the click-through rate prediction model to a super network.

[0016] Optionally, in an embodiment of the present application, the training method used in the joint optimization and update is specifically:

[0017] In each round of training, first train the sub-network with the largest number of parameters as the teacher network, and then use the output of the teacher network to guide the training of the remaining sub-networks.

[0018] To achieve the above object, an automatic integrated architecture search method for a click-through rate prediction model according to an embodiment of the second aspect of the present application includes: searching for the dimension of the embedding vector of the feature and finding a suitable embedding vector dimension for each feature; searching for a sub-network of feature interaction according to the feature combination and simultaneously searching for the feature interaction combination to obtain a feature interaction result; incorporating the feature interaction result into the search space of implicit high-order interaction, and then performing implicit high-order interaction in deep networks of different layers to finally obtain a click-through rate prediction value.

[0019] To achieve the above object, a third aspect embodiment of the present application proposes a non-transitory computer-readable storage medium. When the instructions in the storage medium are executed by a processor, an automatic integrated architecture search method for a click-through rate prediction model can be executed.

[0020] The automatic integrated architecture search system for a click-through rate prediction model, the automatic integrated architecture search method for a click-through rate prediction model, and the non-transitory computer-readable storage medium according to the embodiments of the present application solve the problems that the existing network design method for click-through rate prediction models relies on experts' design experience for target tasks, data, and domains, as well as a large amount of tuning and trial-and-error, resulting in low design efficiency and being prone to obtaining suboptimal solutions. At the same time, it solves the problem that the search space and search algorithm design of the existing neural network architecture search methods are limited to a specific module, and there is no integrated search for the entire system, so the optimal solution cannot be obtained. For the first time, a complete model search space including all modules is designed, and an automatic integrated search algorithm is proposed, which can generate a better model architecture, greatly improving the network construction efficiency of the click-through rate prediction model, as well as the integrity and expression ability of the click-through rate prediction model. At the same time, it greatly reduces the time, resources, and human technical costs of designing neural networks, and improves the efficiency and effect of the design process.

[0021] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of the embodiments in conjunction with the drawings, where:

[0023] Figure 1 is a schematic structural diagram of an automatic integrated architecture search system for a click-through rate prediction model provided by Embodiment 1 of the present application;

[0024] Figure 2 is a system block diagram of an automatic integrated architecture search system for a click-through rate prediction model according to an embodiment of the present application;

[0025] Figure 3 is a flowchart of an automatic integrated architecture search method for a click-through rate prediction model provided by Embodiment 2 of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] Embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present application and should not be construed as limiting the present application.

[0027] The automatic integrated architecture search system and method for the click-through rate prediction model according to the embodiments of the present application will be described below with reference to the accompanying drawings.

[0028] Figure 1 It is a schematic structural diagram of an automatic integrated architecture search system for the click-through rate prediction model provided in Embodiment 1 of the present application.

[0029] As Figure 1 shown, the automatic integrated architecture search system for the click-through rate prediction model includes: a search space for the feature embedding module, a search space for the feature interaction module, and a search space for the deep network module. Among them,

[0030] The search space 10 for the feature embedding module is used to search for the dimension of the embedding vector of the feature and find a suitable embedding vector dimension for each feature;

[0031] The search space 20 for the feature interaction module is used to search for the sub-network of the feature interaction according to the feature combination and simultaneously search for the feature interaction combination to obtain the feature interaction result;

[0032] The search space 30 for the deep network module is used to incorporate the feature interaction result into the search space of the implicit high-order interaction, and then perform implicit high-order interaction in the deep network with different numbers of layers, and finally obtain the click-through rate prediction value.

[0033] The automatic integrated architecture search system for the click-through rate prediction model according to the embodiments of the present application includes a search space for a feature embedding module, a search space for a feature interaction module, and a search space for a deep network module. Among them, the search space for the feature embedding module is used to search for the dimension of the embedding vector of the feature and find a suitable embedding vector dimension for each feature; the search space for the feature interaction module is used to search for a sub-network of feature interaction according to the feature combination and simultaneously search for the feature interaction combination to obtain a feature interaction result; the search space for the deep network module is used to incorporate the feature interaction result into the search space of implicit high-order interaction, and then perform implicit high-order interaction in deep networks with different numbers of layers to finally obtain a click-through rate prediction value. Thus, it can solve the problems that the existing network design method for the click-through rate prediction model relies on experts' design experience for the target task, data, and domain, as well as a large amount of tuning and trial and error, resulting in low design process efficiency and being prone to obtaining sub-optimal solutions. At the same time, it can solve the problem that the search space and search algorithm design of the existing neural network architecture search method are limited to a specific module, and there is no integrated search for the entire system, so the optimal solution cannot be obtained. For the first time, a complete model search space including all modules is designed, and an automatic integrated search algorithm is proposed, which can generate a better model architecture, greatly improving the efficiency of building the click-through rate prediction model network, as well as the integrity and expressive ability of the click-through rate prediction model. At the same time, it greatly reduces the time, resources, and human and technical costs of designing the neural network, and improves the efficiency and effect of the design process.

[0034] This application mainly includes the integrated design of the search space of the click-through rate prediction model and the corresponding search algorithm design. For the design of the search space of the click-through rate prediction model network, this application proposes a self-consistent and unified representation network search space among various modules, which not only systematically summarizes and includes all the previously manually designed network architectures, but also includes the search spaces separately designed for searching a certain module in previous work. Three modules and their connection methods are designed, and an integrated context design is made for the connections between different module spaces, so that the connections between modules are adapted to the two modules.

[0035] Further, in the embodiments of the present application, it further includes a mapping space, and the mapping space is used to map feature combinations of different dimensions into a unified dimension space where interaction can be performed.

[0036] In the search for the dimension of the embedding layer vector, in addition to including the architecture space proposed in previous work, this application newly proposes a new space designed for unifying the interaction dimension, in which different feature combinations can be mapped to different interaction spaces.

[0037] Further, in the embodiments of the present application, the search space of the feature interaction module is correlated with the selection of the feature embedding layer structure.

[0038] In the interaction layer, the present application simultaneously includes the search for interaction functions and the search for interaction combinations, and is related to the selection of the embedding layer structure.

[0039] Further, in the embodiment of the present application, there is also an architecture generator, which is used in the automatic integrated architecture search system for the click-through rate prediction model to generate the architecture of the current module by considering the structure selection of all previous search spaces before each search space, and then generate the complete integrated architecture of the click-through rate prediction model.

[0040] The present application proposes to use a hypernetwork training method based on knowledge distillation technology to construct an integrated model architecture generator that models the relationships between modules, making the search process more efficient.

[0041] Further, in the embodiment of the present application, joint optimization and update are performed by reducing all sub-networks in the search space of the automatic integrated architecture search system for the click-through rate prediction model to a single hypernetwork.

[0042] By reducing all sub-networks in the space to a single hypernetwork, all the parameters of the space sub-networks can be jointly optimized and updated, thus greatly improving the efficiency of the search algorithm.

[0043] Further, in the embodiment of the present application, the training method used in the joint optimization and update is specifically:

[0044] In each round of training, first train the sub-network with the largest number of parameters as the teacher network, and then use the output of this teacher network to guide the training of the remaining sub-networks.

[0045] For the hypernetwork of the multi-module set, the sub-networks obtained in each training usually affect each other, and the training parameters are often not unified. To solve this problem, the present application proposes to use a method based on knowledge distillation. In each round of training, first train the sub-network with the largest number of parameters, and then use the output of this network to guide the training of the remaining sub-networks. By this method, the overall update direction of the hypernetwork is consistent in each round of training, and the performance evaluation of the sub-networks therein is more stable.

[0046] Figure 2 This is the system block diagram of the automatic integrated architecture search system for the click-through rate prediction model in the embodiment of the present application.

[0047] As Figure 2 shown, on the left is the architecture generator, which is used to explicitly model the structural relationships between modules. The architecture distribution generation of each module depends on the architecture selection of the previous module; on the right is the search space of the click-through rate prediction model, and the figure shows the architecture selection of a pair of feature interactions therein.

[0048] Figure 3 The flowchart of an automatic integrated architecture search method for a click-through rate prediction model provided in the second embodiment of the present application.

[0049] As Figure 3 shown, the automatic integrated architecture search method for the click-through rate prediction model includes:

[0050] Step 101: Search for the dimension of the embedding vector of the feature to find a suitable embedding vector dimension for each feature.

[0051] Step 102: Search for the sub-network of feature interaction according to the feature combination and simultaneously search for the feature interaction combination to obtain the feature interaction result.

[0052] Step 103: Incorporate the feature interaction result into the search space of implicit high-order interaction, and then perform implicit high-order interaction in deep networks with different numbers of layers to finally obtain the click-through rate prediction value.

[0053] The automatic integrated architecture search method for the click-through rate prediction model in the embodiment of the present application includes searching for the dimension of the embedding vector of the feature to find a suitable embedding vector dimension for each feature; searching for the sub-network of feature interaction according to the feature combination and simultaneously searching for the feature interaction combination to obtain the feature interaction result; incorporating the feature interaction result into the search space of implicit high-order interaction, and then performing implicit high-order interaction in deep networks with different numbers of layers to finally obtain the click-through rate prediction value. Thus, it can solve the problems that the existing network design method for the click-through rate prediction model relies on the design experience of experts for the target task, data, and domain, as well as a large amount of tuning and trial-and-error, resulting in low efficiency in the design process and being prone to obtaining sub-optimal solutions. At the same time, it can solve the problem that the search space and search algorithm design of the existing neural network architecture search method are limited to a specific module and do not perform an integrated search for the entire system, so the optimal solution cannot be obtained. For the first time, a complete model search space including all modules is designed, and an automatic integrated search algorithm is proposed, which can generate a better-quality model architecture, greatly improving the network construction efficiency of the click-through rate prediction model, as well as the integrity and expressive ability of the click-through rate prediction model. At the same time, it greatly reduces the time, resources, and human technical costs of designing the neural network and improves the efficiency and effect of the design process.

[0054] To implement the above embodiment, the present application also proposes a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the automatic integrated architecture search method for the click-through rate prediction model in the above embodiment.

[0055] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "examples", "specific examples", or "some examples", etc., mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0056] In addition, the terms "first" and "second" are used only for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of this application, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0057] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment, or portion of code including one or more executable instructions for implementing a customized logic function or process, and the scope of the preferred embodiments of this application includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in the reverse order according to the functions involved, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of this application belong.

[0058] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definitional sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in conjunction with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion having one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then storing it in a computer memory.

[0059] It should be understood that various parts of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0060] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of implementing the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0061] In addition, each functional unit in various embodiments of the present application may be integrated into one processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0062] The above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. An automatic integrated architecture search system for click-through rate prediction models, characterized in that, Including a feature embedding module search space, a feature interaction module search space, and a deep network module search space. Among them, the feature embedding module search space is used to search for the dimension of the embedding vector of the feature and find a suitable embedding vector dimension for each feature; the feature interaction module search space is used to search for the sub-network of feature interaction according to the feature combination and simultaneously search for the feature interaction combination to obtain the feature interaction result; the deep network module search space is used to incorporate the feature interaction result into the search space of implicit high-order interaction, and then perform implicit high-order interaction in deep networks with different numbers of layers, and finally obtain the click-through rate prediction value.

2. The system according to claim 1, wherein It further includes a mapping space, which is used to map feature combinations of different dimensions into a unified dimensional space where interaction can be performed.

3. The system according to claim 1, wherein The feature interaction module search space is correlated with the selection of the feature embedding layer structure.

4. The system according to claim 1, wherein It further includes an architecture generator, which is used to generate the architecture of the current module in the automatic integrated architecture search system for the click-through rate prediction model, considering the structure selection of all previous search spaces before each search space, and then generate the complete integrated architecture of the click-through rate prediction model.

5. The system according to claim 1, wherein Joint optimization and update are performed by reducing all sub-networks in the search space of the automatic integrated architecture search system for the click-through rate prediction model to a super-network.

6. The system according to claim 5, wherein The training method used in the joint optimization and update is specifically as follows: In each round of training, first train the sub-network with the largest number of parameters as the teacher network, and then use the output of this teacher network to guide the training of the remaining sub-networks.

7. An automatic integrated architecture search method for a click-through rate prediction model, characterized in that, It includes the following steps: Search for the dimension of the embedding vector of the feature and find a suitable embedding vector dimension for each feature; Search for the sub-network of feature interaction according to the feature combination and simultaneously search for the feature interaction combination to obtain the feature interaction result; Incorporate the feature interaction result into the search space of implicit high-order interaction, and then perform implicit high-order interaction in deep networks with different numbers of layers, and finally obtain the click-through rate prediction value.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to claim 7.

Images