Method and device for hyper-parameter evaluation, equipment and storage medium

Abstract

The embodiment of the invention relates to a hyper-parameter evaluation method and device, equipment and a storage medium. The method includes the steps that multiple sets of hyper-parameters for a first machine learning model are obtained, and at least two sets of hyper-parameters in the multiple sets of hyper-parameters are heterogeneous hyper-parameters; on the basis of the multiple groups of hyper-parameters, global feature representations associated with the hyper-parameters are generated for each group of hyper-parameters in the multiple groups of hyper-parameters to obtain multiple global feature representations, and all the global feature representations in the multiple global feature representations are isomorphic global feature representations; based on the plurality of global feature representations, utilizing a second machine learning model to determine similarity among a plurality of groups of hyper-parameters associated with the plurality of global feature representations; and determining an evaluation result of at least one group of hyper-parameters in the plurality of groups of hyper-parameters based on the determined similarity.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computer technology, and more specifically, to a method, apparatus, device, and storage medium for hyperparameter evaluation. Background Technology

[0002] Hyperparameters play a crucial role in machine learning. They are parameters set before model training to control the training process and influence the model's performance and generalization ability. For example, in neural networks, the number of layers and the number of neurons in each layer are hyperparameters. Choosing appropriate hyperparameters is key to ensuring the model can learn data features and make accurate predictions. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for hyperparameter evaluation is provided. The method includes: acquiring multiple sets of hyperparameters for a first machine learning model, wherein at least two sets of hyperparameters are heterogeneous; generating associated global feature representations for each set of hyperparameters in the multiple sets of hyperparameters based on the multiple sets of hyperparameters, to obtain multiple global feature representations, wherein each global feature representation in the multiple global feature representations is isomorphic; determining, using a second machine learning model, a similarity between the multiple sets of hyperparameters associated with the multiple global feature representations based on the multiple global feature representations; and determining an evaluation result for at least one set of hyperparameters in the multiple sets of hyperparameters based on the determined similarity.

[0004] In a second aspect of this disclosure, an apparatus for hyperparameter evaluation is provided. The apparatus includes: a hyperparameter acquisition module configured to acquire multiple sets of hyperparameters for a first machine learning model, wherein at least two sets of hyperparameters are heterogeneous; a global feature representation generation module configured to generate associated global feature representations for each set of hyperparameters based on the multiple sets of hyperparameters, to obtain multiple global feature representations, wherein each global feature representation is isomorphic; a similarity determination module configured to determine, using a second machine learning model, the similarity between the multiple sets of hyperparameters associated with the multiple global feature representations; and an evaluation result determination module configured to determine an evaluation result for at least one set of hyperparameters based on the determined similarity.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method according to a first aspect of this disclosure.

[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 A flowchart illustrating an example process of a method for hyperparameter evaluation according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram illustrating an example of generating a global feature representation for a set of hyperparameters according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic structural block diagram of an apparatus for model training according to some embodiments of the present disclosure is shown; and

[0014] Figure 5 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0017] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0018] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0019] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0020] As briefly described above, selecting appropriate hyperparameters is crucial to ensuring that the model can learn data features and make accurate predictions. Bayesian optimization algorithms, with their advantages in hyperparameter selection efficiency, global optimization capabilities, ability to handle complex objective functions, and adaptability, have become one of the advanced methods in the field of hyperparameter optimization.

[0021] As an example, based on the Bayesian optimization algorithm, multiple sets of hyperparameters can be collected from a search space containing multiple candidate hyperparameters, and the performance metrics of these sets can be evaluated using a Gaussian process model to select the optimal combination of hyperparameters. However, it's important to note that the Gaussian process model is only suitable for evaluating homogeneous sets of hyperparameters—that is, sets of hyperparameters where the dimension (or the number of hyperparameters in a set) is the same as the semantic (or physical meaning) of each dimension (or individual hyperparameter in a set). For sets of hyperparameters with different dimensions or semantics (or heterogeneity), the Gaussian process model struggles to evaluate them. This is because the Gaussian process model determines performance metrics by calculating the similarity between two sets of hyperparameters, a process that cannot be performed on heterogeneous sets of hyperparameters.

[0022] Due to the aforementioned problems, traditional Bayesian optimization algorithms are typically only applicable to simple search spaces where hyperparameters are independent, and are difficult to apply to complex search spaces where hyperparameters are dependent. This is because sampling such complex search spaces often results in heterogeneous sets of hyperparameters.

[0023] As an example, hyperparameters can include the learning rate and batch size. In neural networks, there may be a dependency between the learning rate and batch size; for example, a larger batch size may require a smaller learning rate to maintain a stable learning process. This inherent connection leads to dependencies between hyperparameters in the search space.

[0024] As an example, a tree search space is a typical complex search space where hyperparameters have dependencies. Currently, multiple sets of hyperparameters can be obtained by sampling each branch in the tree search space. In this process, the same set of hyperparameters in multiple sets is obtained based on sampling the same branch. However, in a tree search space, different branches (e.g., branch A and branch B) may have different numbers of nodes, and the number of hyperparameters contained in different nodes and the semantics of each hyperparameter may also be different.

[0025] Suppose we obtain a set of hyperparameters C by sampling branch A and a set of hyperparameters D by sampling branch B. The number of hyperparameters in set C may differ from the number of hyperparameters in set D. Even if they have the same number of hyperparameters, they may contain hyperparameters with different semantics (e.g., two different decision outcomes under a single decision variable). In either case, set C and set D are heterogeneous sets of hyperparameters, making them unsuitable for handling by the Gaussian process model in traditional Bayesian optimization algorithms.

[0026] There are currently two solutions to this problem.

[0027] In the first solution, predefined content (e.g., invalid values ​​that can be distinguished from valid values ​​of the hyperparameters) can be used to fill in each group of hyperparameters, thereby aligning the hyperparameter groups. Then, the Gaussian process model described above is used for processing. However, this approach introduces invalid content into one or more groups of hyperparameters, which reduces processing efficiency. Furthermore, introducing a large amount of invalid content can cause a series of high-dimensional problems.

[0028] In the second solution, one or more sets of hyperparameters sampled from the same branch are treated as a collection. Each collection is then processed independently using a Gaussian process model. However, this solution requires setting up multiple surrogate models, which leads to a significant increase in the number of observation points during hyperparameter optimization, resulting in a substantial decrease in overall efficiency. Furthermore, the second solution ignores potential relationships between different collections E2.

[0029] Based on the above, it can be seen that traditional Bayesian optimization algorithms are difficult to apply to complex search spaces where hyperparameters are dependent on each other.

[0030] In view of this, embodiments of the present disclosure provide a scheme for hyperparameter evaluation. The proposed scheme includes: obtaining multiple sets of hyperparameters for a first machine learning model, wherein at least two sets of hyperparameters are heterogeneous. Then, based on the multiple sets of hyperparameters, generating associated global feature representations for each set of hyperparameters to obtain multiple global feature representations, wherein each global feature representation is isomorphic. Subsequently, based on the multiple global feature representations, using a second machine learning model, determining the similarity between the multiple sets of hyperparameters associated with the multiple global feature representations. Then, based on the determined similarity, determining an evaluation result for at least one set of hyperparameters among the multiple sets of hyperparameters.

[0031] As will be more clearly understood from the following description, embodiments of this disclosure can transform heterogeneous sets of hyperparameters into homogeneous global feature representations when faced with multiple sets of hyperparameters. As an example, each global feature representation may contain three parts: the first part indicates the dependencies between hyperparameters in its associated set; the second part indicates the structural information of each hyperparameter in its associated set; and the third part indicates the value of each hyperparameter in its associated set. In this way, all global feature representations have the same three dimensions and the same semantics (dependencies, structural information, and values) for each dimension. Such global feature representations can be processed by a second machine learning model (such as a Gaussian process model) to determine the evaluation results of each set of hyperparameters.

[0032] In this way, embodiments of the present disclosure can process heterogeneous sets of hyperparameters into a form that can be processed by Gaussian process models. This means that embodiments of the present disclosure can extend Bayesian optimization algorithms to handle complex search spaces where hyperparameters have dependencies on each other. This helps to accelerate the development process of machine learning model applications, thereby improving productivity.

[0033] The following will further describe in detail various example implementations of this scheme with reference to the accompanying drawings.

[0034] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. (Refer to...) Figure 1 Example environment 100 may include electronic device 120.

[0035] In this example environment 100, electronic device 120 can determine a target search space 110 in response to a user's instruction. After determining the target search space 110, electronic device 120 can sample the target search space 110 to obtain multiple sets of hyperparameters 130 for a first machine learning model. As an example, the multiple sets of hyperparameters 130 may include a set of hyperparameters 131, a set of hyperparameters 132, and a set of hyperparameters 133. Then, electronic device 120 can generate multiple global feature representations 140 associated with the multiple sets of hyperparameters based on the multiple sets of hyperparameters 130. Furthermore, electronic device 120 can use the multiple global feature representations 140 to determine the evaluation results of at least one set of hyperparameters using a second machine learning model. It should be noted that, depending on actual needs, the multiple sets of hyperparameters 130 may include more sets of hyperparameters, and the multiple sets of hyperparameters are not limited to a specific set. Figure 1 The form shown.

[0036] In some embodiments, a user can interact with electronic device 120 through a terminal device to achieve the above-described process. The terminal device can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal device 110 can also support any type of user-facing interface (such as "wearable" circuitry).

[0037] In some embodiments, electronic device 120 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Electronic device 120 may include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in a cloud environment, etc.

[0038] A communication connection can be established between the electronic device 120 and the terminal device. This communication connection can be established via wired or wireless means. The communication connection may include, but is not limited to, Bluetooth connections, mobile network connections, Universal Serial Bus connections, and Wi-Fi connections; the embodiments of this disclosure are not limited in this respect. In the embodiments of this disclosure, the electronic device 120 and the terminal device can achieve signaling interaction through the communication connection between them.

[0039] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0040] Figure 2 A flowchart illustrating an example process 200 for a method of hyperparameter evaluation according to some embodiments of the present disclosure is shown. Process 200 may be implemented at an electronic device 120.

[0041] Combined with reference Figure 1 and Figure 2 In box 210, electronic device 120 acquires multiple sets of hyperparameters 130 for a first machine learning model, wherein at least two sets of hyperparameters 130 are heterogeneous hyperparameters.

[0042] In some specific embodiments, the first machine learning model can perform tasks with classification functions, such as image recognition. These machine learning models typically rely on large amounts of data and complex algorithms to accurately identify and classify different images. Hyperparameters, on the other hand, are parameters that need to be pre-set during the training process of such machine learning models. They are not learned directly from the training data, but they have a crucial impact on the performance and effectiveness of the machine learning model.

[0043] Specifically, hyperparameters may include the learning rate, regularization coefficient, and batch size of the first machine learning model, which together determine the learning speed and generalization ability of the first machine learning model during training. Setting multiple sets of hyperparameters 130 is essentially exploring how different combinations of these hyperparameters affect the final performance of the first machine learning model. It should be noted that, depending on actual needs, hyperparameters can include more components, and the first machine learning model can undertake more tasks; the various embodiments of this disclosure do not impose limitations in this regard.

[0044] Electronic device 120 can select the optimal set of hyperparameters for the first machine learning model by conducting a detailed evaluation of multiple sets of hyperparameters 130. The ultimate goal of this process is to find the combination of hyperparameters that enables the first machine learning model to achieve optimal performance.

[0045] Figure 3 A schematic diagram of example 300 for generating a global feature representation for a set of hyperparameters according to some embodiments of the present disclosure is shown. The following is in conjunction with... Figure 3 Example procedure 200 is explained.

[0046] In some embodiments, the electronic device 120 determines a target search space 110, which includes at least a tree search space 310. The electronic device 120 then samples each branch of the tree search space 310 to obtain multiple sets of hyperparameters 130, wherein the same set of hyperparameters is determined based on the sampling results of the same branch.

[0047] In some embodiments, the search space can be viewed as a vast exploration region containing various possible hyperparameter configurations for the first machine learning model. The target search space 110 includes at least a tree search space 310, which is a complex search space simulating a tree structure with multiple levels of branches, and each branch may have different hyperparameter configurations.

[0048] After determining the target search space 110, the electronic device 120 further performs detailed sampling (e.g., full sampling) on ​​each branch of the tree search space 310. The purpose of this step is to extract representative hyperparameter configurations (e.g., values) from each branch to form multiple sets of hyperparameters 130. It is noteworthy that the same set of hyperparameters is determined based on sampling results from the same branch in the tree search space 310, ensuring that there are no dependencies between different sets of hyperparameters. By exploring this complex tree search space 310, the electronic device 120 can more comprehensively search for possible combinations of hyperparameters for the first machine learning model.

[0049] In some embodiments, the electronic device 120 samples each branch of the tree search space 310 based on a predetermined sampling function. As an example, the predetermined sampling function includes an expected improvement function and / or a Thompson sampling function.

[0050] In some embodiments, a predetermined sampling function can instruct how to efficiently select and extract information from a complex tree-like search space 310 during the sampling process. As an example, in the tree-like search space 310, the electronic device 120 can evaluate the potential value of each branch based on an expected improvement function, i.e., the extent to which the performance of the first machine learning model might improve if that branch is sampled. As an example, in the tree-like search space 310, the electronic device 120 assigns a probability to each branch based on a Thompson sampling function, reflecting the likelihood that the branch is the optimal choice. During sampling, the electronic device 120 randomly selects a branch to explore based on these probabilities. These probabilities are continuously updated over time and with the collection of more data to reflect the true performance of each branch.

[0051] In some embodiments, the electronic device 120 may choose to use the expectation boosting function, the Thompson sampling function, or a combination of both, depending on the specific circumstances. Such a sampling strategy helps to improve sampling efficiency.

[0052] It should be noted that, depending on actual needs, the predefined sampling function can include more functions, and the combination of these functions can include more forms.

[0053] In some embodiments, the electronic device 120 samples a first branch in the tree search space 310 to obtain a first set of hyperparameters 320. Then, the electronic device 120 samples a second branch in the tree search space 310 to obtain a second set of hyperparameters heterogeneous to the first set of hyperparameters 320. As an example, the first set of hyperparameters 320 can be any of a set of hyperparameters 131, a set of hyperparameters 132, and a set of hyperparameters 133, and the second set of hyperparameters can be a set of hyperparameters 131, a set of hyperparameters 132, and a set of hyperparameters 133 that is different from the first set of hyperparameters 320.

[0054] In some embodiments, the first branch can be any branch in the tree search space 310, and the second branch is a branch in the tree search space 310 that is different from the first branch. In the tree search space 310, different branches may represent different execution strategies, constraints, or parameter spaces, and therefore the combinations of hyperparameters sampled from different branches will also differ. Specifically, "heterogeneity" can be reflected in at least two aspects.

[0055] Dimensional heterogeneity: The two sets of hyperparameters may have different dimensions, meaning that the number of hyperparameters contained in each set may differ. For example, a set of hyperparameters 131 may contain six hyperparameters, while a set of hyperparameters 132 may contain five hyperparameters.

[0056] Semantic heterogeneity: Even if two sets of hyperparameters have the same dimensions, the semantic or physical meaning of each dimension (i.e., each hyperparameter) may be different. This means that hyperparameters at the same position in two sets of hyperparameters represent completely different concepts. For example, both a set of hyperparameters 131 and a set of hyperparameters 133 contain six hyperparameters, but the last hyperparameter in set 131 may represent the learning rate, while in set 133, hyperparameters at the same position may represent the regularization strength.

[0057] As an example, hyperparameters with different semantics can represent two different decision outcomes under a certain decision variable. For instance, a decision variable might be the choice of which type of activation function to use, while two heterogeneous hyperparameters could represent two different choices of activation functions.

[0058] By sampling different branches in the tree search space 310, the electronic device 120 can obtain multiple sets of heterogeneous hyperparameters, which reflect the characteristics of different regions in the target search space 110.

[0059] In some embodiments, the first set of hyperparameters 320 and the second set of hyperparameters heterogeneous with the first set of hyperparameters 320 satisfy at least one of the following relationships: at least one hyperparameter in the first set of hyperparameters 320 has a different semantic from at least one hyperparameter in the second set of hyperparameters; or the number of hyperparameters in the first set of hyperparameters 320 is different from the number of hyperparameters in the second set of hyperparameters.

[0060] As an example, suppose the tree search space 310 includes two branches, and the first set of hyperparameters 320 obtained by sampling one of the branches can be represented as (p1, p2, p3, p4, p5), where p1, p2, p3, p4, and p5 are the hyperparameters in the first set of hyperparameters 320. The second set of hyperparameters obtained by sampling the other branch can be represented as (p1, p5, p6), where p1, p5, and p6 are the hyperparameters in the second set of hyperparameters. The difference between the two sets of hyperparameters is evident from the above. Specifically, the first set of hyperparameters 320 has five hyperparameters, while the second set has only three. Furthermore, although both the first set of hyperparameters 320 and the second set of hyperparameters have one hyperparameter p1 and one hyperparameter p5, the remaining hyperparameters are semantically completely different. To handle these two heterogeneous sets of hyperparameters, embodiments of this disclosure can convert them into two isomorphic global feature representations using box 220.

[0061] In box 220, electronic device 120 generates associated global feature representations for each set of hyperparameters 130 based on multiple sets of hyperparameters 130 to obtain multiple global feature representations 140, wherein each global feature representation in the multiple global feature representations 140 is an isomorphic global feature representation.

[0062] In contrast to "heterogeneity", isomorphism here is also reflected in at least two aspects.

[0063] Dimensional isomorphism: Multiple global feature representations have the same dimension (140). For example, two global feature representations contain the same number of feature representations. For instance, as described below, multiple global feature representations can each contain three feature representations.

[0064] Semantic isomorphism: In multiple global feature representations 140, the semantic or physical meaning of each dimension is the same in any two global feature representations. For example, the semantics of the feature representations contained in two global feature representations are the same. For example, as described below, the feature representations contained in multiple global feature representations can all be dependency information feature representations, value feature representations, and structural feature representations.

[0065] In some embodiments, for any one of the multiple sets of hyperparameters 130, the electronic device 120 may first embed target information into each of the hyperparameters in that set, and then use the encoder to determine the corresponding global feature representation based on the set of hyperparameters with embedded target information. As an example, the global feature representation may include the dependencies, values, and structures of each hyperparameter in the set of hyperparameters, which will be described in detail below, and will not be repeated here.

[0066] In some embodiments, the electronic device 120 acquires a first set of hyperparameters 320 from a plurality of sets of hyperparameters 130. Then, the electronic device 120 embeds target information into the first hyperparameters in the first set of hyperparameters 320 to generate a first hyperparameter feature representation. Subsequently, based on the first hyperparameter feature representation, the electronic device 120 uses the encoder 350 to generate a first global feature representation 360 associated with the first set of hyperparameters 320.

[0067] As an example, the first set of hyperparameters 320 includes multiple hyperparameters p1-p5, and the first hyperparameter can be any one of the multiple hyperparameters p1-p5 (for example, the first hyperparameter can be hyperparameter p1). It should be noted that, depending on actual needs, the first set of hyperparameters 320 may include more hyperparameters, and the embodiments of this disclosure do not limit this.

[0068] In some embodiments, the target information embedded in the first hyperparameter (e.g., if the first hyperparameter is hyperparameter p1, then the target information embedded in the first hyperparameter is target information 331 among multiple target information 331-335) can be any appropriate auxiliary information related to the first hyperparameter, such as the correlation between the first hyperparameter and other hyperparameters. The process of embedding the target information is essentially combining this information with the first hyperparameter to generate a richer feature representation, namely, the first hyperparameter feature representation. This feature representation not only includes the value of the first hyperparameter itself but also incorporates other related information, thereby providing a more comprehensive description of the characteristics of the first hyperparameter.

[0069] In some embodiments, the electronic device 120 can embed target information for each hyperparameter in the first set of hyperparameters 320 to obtain a hyperparameter feature representation for each hyperparameter. For example, the electronic device 120 embeds target information 331 for hyperparameter p1 to obtain a hyperparameter feature representation 341 for hyperparameter 331; embeds target information 332 for hyperparameter p2 to obtain a hyperparameter feature representation 342 for hyperparameter 332; embeds target information 333 for hyperparameter p3 to obtain a hyperparameter feature representation 343 for hyperparameter 333; embeds target information 334 for hyperparameter p4 to obtain a hyperparameter feature representation 344 for hyperparameter 334; and embeds target information 335 for hyperparameter p5 to obtain a hyperparameter feature representation 345 for hyperparameter 335.

[0070] Then, the electronic device 120 can provide the hyperparameter feature representation of each hyperparameter in the first set of hyperparameters 320 to the encoder 350. The encoder 350 can understand the characteristics of each hyperparameter based on the hyperparameter feature representation of each hyperparameter in the first set of hyperparameters 320, and then generate a first global feature representation 360 associated with the first set of hyperparameters 320 based on the characteristics of each hyperparameter.

[0071] In some embodiments, the target information may include at least one of first information, second information, third information, and fourth information. As an example, the first information (e.g., information 3311) indicates the parent node of the node where the first hyperparameter is located in the tree search space 310; the second information (e.g., information 3312) indicates the name of the first hyperparameter; the third information (e.g., information 3313) indicates the value of the first hyperparameter; and the fourth information (e.g., information 3314) indicates the dimension information of the first hyperparameter.

[0072] In some embodiments, in the tree search space 310, each hyperparameter resides on a specific node, and the information of the parent node helps to represent the dependency relationship between the first hyperparameter and other hyperparameters. The name of a hyperparameter can reflect its function or scope of action, and the name helps to represent the semantics of the hyperparameter. The specific value of the hyperparameter directly affects the performance and behavior of the first machine learning model.

[0073] In some embodiments, certain hyperparameters may be multidimensional. For example, in a neural network, the number of units in a hidden layer may vary depending on the number of network layers. Dimensional information helps to accurately describe the structure and properties of such hyperparameters.

[0074] In some embodiments, the electronic device 120 can vectorize the above content to embed it into the first hyperparameter. As an example, the target information can simultaneously indicate multiple items within the above content; in this way, the electronic device 120 can provide a more comprehensive feature representation of the first hyperparameter.

[0075] In some embodiments, the electronic device 120 combines the above-mentioned elements to form a comprehensive hyperparameter feature representation. This representation not only considers the values ​​of the hyperparameters but also incorporates key information such as their position, name, and dimension in the tree search space 310, thereby providing the encoder 350 with richer context and more accurate guidance.

[0076] It should be noted that, depending on the needs, the target information can also indicate more content to meet the relevant requirements in specific scenarios.

[0077] As an example, the first hyperparameter feature representation It can be expressed by formula (1):

[0078]

[0079] in, This represents the j-th set of hyperparameters obtained by sampling the i-th branch in the tree search space 310. Denotes the j-th group of hyperparameters The k-th hyperparameter in the list represents the parent node of the node containing the hyperparameter. The parent node of the top-level hyperparameter is the virtual root. express Name embedding, express The name of the parent node of the current node is embedded. express Embedding of dimensional information, express Value embedding, di This represents the total number of hyperparameters in the set of hyperparameters obtained by sampling the i-th branch.

[0080] As described above, the electronic device 120 can embed corresponding target information into each hyperparameter in the first set of hyperparameters 320. This results in a set of hyperparameter feature representations containing the hyperparameter feature representations of each hyperparameter in the first set of hyperparameters 320. As an example, a set of hyperparameter feature representations This can be expressed by formula (2):

[0081]

[0082] In some embodiments, the first set of hyperparameters 320 can be any one of the multiple sets of hyperparameters 130. As an example, the electronic device 120 can perform the above-described steps of embedding target information on each of the multiple sets of hyperparameters 130. In this way, the electronic device 120 can convert each hyperparameter in each set of hyperparameters into a vector representation that includes semantic information and dependencies, etc. (the hyperparameter feature representations of the same set of hyperparameters constitute a set of hyperparameter feature representations).

[0083] In some embodiments, the electronic device 120 embeds target information for a first hyperparameter in a first set of hyperparameters 320 based on a predetermined neural network 370, wherein the predetermined neural network 370 includes an embedding layer and a fully connected layer connected in sequence.

[0084] As an example, the electronic device 120 can encode the names of all hyperparameters in the target search space 110 sequentially, starting from 0. This encoding method assigns a unique numerical identifier to each hyperparameter name, facilitating processing by the pre-defined neural network 370. The electronic device 120 can then embed the numerical identifier into the corresponding hyperparameter using an embedding layer (e.g., the nn.Embedding layer in the PyTorch deep learning framework, commonly used in Python) within the pre-defined neural network 370. For the hyperparameter values ​​and dimensionality information, the electronic device 120 can use a single fully connected layer (e.g., nn.Linear) for embedding.

[0085] As an example, the predefined neural network 370 can be configured as multiple embedding modules, each of which is configured to embed information (such as hyperparameter dependency information, values, names, and dimension information) into the hyperparameters based on the input data (e.g., target information).

[0086] As an example, all embedding modules of the predefined neural network 370 have the same dimension (e.g., 64). This means that regardless of the original dimension of the input data, it will be converted into a 64-dimensional vector representation after being processed by the embedding modules.

[0087] It should be noted that the number of embedded modules in the predetermined neural network 370, as well as the input and output dimensions of the embedded modules, can be determined according to actual needs. In addition, the predetermined neural network 370 may include more layers, and the embodiments disclosed herein do not limit this.

[0088] As an example, after performing the above-described steps of embedding target information on the first set of hyperparameters 320 to obtain the set of hyperparameter feature representations of the first set of hyperparameters 320, the electronic device 120 provides the set of hyperparameter feature representations of the first set of hyperparameters 320 to the encoder 350, thereby generating the first global feature representation 360 by means of the encoder 350.

[0089] After embedding the hyperparameters, since the dimensions and semantics of different hyperparameter feature representation sets may differ, the electronic device 120 can address this issue using an encoder 350 capable of handling variable-length sequences and capturing global relationships. As an example, the encoder 350 can mine meta-features (such as dependencies between hyperparameters) from the hyperparameter feature representation set of the first set of hyperparameters 320 and encode these meta-features into a latent space Z with predetermined dimensions and semantics, thereby obtaining a first global feature representation 360. Based on this approach, the electronic device 120 can utilize the encoder 350 to process the hyperparameter feature representation set of each set of hyperparameters, thereby obtaining a global feature representation for each set of hyperparameters. Furthermore, since these global feature representations are all encoded into the latent space Z, they have the same dimensions and semantics; that is, these global feature representations are isomorphic global feature representations.

[0090] In some embodiments, the first global feature representation 360 includes at least one of the following determined based on target information: a dependency information feature representation, indicating the hyperparameters on which the first hyperparameter depends in the first set of hyperparameters 320; a value feature representation, indicating the value of the first hyperparameter; or a structural feature representation, indicating the structure of the first hyperparameter. As an example, the first global feature representation 360 includes a dependency information feature representation, a value feature representation, and a structural feature representation simultaneously.

[0091] It should be noted that, depending on actual needs, the first global feature representation 360 may include more feature representations, and the embodiments disclosed herein do not impose any limitations on this.

[0092] In some embodiments, the various feature representations of the first global feature representation 360, such as dependency information feature representation, value feature representation, and structural feature representation, can represent the characteristics of each hyperparameter in the first set of hyperparameters 320. Specifically, the dependency information feature representation can indicate the hyperparameters that each hyperparameter in the first set of hyperparameters 320 depends on; the value feature representation can indicate the value of each hyperparameter in the first set of hyperparameters 320; and the structural feature representation can indicate the structure of each hyperparameter in the first set of hyperparameters 320. In this way, the first global feature representation 360 can comprehensively reflect the characteristics of each hyperparameter in the first set of hyperparameters 320 with a predetermined structure and semantics.

[0093] In some embodiments, encoder 350 includes attention-based encoder 350, and the second machine learning model includes a Gaussian process model. As an example, attention-based encoder 350 includes attention blocks 380 (e.g., attention-based encoder 350 includes six attention blocks 380) and each attention block has one or more (e.g., two parallel) attention heads. In this way, the ability to focus and process information can be enhanced.

[0094] It should be noted that, depending on actual needs, the attention-based encoder 350 may include more attention blocks and each attention block may have more attention heads.

[0095] In some embodiments, the input and output dimensions of encoder 350 are determined based on the output dimension of the fully connected layer in a predetermined neural network 370. As an example, the input and output dimensions of encoder 350 are set to 256 (calculated based on the output dimension of the embedding module, which is 64×4), while its internal hidden layers can contain 512 units. This design allows encoder 350 to match the output of the embedding module while also improving encoder 350's ability to process complex features.

[0096] It should be noted that, according to actual needs, the input dimension and output dimension of encoder 350 and the number of units contained in its hidden layer can be determined according to actual needs, and the embodiments of this disclosure do not limit this.

[0097] In some embodiments, the electronic device 120 performs at least one compression on the output of the encoder 350 to obtain a first set of global feature representations. As an example, the electronic device 120 may employ at least two compression methods: one is to integrate the output of the encoder 350 using average pooling, and the other is to project the features using a multi-layer perceptron (MLP). As an example, the MLP can be a multi-layer perceptron with four hidden layers, each containing [128, 128, 128, 32] units, ultimately compressing the configured features into a 32-dimensional vector, thereby achieving an efficient representation of the features.

[0098] It should be noted that the number of hidden layers in a multilayer perceptron, the number of units contained in the hidden layers, and the dimension of the final output can be determined according to actual needs, and the embodiments disclosed herein do not impose any restrictions on this.

[0099] Based on the above method, the electronic device 120 can generate corresponding global feature representations for each set of hyperparameters. These global feature representations not only comprehensively reflect the characteristics of each hyperparameter in each set of hyperparameters, but also include the dependencies between hyperparameters, their specific values, and structural information, which enables the heterogeneous sets of hyperparameters 130 to be evaluated by the subsequent second machine learning model.

[0100] In box 230, electronic device 120 uses a second machine learning model (e.g., Gaussian process model) to determine the similarity between multiple sets of hyperparameters 130 associated with multiple global features 140 based on multiple global features 140.

[0101] In some embodiments, a Gaussian process model can be constructed by taking multiple global features 140 as input and the performance (such as accuracy, loss, etc.) of the first machine learning model as output (i.e., the evaluation results mentioned below). The Gaussian process model can determine the similarity between multiple global features using a kernel function, and then determine the similarity between multiple sets of hyperparameters 130 based on the similarity between the multiple global features. The kernel function is a key component of the Gaussian process model, determining how to calculate the similarity between the features of the input data. As an example, the kernel function may include at least one of linear kernel functions, polynomial kernel functions, Gaussian kernel functions, or Matérn kernel functions. As an example, in this disclosure, the kernel function used is the Matérn 5 / 2 kernel function. The Matérn 5 / 2 kernel function is a commonly used covariance function, where the function values ​​of two points separated by a distance d are defined as... Where 'a' is the amplitude parameter and 'ρ' is the length scale parameter. Compared to other kernel functions, such as the Gaussian kernel, the Matérn 5 / 2 kernel offers more flexibility, especially in controlling the smoothness of the function.

[0102] It should be noted that, depending on the actual needs, the kernel function in the Gaussian process model can be any suitable kernel function, and the embodiments disclosed herein will not be listed one by one.

[0103] In some embodiments, electronic device 120 (or other electronic devices different from electronic device 120) trains a second machine learning model based on a predetermined function until the predetermined function reaches a convergence condition, wherein the predetermined function is based at least on a log-marginal likelihood function.

[0104] In some embodiments, the log-marginal likelihood function comprehensively considers both the Gaussian process model's fit to the data and its complexity. By maximizing the log-marginal likelihood function, an optimal set of hyperparameters can be found. Specifically, the log-marginal likelihood function comprises several parts: a penalty term for the complexity of the Gaussian process model, which prevents the model from becoming too complex and overfitting the data; a data fitting term, which ensures the Gaussian process model accurately fits the observed data; and a log-normalization term, which is typically independent of the data but is still considered during computation. By balancing these terms, an optimal set of hyperparameters can be found. Furthermore, this process is highly automated. Once a suitable kernel function and log-marginal likelihood function are set, the Gaussian process model can automatically find the optimal set of hyperparameters without manual intervention. This automation not only simplifies the training process but also improves the efficiency and accuracy of hyperparameter selection.

[0105] In summary, Gaussian process models measure the similarity between input data using a kernel function and automatically optimize hyperparameter settings using a log-marginal likelihood function. This approach makes Gaussian process models more flexible and accurate when handling complex data.

[0106] As an example, for training a Gaussian process model, embodiments of this disclosure can employ a deep kernel learning framework, which jointly trains the parameters of the embedding module, the attention-based encoder 350, and the kernel function in the Gaussian process model by maximizing the log marginal likelihood. This process can be represented by formula (3):

[0107]

[0108] Where logp(Y│X,θ,ω) represents the log probability (or log likelihood) of the target variable Y given input data X, parameters θ, and ω, ∝ indicates proportionality to or in proportion to the target variable, and K... deep Represents the depth kernel matrix. K represents deep The inverse matrix, Y T This represents the transpose of Y. As an example, in, Let N represent the response value of the j-th observation point in the i-th branch of the tree search space 310, where n represents the number of branches in the tree search space 310. i Y represents the number of observation points on the i-th branch, and Y represents a set containing the response values ​​of all observation points.

[0109] As an example, K deep It can be determined by formulas (4) and (5):

[0110] K deep =k deep (X,X|θ,ω)+σ 2 I; (4)

[0111] k deep (X,X|θ,ω)+σ 2 I=k(φ(emb(X,ω1),ω2)|θ)+σ 2 I; (5) where k deep (X, X|θ, ω) represents a depth kernel function, which calculates the kernel matrix based on the input X, parameters θ and ω, σ 2 Let I represent the noise variance, φ(emb(X,ω1),ω2) represent a deep learning model. First, the input X is embedded into a new space using an embedding function emb(X,ω1), and then this embedding is further processed using another function φ (with parameter ω2). k(·|θ) represents the kernel matrix calculated based on the output of the deep learning model and the parameter θ. ω1 and ω2 represent the weight parameters of the embedding module and the attention-based encoder 350, respectively.

[0112] As an example, during training, for a Gaussian process model, the mean prior can be set to zero. A first optimizer is then used to train by maximizing the log-likelihood. For example, the first optimizer could be an Adaptive Moment Estimation (Adam) optimizer, applicable to deep learning. As an example, during training, for the Gaussian process model, the iteration cycle can be 100, the learning rate can be set to 0.001, and it can decay by a factor of 0.5 every 30 cycles. As an example, during the sampling phase, a second optimizer can be used to optimize a predetermined sampling function (e.g., the expected boosting function) in each branch of the tree search space 310. For example, the second optimizer could be a Limited-memory Broyden-Fletcher-Goldfarb-Shanno (LBFGS) optimizer, an optimization algorithm for unconstrained optimization problems. It belongs to the quasi-Newton method family and is an iterative optimization technique for finding local minima of differentiable scalar functions. As an example, when the number of branches is too large, Thompson sampling can be used as the sampling function.

[0113] It should be noted that the above parameters can be determined according to actual needs, and the embodiments disclosed herein do not impose any restrictions on them.

[0114] In box 240, electronic device 120 determines the evaluation result of at least one set of hyperparameters 130 based on the determined similarity.

[0115] In some embodiments, based on the similarity among multiple sets of hyperparameters 130, the Gaussian process model can predict the performance metric of the first machine learning model under at least one set of hyperparameters, such performance metric serving as an evaluation result of the set of hyperparameters. As an example, the evaluation result may include not only a specific value but also a variance value representing uncertainty. This allows the reliability of the evaluation result to be quantified. By iterating this process, the electronic device 120 can find the optimal set of hyperparameters based on the evaluation results of each set of hyperparameters.

[0116] As can be clearly understood from the description of the various embodiments of this disclosure, when faced with heterogeneous sets of hyperparameters, the embodiments of this disclosure can transform these heterogeneous sets of hyperparameters into multiple homogeneous global feature representations 140. As an example, each global feature representation may contain three parts: the first part may indicate the dependencies between the hyperparameters in its associated set of hyperparameters; the second part may indicate the structural information of the hyperparameters in its associated set of hyperparameters; and the third part may indicate the values ​​of the hyperparameters in its associated set of hyperparameters. In this way, all global feature representations have the same dimensions (three) and the same semantics (dependencies, structural information, and values) for each dimension. Such global feature representations can be processed by a second machine learning model (such as a Gaussian process model) to determine the evaluation results of each set of hyperparameters.

[0117] In this way, embodiments of the present disclosure can process heterogeneous sets of hyperparameters into a form that can be processed by Gaussian process models. This means that embodiments of the present disclosure can extend Bayesian optimization algorithms to handle complex search spaces where hyperparameters have dependencies on each other. This helps to accelerate the development process of machine learning model applications, thereby improving productivity.

[0118] Furthermore, in the embodiments of this disclosure, it is not necessary to introduce invalid variables into any set of hyperparameters, thereby improving the processing efficiency of hyperparameters. Moreover, in the above process, only a single surrogate model is needed to process the complex tree-shaped search space 310, and the number of observation points remains consistent with traditional Bayesian optimization methods, thus facilitating control over processing time.

[0119] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 4 A schematic structural block diagram of an apparatus 400 for model training according to some embodiments of the present disclosure is shown. The apparatus 400 may be implemented as or included in an electronic device 120. Various modules / components in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0120] Reference Figure 4The apparatus 400 includes a hyperparameter acquisition module 410, a global feature representation generation module 420, a similarity determination module 430, and an evaluation result determination module 440. In some embodiments, the hyperparameter acquisition module 410 is configured to acquire multiple sets of hyperparameters for a first machine learning model, wherein at least two sets of hyperparameters are heterogeneous. The global feature representation generation module 420 is configured to generate associated global feature representations for each set of hyperparameters based on the multiple sets of hyperparameters, to obtain multiple global feature representations, wherein each global feature representation is isomorphic. The similarity determination module 430 is configured to determine the similarity between the multiple sets of hyperparameters associated with the multiple global feature representations using a second machine learning model based on the multiple global feature representations. The evaluation result determination module 440 is configured to determine an evaluation result for at least one set of hyperparameters based on the determined similarity.

[0121] In some embodiments, the hyperparameter acquisition module 410 is further configured to: determine a target search space, the target search space including at least a tree search space; and sample each branch of the tree search space to obtain multiple sets of hyperparameters, wherein the same set of hyperparameters is determined based on the sampling results of the same branch.

[0122] In some embodiments, the hyperparameter acquisition module 410 is further configured to: sample a first branch in the tree search space to obtain a first set of hyperparameters; and sample a second branch in the tree search space to obtain a second set of hyperparameters that is heterogeneous with the first set of hyperparameters.

[0123] In some embodiments, the first set of hyperparameters and the second set of hyperparameters, which are heterogeneous with the first set of hyperparameters, satisfy at least one of the following relationships: at least one hyperparameter in the first set of hyperparameters has a different semantic from at least one hyperparameter in the second set of hyperparameters; or the number of hyperparameters in the first set of hyperparameters is different from the number of hyperparameters in the second set of hyperparameters.

[0124] In some embodiments, the hyperparameter acquisition module 410 is further configured to: sample each branch of the tree search space based on a predetermined sampling function, wherein the predetermined sampling function includes an expected boosting function and / or a Thompson sampling function.

[0125] In some embodiments, the global feature representation generation module 420 is further configured to: obtain a first set of hyperparameters from a plurality of hyperparameters; embed target information into the first hyperparameters in the first set of hyperparameters to generate a first hyperparameter feature representation; and, based on the first hyperparameter feature representation, use an encoder to generate a first global feature representation associated with the first set of hyperparameters.

[0126] In some embodiments, the target information indicates at least one of the following: the parent node of the node where the first hyperparameter is located in the tree search space; the name of the first hyperparameter; the value of the first hyperparameter; or the dimension information of the first hyperparameter.

[0127] In some embodiments, the first global feature representation includes at least one of the following determined based on target information: a dependency information feature representation, which indicates the hyperparameters on which the first hyperparameter depends in the first set of hyperparameters; a value feature representation, which indicates the value of the first hyperparameter; or a structural feature representation, which indicates the structure of the first hyperparameter.

[0128] In some embodiments, the encoder includes an attention-based encoder, and the second machine learning model includes a Gaussian process model.

[0129] In some embodiments, the global feature representation generation module 420 is further configured to: embed target information into a first hyperparameter in a first set of hyperparameters based on a predetermined neural network, wherein the predetermined neural network includes an embedding layer and a fully connected layer connected in sequence.

[0130] In some embodiments, the input and output dimensions of the encoder are determined based on the output dimension of the fully connected layer in a predetermined neural network.

[0131] In some embodiments, the global feature representation generation module 420 is further configured to perform at least one compression on the encoder output to obtain a first global feature representation.

[0132] In some embodiments, the global feature representation generation module 420 is further configured to: compress the encoder output using average pooling; or compress the encoder output using a multilayer perceptron.

[0133] In some embodiments, the apparatus 400 further includes a training module configured to train a second machine learning model based on a predetermined function until the predetermined function reaches a convergence condition, wherein the predetermined function is based at least on a log-marginal likelihood function.

[0134] Figure 5 A block diagram is shown of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. The electronic device 500 may, for example, be used to implement... Figure 1 The electronic device 120 shown. It should be understood that, Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein.

[0135] Reference Figure 5Electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage devices 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0136] Electronic device 500 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data and accessible within electronic device 500.

[0137] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0138] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0139] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0140] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0141] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0142] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0143] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0144] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0145] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is determined to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for hyperparameter evaluation, comprising: Obtain multiple sets of hyperparameters for the first machine learning model, wherein at least two sets of hyperparameters are heterogeneous hyperparameters. Based on the multiple sets of hyperparameters, a global feature representation associated with each set of hyperparameters is generated to obtain multiple global feature representations, wherein each global feature representation in the multiple global feature representations is an isomorphic global feature representation. Based on the multiple global feature representations, a second machine learning model is used to determine the similarity between multiple sets of hyperparameters associated with the multiple global feature representations; as well as Based on the determined similarity, the evaluation result of at least one set of hyperparameters among the plurality of hyperparameters is determined.

2. The method according to claim 1, wherein obtaining the plurality of hyperparameters comprises: Determine the target search space, which includes at least a tree-structured search space; as well as Each branch of the tree search space is sampled to obtain the multiple sets of hyperparameters, wherein the same set of hyperparameters is determined based on the sampling results of the same branch.

3. The method according to claim 2, wherein sampling each branch of the tree search space comprises: The first branch in the tree search space is sampled to obtain the first set of hyperparameters; as well as The second branch in the tree search space is sampled to obtain a second set of hyperparameters that is heterogeneous with the first set of hyperparameters.

4. The method according to claim 3, wherein the first set of hyperparameters and the second set of hyperparameters heterogeneous with the first set of hyperparameters satisfy at least one of the following relationships: At least one hyperparameter in the first group of hyperparameters has a different semantic from at least one hyperparameter in the second group of hyperparameters; or The number of hyperparameters in the first group is different from the number of hyperparameters in the second group.

5. The method according to claim 2, wherein sampling each branch of the tree search space includes: Based on a predetermined sampling function, each branch of the tree search space is sampled, wherein the predetermined sampling function includes an expected boosting function and / or a Thompson sampling function.

6. The method of claim 1, wherein generating a global feature representation associated with each of the plurality of hyperparameters comprises: Obtain the first set of hyperparameters from the plurality of sets of hyperparameters; Target information is embedded into the first hyperparameter in the first group of hyperparameters to generate a first hyperparameter feature representation; as well as Based on the first hyperparameter feature representation, an encoder is used to generate a first global feature representation associated with the first set of hyperparameters.

7. The method of claim 6, wherein the target information indicates at least one of the following: In the tree search space, the parent node of the node where the first hyperparameter is located; The name of the first hyperparameter; The value of the first hyperparameter; or The dimensional information of the first hyperparameter.

8. The method of claim 6, wherein the first global feature representation comprises at least one of the following determined based on the target information: Dependency information feature representation, wherein the dependency information feature representation indicates the hyperparameters that the first hyperparameter depends on in the first set of hyperparameters; Value feature representation, wherein the value feature representation indicates the value of the first hyperparameter; or The structural feature representation indicates the structure of the first hyperparameter.

9. The method of claim 6, wherein the encoder comprises an attention-based encoder, and the second machine learning model comprises a Gaussian process model.

10. The method of claim 6, wherein embedding target information for the first hyperparameter in the first set of hyperparameters includes: Based on a predetermined neural network, target information is embedded into the first hyperparameter in the first set of hyperparameters, wherein the predetermined neural network includes an embedding layer and a fully connected layer connected in sequence.

11. The method of claim 10, wherein the input dimension and output dimension of the encoder are determined based on the output dimension of the fully connected layer in the predetermined neural network.

12. The method of claim 6, wherein generating a first global feature representation associated with the first set of hyperparameters comprises: The output of the encoder is compressed at least once to obtain the first global feature representation.

13. The method of claim 12, wherein performing at least one compression on the output of the encoder comprises at least one of the following: The encoder output is compressed using average pooling; or The output of the encoder is compressed using a multilayer perceptron.

14. The method according to claim 1, further comprising: The second machine learning model is trained based on a predetermined function until the predetermined function reaches a convergence condition, wherein the predetermined function is at least based on a log-marginal likelihood function.

15. An apparatus for hyperparameter evaluation, comprising: The hyperparameter acquisition module is configured to acquire multiple sets of hyperparameters for the first machine learning model, wherein at least two sets of hyperparameters are heterogeneous hyperparameters. The global feature representation generation module is configured to generate a global feature representation associated with each of the multiple sets of hyperparameters based on the multiple sets of hyperparameters, so as to obtain multiple global feature representations, wherein each of the multiple global feature representations is an isomorphic global feature representation. The similarity determination module is configured to determine the similarity between multiple sets of hyperparameters associated with the multiple global feature representations using a second machine learning model based on the multiple global feature representations. as well as The evaluation result determination module is configured to determine the evaluation result for at least one set of hyperparameters among the plurality of hyperparameters based on the determined similarity.

16. An electronic device comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 14 when executed by the at least one processing unit.

17. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 14.

18. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 14.

Images