Tranform-based hyper-parameter space analysis method and device, electronic equipment and storage medium

Abstract

The invention provides a hyperparameter space analysis method and device based on Transform, electronic equipment and a storage medium. The method provided by the invention comprises the steps that hyper-parameter data is acquired, the hyper-parameter data has a plurality of dimensions, and each dimension represents one hyper-parameter; executing forward propagation of a Transform model by using data on the second dimension in the hyper-parameter data to obtain prediction data on the first dimension in the hyper-parameter data; executing back propagation of a Transform model by using the prediction data and the data on the first dimension in the hyper-parameter data so as to obtain gradient information of each dimension in the second dimension; according to the gradient information of each dimension in the second dimensions, a hyper-parameter space boundary is determined, the hyper-parameter space boundary comprises a first value range of each dimension in the second dimensions, and when the data on each dimension in the second dimensions are in the first value range, the influence on the value of the data on the first dimension is maximum. According to the embodiment of the invention, the hyper-parameter space boundary can be efficiently, quickly and accurately obtained by utilizing Transform.

Description

Technical Field

[0001] The present application relates to the field of data processing, and specifically to a Transformer-based hyperparameter space analysis method, device, electronic device and storage medium. Background Art

[0002] In-depth exploration and research of hyperparameter space is one of the important topics in the field of multidimensional big data analysis, and its goal is to deeply understand the correlation and structural information hidden in the data.

[0003] In the current data analysis field, various correlation analysis methods such as Pearson correlation coefficient and Spearman rank correlation coefficient are often used to reveal the relationship between features in the data. In addition, methods such as causal relationship analysis, regression analysis, factor analysis, association rule mining and time series analysis are widely used. Among them, one of the main disadvantages of the correlation coefficient is that it can only describe linear relationships and has a weak ability to describe nonlinear relationships. In addition, the correlation coefficient cannot determine the causal relationship between variables, but only reflects the degree of association between them. In addition, the correlation coefficient is sensitive to outliers and data distribution, which may lead to deviations in the results. Compared with correlation coefficient analysis, causal analysis focuses more on determining the causal relationship between variables, that is, how one variable affects another variable. However, causal analysis requires more data and more rigorous experimental design to accurately infer causal relationships. In addition, the computational complexity of causal analysis is usually high, which may require more time and computing resources. As a common statistical method used to explore the relationship between independent variables and dependent variables, regression analysis also has some disadvantages. First, most regression models are based on linear assumptions and ignore possible nonlinear relationships, resulting in limited fitting. Secondly, it is easily affected by data noise, and may overfit or underfit, which reduces the generalization ability of the model. And it has high requirements on data and needs to meet some basic assumptions, such as linear relationship, independence, normality, etc. Otherwise, it will affect the accuracy and reliability of the model.

[0004] It can be seen that the above methods have limitations when dealing with complex data relationships. They are unable to find the boundaries of the hyperparameter space, which is of great significance in helping staff to tune the model. Therefore, a solution that can analyze the hyperparameter space is needed to accurately determine the boundaries of the hyperparameter space. Summary of the invention

[0005] In view of this, the embodiments of the present application are committed to providing a Transformer-based hyper-parameter space analysis method, device, electronic device and storage medium to efficiently and accurately find the hyper-parameter space boundary.

[0006] In one aspect of the present application, a Transformer-based hyperparameter space analysis method is provided, comprising:

[0007] Obtaining hyperparameter data, where the hyperparameter data has multiple dimensions, each dimension representing a hyperparameter;

[0008] Using the data on the second dimension in the hyperparameter data to perform forward propagation of the Transformer model to obtain predicted data of the first dimension in the hyperparameter data;

[0009] Performing back propagation of a Transformer model using the predicted data and the data on the first dimension in the hyperparameter data to obtain gradient information of each dimension in the second dimension;

[0010] The hyper-parameter space boundary is determined according to the gradient information of each dimension in the second dimension, and the hyper-parameter space boundary includes a first value range of each dimension in the second dimension. When the data on each dimension in the second dimension is in the first value range, the data on the first dimension has the greatest impact on the value of the data on the first dimension.

[0011] In some implementations of the first aspect of the present application, it also includes: performing data preprocessing on the hyperparameter data, and the data preprocessing includes one or more of the following: data cleaning, missing value processing, and consistency processing.

[0012] In some implementations of the first aspect of the present application, it also includes: by constructing a minimum mean square error loss, optimizing the Transformer model using a gradient descent algorithm so that the Transformer model can fit the mapping relationship between the first dimension and each dimension in the second dimension.

[0013] In some embodiments of the first aspect of the present application, the forward propagation of the Transformer model is performed using the data on the second dimension in the hyperparameter data to obtain the predicted data of the first dimension in the hyperparameter data, including: extracting the data on each dimension in the second dimension in the hyperparameter data respectively to form a feature vector of each dimension in the second dimension; inputting the feature vector of each dimension in the second dimension into the encoder of the Transformer model to generate an encoding information matrix; and sending the encoding information matrix into the decoder of the Transformer model to generate an output feature vector, which is the vector representation of the predicted data.

[0014] In some implementations of the first aspect of the present application, the first dimension includes one or more dimensions in the hyperparameter data, and the second dimension includes all dimensions in the hyperparameter data except the first dimension.

[0015] In some embodiments of the first aspect of the present application, the method of performing back propagation of the Transformer model using the data on the first dimension in the predicted data and the hyperparameter data to obtain gradient information of each dimension in the second dimension includes: calculating the reverse gradient of each layer in the Transformer model using the data on the first dimension in the predicted data and the hyperparameter data, and back-propagating the reverse gradient layer by layer in the Transformer model to the input layer of the Transformer model using the chain rule, so that the gradient of the data on the first dimension to the data on one dimension in the second dimension is obtained through gradient iteration.

[0016] In some embodiments of the first aspect of the present application, determining the hyper-parameter space boundary based on the gradient information of each dimension in the second dimension includes: determining a focus dimension in the second dimension; traversing all of the gradient information, statistically obtaining and storing the gradient information corresponding to the data on the focus dimension; and finding a numerical interval with a maximum gradient absolute value in the data on the focus dimension, wherein the numerical interval is the first value range of the focus dimension.

[0017] In a second aspect of the present application, a Transformer-based hyperparameter spatial analysis device is provided, comprising:

[0018] An acquisition unit, used to acquire hyperparameter data, wherein the hyperparameter data has multiple dimensions, each dimension representing a hyperparameter;

[0019] A forward processing unit, configured to perform forward propagation of a Transformer model using data on a second dimension in the hyperparameter data to obtain predicted data on a first dimension in the hyperparameter data;

[0020] A gradient analysis unit, configured to perform back propagation of a Transformer model using the predicted data and the data on the first dimension in the hyperparameter data to obtain gradient information of each dimension in the second dimension;

[0021] A boundary determination unit is used to determine the boundary of the hyper-parameter space according to the gradient information of each dimension in the second dimension, and the boundary of the hyper-parameter space includes a first value range of each dimension in the second dimension. When the data on each dimension in the second dimension is in the first value range, the data on the first dimension has the greatest impact on the value of the data in the first dimension.

[0022] The third aspect of the present application provides an electronic device, comprising: a processor and a memory; wherein the memory is connected to the processor, and the memory is used to store a computer program; the processor is used to implement the above method by running the computer program stored in the memory.

[0023] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above method is implemented.

[0024] According to an embodiment of the present application, the hyperparameter data is fitted through the Tranformer model so that the Tranformer model learns the mapping relationship between each dimension in the second dimension and the first dimension in the hyperparameter data, and uses the mapping relationship learned by the Tranformer model to seek the gradient. Thereafter, the gradient is used to find the value range of each dimension in the second dimension that has the greatest impact on the change in data value in the first dimension, thereby efficiently, quickly and accurately obtaining the hyperparameter space boundary, and at the same time, the accuracy of the hyperparameter space boundary can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 It is a flowchart of a Transformer-based hyperparameter space analysis method provided in an embodiment of the present application;

[0027] Figure 2 is an exemplary implementation flowchart of determining the hyper-parameter space boundary using gradient information according to an embodiment of the present application;

[0028] Figure 3 It is a structural schematic diagram of a Transformer-based hyperparameter spatial analysis device provided in an embodiment of the present application;

[0029] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] Terminology explanation:

[0032] Hyperparameter space: Hyperparameter space. In machine learning, hyperparameters refer to parameters that control the learning process. Unlike model parameters, which are automatically learned from data during the learning process (such as the weights of a neural network), hyperparameters are usually set before learning begins and remain unchanged during the learning process. Hyperparameter space refers to the set of all possible hyperparameter values. In the hyperparameter space, each dimension represents a hyperparameter, and different dimensions have different degrees of correlation, while each point represents a specific set of hyperparameter settings. For example, if there are two hyperparameters: learning rate and batch size, then the hyperparameter space is a two-dimensional space, with one axis representing the learning rate and the other axis representing the batch size. The size and complexity of the hyperparameter space depends on the number of hyperparameters and the range of their possible values. High-dimensional spaces or hyperparameters with a large number of possible values ​​make it more difficult to search for the optimal hyperparameter configuration, which is the so-called "curse of dimensionality". Therefore, choosing an appropriate hyperparameter space is crucial for model tuning.

[0033] Hyperparameter space boundary: the boundary of the hyperparameter space, which refers to a complete set of possible value ranges for the hyperparameters. The boundary of the hyperparameter space has an important impact on the optimization and selection of machine learning models. In the disclosed embodiment, the hyperparameter space boundary refers to the sampling interval where the other dimensions in the hyperparameter space have the greatest impact on a specified dimension.

[0034] Exemplary Methods

[0035] Figure 1 FIG. 1 is a flow chart of a Transformer-based hyperparameter space analysis method provided in an embodiment of the present disclosure. Figure 1 , the method may include the following steps:

[0036] Step 101, obtaining hyperparameter data, where the hyperparameter data has multiple dimensions, and each dimension represents a hyperparameter;

[0037] Step 102, using the data on the second dimension in the hyperparameter data to perform forward propagation of the Transformer model to obtain the predicted data of the first dimension in the hyperparameter data;

[0038] Step 103, performing back propagation of the Transformer model using the data on the first dimension in the prediction data and the hyperparameter data to obtain the gradient information of each dimension in the second dimension;

[0039] Step 104, determine the hyper-parameter space boundary according to the gradient information of each dimension in the second dimension, the hyper-parameter space boundary includes the first value range of each dimension in the second dimension, and when the data on each dimension in the second dimension is in the first value range, the value of the data on the first dimension has the greatest impact.

[0040] In the disclosed embodiments, the hyperparameter data is a data set of hyperparameters of the model, and the model can be any type of machine learning model. In some embodiments, if there are N kinds of hyperparameters of the model, N is an integer greater than 1, the hyperparameter data of the model is N-dimensional data, and the hyperparameter space corresponding to the hyperparameter data is N-dimensional space. If the model has thousands or more hyperparameters, its hyperparameter data is high-dimensional data of thousands or ten thousand dimensions, and the corresponding hyperparameter space is a high-dimensional space such as thousands of dimensions and ten thousand dimensions. Exemplarily, the hyperparameter data can be represented as multidimensional feature data (X1, X2, ..., Xn), n represents the total number of dimensions of the hyperparameter data, n is an integer greater than 1, Xi (i = 1, 2, ..., n) represents the data on dimension i, that is, the possible value of hyperparameter i.

[0041] In step 101, the hyper-parameter data may be input by a staff member or may come from other electronic devices. The specific method of obtaining the hyper-parameter data is not limited in the embodiments of the present disclosure.

[0042] In some implementations, before step 102, the following may also be included: performing data preprocessing on the hyperparameter data, where the data preprocessing includes one or more of the following: data cleaning and normalization. Data preprocessing can effectively improve the data quality of the hyperparameter data, reduce the risk of errors, and provide a reliable basis for subsequent processing.

[0043] In some examples, data cleaning may include checking, correcting, and screening the first hyperparameter data to ensure the accuracy and completeness of the first hyperparameter data.

[0044] In some examples, the normalization process may include: unifying the units and scales of data of different dimensions in the first hyperparameter data. By normalizing the first hyperparameter data, the dimensional differences between different features may be eliminated.

[0045] In other implementations, data preprocessing may also include: inconsistency processing, missing value processing, etc. The specific process and implementation of data preprocessing are not limited in the embodiments of the present disclosure.

[0046] Before step 102, the method of the embodiment of the present application may further include: a training step of a Transformer model. Specifically, the training step of the Transformer model may include but is not limited to parameter initialization, hyperparameter selection, training and testing of the Transformer model. Among them, parameter initialization and hyperparameter selection of the Transformer model are optional operations.

[0047] Specifically, before step 102, the method may further include: by constructing a minimum mean square error loss, optimizing the Transformer model using a gradient descent algorithm so that the Transformer model can fit the mapping relationship between the first dimension and each dimension in the second dimension. Specifically, the constructed data set is loaded into a training environment, the Transformer model is trained using the loaded data set, and the model parameters of the Transformer model are continuously adjusted by the gradient descent algorithm to minimize the loss value of the loss function until convergence. After the training is completed, the model parameters of the Transformer model are saved as a file and the file is saved in a specified path. In this way, in step 102, only the file under the specified path needs to be loaded and the Transformer model needs to be run to perform forward propagation for the hyperparameter data.

[0048] The dataset used to train the Transformer model can be obtained from a reliable source to ensure the quality of the data in the dataset and the accuracy of the labels. Before training the Transformer model, the data in the dataset can be preprocessed to further improve the instructions of the data in the dataset.

[0049] After training, the Transformer model can characterize the mapping relationship between the first dimension and the second dimension in the hyperparameter space corresponding to the hyperparameter data. For example, if the hyperparameter data has n dimensions, the first dimension is selected as the third dimension of the hyperparameter data, and the first, second, and fourth to nth dimensions of the hyperparameter data all belong to the second dimension, then the Transformer model can characterize the mapping relationship between the third dimension in the hyperparameter space and the other dimensions in the second dimension, that is, the Transformer model can use the data in the first, second, and fourth to nth dimensions of the hyperparameter space as input features, and output the predicted value of the data in the first dimension in the hyperparameter space.

[0050] The first dimension may include one or more dimensions in the hyperparameter data, and the second dimension may include all dimensions in the hyperparameter except the first dimension. For example, if the hyperparameter data has n dimensions, and the first dimension is selected as the third dimension of the hyperparameter data, then the first dimension, the second dimension, and the fourth dimension to the nth dimension of the hyperparameter data all belong to the second dimension. By using the embodiment of the present disclosure, the Transformer model is used to model the hyperparameter space, and the discretized hyperparameter data can be converted into continuous hyperparameter data, so that the granularity of the hyperparameter space is refined, which facilitates the accurate finding of the hyperparameter data boundary with higher precision through subsequent steps.

[0051] In step 102, a forward propagation process is performed, that is, the data on each dimension in the second dimension is used as input features through the encoder and decoder of the Transformer model layer by layer to generate predicted data of the first dimension.

[0052] In some implementations, the exemplary implementation process of step 102 may include the following steps a1 to a3:

[0053] Step a1, extracting data on each dimension in the second dimension of the hyper-parameter data to form a feature vector of each dimension in the second dimension;

[0054] Step a2, inputting the feature vector of each dimension in the second dimension into the encoder of the Transformer model to generate a coding information matrix;

[0055] Step a3, sending the encoded information matrix to the decoder of the Transformer model to generate an output feature vector, which is a vector representation of the predicted data of the first dimension in the hyperparameter data.

[0056] The Transformer model of the disclosed embodiment may include N encoders (Encoder block) and N decoders (Decoder block), where N is an integer greater than or equal to 1. A multi-head attention unit (Multi-Head Attention) and a feedforward neural network (Feed Forward) are provided in each encoder and decoder, and a residual normalization (Add&Norm) layer is provided above each multi-head attention unit.

[0057] The encoder contains one multi-head attention unit. The decoder contains two multi-head attention units, one of which uses a mask. In the Add&Norm layer, Add represents the residual connection, which is used to prevent network degradation, and Norm represents the layer normalization, which is used to normalize the activation values ​​of each layer.

[0058] The multi-head attention unit is composed of multiple self-attention units (Self-Attention). The query matrix Q, key value matrix K and value V are required for calculation of the self-attention unit. In the embodiment of the present disclosure, the query matrix Q, key value matrix K and value V required by the self-attention unit are obtained by linear transformation based on the data on each dimension in the second dimension of the hyperparameter data or the encoding information matrix output by the previous encoder.

[0059] In some implementations, assuming that the feature vectors in each dimension of the second dimension in the hyperparameter data or the encoding information matrix output by the previous encoder is a matrix X, the linear transformation matrix W can be used. Q、 W K , W V Multiplying by matrix X gives the query matrix Q, key-value matrix K, and value V required by the self-attention unit.

[0060] It should be noted that the architecture of the Tranformer model used in the embodiment of the present disclosure is not limited to the above-mentioned method. The embodiment of the present disclosure does not limit the specific structure of the Tranformer model.

[0061] Taking into account that the multi-dimensional hyper-parameter data often has complex internal data relationships, and different dimensions often have a certain correlation with each other, therefore, in order to solve the problem that the existing correlation method cannot model nonlinear complex features, the Transformer model is used in the embodiment of the present disclosure to model the multi-dimensional hyper-parameter data. Since the Transformer model is built based on the self-attention mechanism, the self-attention mechanism can enable the Transformer model to effectively capture the dependencies in the input features. This mechanism also determines that the Transformer model can better pay attention to the correlation between features. Therefore, the Transformer model can more accurately characterize the complex correlations between different dimensions in the hyper-parameter data, thereby determining which values ​​of the data in each dimension in the second dimension have a greater impact on the numerical changes of the data in the first dimension by calculating the gradient of the input features relative to the prediction results, thereby achieving the purpose of accurately locating the boundaries of the hyper-parameter space.

[0062] In step 103, it is difficult to directly obtain the gradient of the output feature of a complex Transformer model with respect to the input feature, so the embodiment of the present disclosure introduces a reverse chain derivation technique. That is, in some implementations, step 103 may include: using a reverse chain to derive the gradient of the output of the Transformer model relative to the input, which is the gradient of the data in the first dimension relative to the other dimensions in the second range.

[0063] Specifically, step 103 may include: using the data on the first dimension in the prediction data and the hyperparameter data to calculate the reverse gradient of each layer in the Transformer model, and using the chain rule to propagate the reverse gradient layer by layer in the Transformer model to the input layer of the Transformer model, so that the gradient of the data on the first dimension to the data on one dimension in the second dimension is obtained through gradient iteration. Among them, the nonlinear function part in the Transformer model is replaced by linear function approximation fitting. In this way, the gradient of the data on the first dimension to the data on each dimension in the second dimension can be obtained, that is, the gradient information is obtained.

[0064] Considering that the gradient will be distributed at various positions according to the degree of influence of the input in the Transformer model during the back-propagation process, the gradient information can be used to analyze the criticality and sensitivity of the input, so as to determine the impact of the specific input variable on the output, that is, to find the input interval corresponding to the fastest output change. It can be seen that the method of step 104 can efficiently, quickly and accurately find the first value range of each dimension in the second dimension.

[0065] In step 104, the boundary of the hyperparameter space can be expressed as Xi∈[a,b], where Xi represents the first value range of dimension i, and dimension i belongs to any dimension in the second dimension. When the data value on dimension i is in the corresponding range of Xi, it has the greatest impact on the numerical change of the data on the first dimension. a is the lower limit of the first value range, and b is the upper limit of the first value range.

[0066] There are many ways to use gradient information to determine the hyper-parameter space boundary in step 104. Figure 2 FIG. 1 shows an exemplary implementation flow chart of determining the hyper-parameter space boundary using gradient information in step 104. Figure 2 In some implementations, the exemplary implementation process in step 104 may include the following steps:

[0067] Step 201, determining a focus dimension in the second dimension;

[0068] Step 202, traverse all gradient information, obtain and store the gradient information corresponding to the data on the dimension of interest;

[0069] Step 203, in the data on the dimension of interest, find a numerical interval with a maximum absolute value of the gradient, and the numerical interval is the first value range of the dimension of interest.

[0070] The larger the absolute value of the gradient, the more sensitive the output of the Transformer model is to its changes. That is, within the numerical range of the data on the dimension of interest corresponding to the maximum absolute value of the gradient, the value of the data on the dimension of interest has the greatest impact on the change in the value of the data on the first dimension.

[0071] In practical applications, it only takes seconds to traverse tens of thousands of data in step 104 using the above method. Therefore, using gradient information can efficiently, quickly and accurately solve the value range of each dimension on the second dimension that has the greatest impact on the value change of the data on the first dimension, that is, using gradient information can obtain the first value range of each dimension on the second dimension, and this value range is the boundary of the hyperparameter space.

[0072] The disclosed embodiment performs gradient analysis through Tranformer, that is, hyper-parameter data is fitted through Tranformer model so that the Tranformer model learns the mapping relationship between each dimension in the second dimension and the first dimension in the hyper-parameter data, and uses the mapping relationship learned by the Tranformer model to seek the gradient, and then uses the gradient to find the value range of each dimension in the second dimension that has the greatest impact on the change of data value in the first dimension, so as to efficiently, quickly and accurately obtain the hyper-parameter space boundary, and at the same time improve the accuracy of the hyper-parameter space boundary.

[0073] Exemplary Devices

[0074] Figure 3 FIG. 1 shows a schematic diagram of the structure of a Transformer-based hyperparameter spatial analysis device provided in an embodiment of the present application. Figure 3 , the Transformer-based hyperparameter spatial analysis device 300 may include:

[0075] An acquisition unit 301 is used to acquire hyperparameter data, where the hyperparameter data has multiple dimensions, and each dimension represents a hyperparameter;

[0076] A forward processing unit 302 is used to perform forward propagation of the Transformer model using the data on the second dimension in the hyperparameter data to obtain predicted data of the first dimension in the hyperparameter data;

[0077] A gradient analysis unit 303 is used to perform back propagation of the Transformer model using the data on the first dimension in the prediction data and the hyperparameter data to obtain gradient information of each dimension in the second dimension;

[0078] The boundary determination unit 304 is used to determine the boundary of the hyper-parameter space according to the gradient information of each dimension in the second dimension. The boundary of the hyper-parameter space includes the first value range of each dimension in the second dimension. When the data on each dimension in the second dimension is in the first value range, the value of the data on the first dimension has the greatest impact.

[0079] In some implementations, the Transformer-based hyperparameter space analysis device 300 may further include: a preprocessing unit 305 for performing data preprocessing on the hyperparameter data, where the data preprocessing includes one or more of the following: data cleaning, missing value processing, and consistency processing.

[0080] In some embodiments, the Transformer-based hyperparameter space analysis device 300 may also include: a model training unit 306, which is used to optimize the Transformer model by constructing a minimum mean square error loss and using a gradient descent algorithm so that the Transformer model can fit the mapping relationship between the first dimension and each dimension in the second dimension.

[0081] In some embodiments, the forward processing unit 302 can be specifically used to: extract data on each dimension in the second dimension of the hyperparameter data to form a feature vector for each dimension in the second dimension; input the feature vector of each dimension in the second dimension into the encoder of the Transformer model to generate an encoding information matrix; and, send the encoding information matrix into the decoder of the Transformer model to generate an output feature vector, which is a vector representation of the predicted data.

[0082] In some embodiments, the gradient analysis unit 303 can be specifically used to: calculate the reverse gradient of each layer in the Transformer model using the data on the first dimension in the prediction data and the hyperparameter data, and use the chain rule to backpropagate the reverse gradient layer by layer in the Transformer model to the input layer of the Transformer model, so that the gradient of the data on the first dimension with respect to the data on one dimension in the second dimension is obtained through gradient iteration.

[0083] In some implementations, the boundary determination unit 304 can be specifically used to: determine a focus dimension in the second dimension; traverse all gradient information, statistically obtain and store the gradient information corresponding to the data on the focus dimension; and find a numerical interval with a maximum gradient absolute value in the data on the focus dimension, where the numerical interval is the first value range of the focus dimension.

[0084] Other technical details of the Transformer-based hyperparameter spatial analysis device 300 in the embodiment of the present application can be found in the previous method section and will not be repeated here.

[0085] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0086] Electronic devices

[0087] Figure 4 FIG. 1 shows an example diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device may include: one or more processors 401, and a memory 402 storing one or more programs, which are executed by the one or more processors 401 to implement the method flow shown in the above embodiments of the present disclosure and / or program units corresponding to each unit in the device.

[0088] The various components are interconnected using different buses and can be mounted on a common motherboard or otherwise as needed. The processor 401 can process instructions executed within the electronic device, including instructions stored in or on the memory to display graphical information of a user interface on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memories if desired.

[0089] The processor 401 may include one or more single-core processors or multi-core processors. The processor 401 may include any combination of general-purpose processors or dedicated processors (such as image processors, application processors, baseband processors, etc.).

[0090] The memory 402 is a computer-readable storage medium provided by the present disclosure, which can be used to store non-transient software programs, non-transient computer executable programs and units, such as the following in the embodiments of the present disclosure: Figure 1 The processor 401 executes the non-transient software programs, instructions and units stored in the memory 402, thereby executing the above method embodiments. Figure 1The methods shown correspond to programs, instructions, and units.

[0091] The electronic device may further include: an input device 403 and an output device 404. The processor 401, the memory 402, the input device 403 and the output device 404 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0092] The input device 403 can receive input digital or character information, and generate signal input related to user settings and function control of the camera calibration device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, and other input devices. The output device 404 may include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0093] The above-mentioned programs (also referred to as software, software applications, or codes) include machine instructions for programmable processors, and these computer programs can be implemented using object-oriented programming languages, assembly or machine languages.

[0094] With the development of time and technology, the meaning of medium is becoming more and more extensive, and the propagation path of computer programs is no longer limited to tangible media, and can also be downloaded directly from the network, etc. Any combination of one or more computer-readable storage media can be used. Computer-readable storage media can be used but not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing programs, which can be used by or in combination with instruction execution systems, devices or devices.

[0095] In one implementation, the electronic device may be, but is not limited to, electronic devices with relatively high power consumption requirements, such as laptop computers and desktop computers commonly used by ARM chip system developers.

[0096] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. The program includes instructions. When the instructions are executed by one or more processors of a computing device, the steps of the method described in any one of the aforementioned method embodiments are executed.

[0097] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the steps of any one of the methods described in the aforementioned method embodiments.

[0098] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A Transformer-based hyperparameter space analysis method, comprising: Obtaining hyperparameter data, where the hyperparameter data has multiple dimensions, each dimension representing a hyperparameter; Perform forward propagation of the Transformer model using the data on the second dimension in the hyperparameter data to obtain predicted data of the first dimension in the hyperparameter data; Performing back propagation of a Transformer model using the predicted data and the data on the first dimension in the hyperparameter data to obtain gradient information of each dimension in the second dimension; The hyper-parameter space boundary is determined according to the gradient information of each dimension in the second dimension, and the hyper-parameter space boundary includes a first value range of each dimension in the second dimension. When the data on each dimension in the second dimension is in the first value range, the data on the first dimension has the greatest impact on the value of the data on the first dimension.

2. The method according to claim 1, characterized in that Also includes: The hyperparameter data is preprocessed, and the data preprocessing includes one or more of the following: data cleaning, missing value processing, and consistency processing.

3. The method according to claim 1, characterized in that Also includes: By constructing a minimum mean square error loss, the Transformer model is optimized using a gradient descent algorithm so that the Transformer model can fit the mapping relationship between the first dimension and each dimension in the second dimension.

4. The method according to claim 1, characterized in that: Utilizing the data on the second dimension in the hyperparameter data to perform forward propagation of the Transformer model to obtain predicted data of the first dimension in the hyperparameter data, including: Respectively extracting data on each dimension in the second dimension of the hyperparameter data to form a feature vector of each dimension in the second dimension; Inputting the feature vector of each dimension in the second dimension into the encoder of the Transformer model to generate an encoding information matrix; The encoded information matrix is ​​fed into the decoder of the Transformer model to generate an output feature vector, which is the vector representation of the predicted data.

5. The method according to claim 1, characterized in that The first dimension includes one or more dimensions in the hyperparameter data, and the second dimension includes all dimensions in the hyperparameter data except the first dimension.

6. The method according to claim 1, characterized in that The performing back propagation of the Transformer model using the predicted data and the data on the first dimension in the hyperparameter data to obtain gradient information of each dimension in the second dimension includes: The reverse gradient of each layer in the Transformer model is calculated using the predicted data and the data on the first dimension of the hyperparameter data, and the reverse gradient is back-propagated layer by layer in the Transformer model to the input layer of the Transformer model using the chain rule. In this way, the gradient of the data on the first dimension with respect to the data on one dimension in the second dimension is obtained through gradient iteration.

7. The method according to claim 6, characterized in that The step of determining the hyper-parameter space boundary according to the gradient information of each dimension in the second dimension includes: Determine the dimension of interest in the second dimension; Traversing all the gradient information, obtaining and storing the gradient information corresponding to the data on the dimension of interest; A numerical interval with a maximum gradient absolute value is found in the data on the focus dimension, and the numerical interval is the first value range of the focus dimension.

8. A hyperparameter space analysis device based on Transformer, characterized in that: include: An acquisition unit, used to acquire hyperparameter data, wherein the hyperparameter data has multiple dimensions, each dimension representing a hyperparameter; A forward processing unit, configured to perform forward propagation of a Transformer model using data on a second dimension in the hyperparameter data to obtain predicted data on a first dimension in the hyperparameter data; A gradient analysis unit, configured to perform back propagation of a Transformer model using the predicted data and the data on the first dimension in the hyperparameter data to obtain gradient information of each dimension in the second dimension; A boundary determination unit is used to determine the boundary of the hyper-parameter space according to the gradient information of each dimension in the second dimension, and the boundary of the hyper-parameter space includes a first value range of each dimension in the second dimension. When the data on each dimension in the second dimension is in the first value range, the data on the first dimension has the greatest impact on the value of the data in the first dimension.

9. An electronic device, characterized in that: include: Processor and memory; Wherein, the memory is connected to the processor, and the memory is used to store a computer program; The processor is configured to implement the method according to any one of claims 1 to 7 by running the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images