Tranform-based hyper-parameter space analysis method and device, electronic equipment and storage medium

Abstract

The invention provides a hyperparameter space analysis method and device based on Transform, electronic equipment and a storage medium. The method comprises the steps that hyper-parameter data of the precision system is acquired, the hyper-parameter data has multiple dimensions, each dimension represents one hyper-parameter, and the hyper-parameters are parameters influencing the operation effect of the precision system; executing forward propagation of a Transform model by using data on the second dimension in the hyper-parameter data to obtain prediction data on the first dimension in the hyper-parameter data; executing back propagation of a Transform model by using the prediction data and the data on the first dimension in the hyper-parameter data so as to obtain gradient information of each dimension in the second dimension; and determining a hyper-parameter space boundary according to the gradient information of each dimension in the second dimensions, wherein the hyper-parameter space boundary is used for setting or optimizing the precision system. According to the embodiment of the invention, the hyper-parameter space boundary can be efficiently, quickly and accurately obtained by utilizing Transform.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular, to a method, apparatus, electronic device, and storage medium for analyzing a hyperparameter space based on Transformer. Background Art

[0002] A precision system refers to a system that integrates advanced technologies and high-precision components and is designed to achieve specific, high-precision functions and tasks. It is increasingly applied in various fields of real life. Generally, for a precision system, there are many factors that affect its operation effect. These factors can be characterized by parameters and their values. Given the large number of types and wide value ranges, they are usually vividly called "superparameters" or "hyperparameters". Then, how to select parameters and set their ranges largely determines the operation effect of the precision system. Given the complexity of the operation environment, system conditions, operation conditions, etc. of the precision system, there are many types of parameters that affect the precision system. How to select parameters and determine the value range, that is, determine the hyperparameter space boundary, has become an urgent problem to be solved.

[0003] Currently, the determination of the hyperparameter space is mostly set based on human experience. However, this method is inefficient on the one hand and highly dependent on expert experience on the other hand, with high costs and difficult to control accuracy. Therefore, it seriously affects the operation effect of the precision system. Summary of the Invention

[0004] In view of this, embodiments of this application are committed to providing a method, apparatus, electronic device, and storage medium for analyzing a hyperparameter space based on Transformer to efficiently and accurately find the hyperparameter space boundary.

[0005] One aspect of this application provides a method for analyzing a hyperparameter space based on Transformer, including:

[0006] Obtain hyperparameter data of a precision system. The hyperparameter data has multiple dimensions, and each dimension represents a hyperparameter, where the hyperparameter is a parameter that affects the operation effect of the precision system;

[0007] Use the data in the second dimension of the hyperparameter data to perform forward propagation of the Transformer model to obtain prediction data in the first dimension of the hyperparameter data;

[0008] Use the prediction data and the data in the first dimension of the hyperparameter data to perform backpropagation of the Transformer model to obtain gradient information for each dimension in the second dimension;

[0009] A hyper-parameter space boundary is determined based on the gradient information of each dimension in the second dimension, and the hyper-parameter space boundary includes a first value range of each dimension in the second dimension. When the data on each dimension in the second dimension is in the first value range, the data on the first dimension has the greatest impact on the value of the data on the first dimension. The hyper-parameter space boundary is used to set or optimize the precision system.

[0010] In some implementations of the first aspect of the present application, it also includes: performing data preprocessing on the hyperparameter data, and the data preprocessing includes one or more of the following: data cleaning, missing value processing, and consistency processing.

[0011] In some embodiments of the first aspect of the present application, the hyperparameter data includes at least one of the following hyperparameters: environmental parameters, component parameters of the precision system, motion parameters and performance parameters of the precision system.

[0012] In some implementations of the first aspect of the present application, it also includes: by constructing a minimum mean square error loss, optimizing the Transformer model using a gradient descent algorithm so that the Transformer model can fit the mapping relationship between the first dimension and each dimension in the second dimension.

[0013] In some embodiments of the first aspect of the present application, the forward propagation of the Transformer model is performed using the data on the second dimension in the hyperparameter data to obtain the predicted data of the first dimension in the hyperparameter data, including: extracting the data on each dimension in the second dimension in the hyperparameter data respectively to form a feature vector of each dimension in the second dimension; inputting the feature vector of each dimension in the second dimension into the encoder of the Transformer model to generate an encoding information matrix; and sending the encoding information matrix into the decoder of the Transformer model to generate an output feature vector, which is the vector representation of the predicted data.

[0014] In some embodiments of the first aspect of the present application, the first dimension includes performance parameters of one or more dimensions in the hyperparameter data, and the second dimension includes all dimensions in the hyperparameter data except the first dimension.

[0015] In some embodiments of the first aspect of the present application, the method of performing back propagation of the Transformer model using the data on the first dimension in the predicted data and the hyperparameter data to obtain gradient information of each dimension in the second dimension includes: calculating the reverse gradient of each layer in the Transformer model using the data on the first dimension in the predicted data and the hyperparameter data, and back-propagating the reverse gradient layer by layer in the Transformer model to the input layer of the Transformer model using the chain rule, so that the gradient of the data on the first dimension to the data on one dimension in the second dimension is obtained through gradient iteration.

[0016] In some embodiments of the first aspect of the present application, determining the hyper-parameter space boundary based on the gradient information of each dimension in the second dimension includes: determining a focus dimension in the second dimension; traversing all of the gradient information, statistically obtaining and storing the gradient information corresponding to the data on the focus dimension; and finding a numerical interval with a maximum gradient absolute value in the data on the focus dimension, wherein the numerical interval is the first value range of the focus dimension.

[0017] In a second aspect of the present application, a Transformer-based hyperparameter spatial analysis device is provided, comprising:

[0018] An acquisition unit, used to acquire hyperparameter data of a precision system, wherein the hyperparameter data has multiple dimensions, each dimension represents a hyperparameter, and the hyperparameter is a parameter that affects the operation effect of the precision system;

[0019] A forward processing unit, configured to perform forward propagation of a Transformer model using data on a second dimension in the hyperparameter data to obtain predicted data on a first dimension in the hyperparameter data;

[0020] A gradient analysis unit, configured to perform back propagation of a Transformer model using the predicted data and the data on the first dimension in the hyperparameter data to obtain gradient information of each dimension in the second dimension;

[0021] A boundary determination unit is used to determine the boundary of the hyper-parameter space according to the gradient information of each dimension in the second dimension, the hyper-parameter space boundary includes a first value range of each dimension in the second dimension, and the data on each dimension in the second dimension has the greatest influence on the value of the data on the first dimension when it is in the first value range. The hyper-parameter space boundary is used to set or optimize the precision system.

[0022] The third aspect of the present application provides an electronic device, comprising: a processor and a memory; wherein the memory is connected to the processor, and the memory is used to store a computer program; the processor is used to implement the above method by running the computer program stored in the memory.

[0023] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above method is implemented.

[0024] According to an embodiment of the present application, the hyperparameter data is fitted through a Tranformer model so that the Tranformer model learns the mapping relationship between each dimension in the second dimension and the first dimension in the hyperparameter data, and uses the mapping relationship learned by the Tranformer model to seek the gradient. Thereafter, the gradient is used to find the value range of each dimension in the second dimension that has the greatest impact on the change in the data value in the first dimension, thereby efficiently, quickly and accurately obtaining the hyperparameter space boundary, and at the same time, the accuracy of the hyperparameter space boundary can be improved. Based on this, a precision system can be set up or optimized without relying on human experience, which can improve efficiency and accuracy and reduce labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 It is a flowchart of a Transformer-based hyperparameter space analysis method provided in an embodiment of the present application;

[0027] Figure 2 is an exemplary implementation flowchart of determining the hyper-parameter space boundary using gradient information according to an embodiment of the present application;

[0028] Figure 3 It is a structural schematic diagram of a Transformer-based hyperparameter spatial analysis device provided in an embodiment of the present application;

[0029] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] Terminology explanation:

[0032] Hyperparameter space: hyperparameter space, but the "hyperparameter" involved in the embodiments of the present application refers to the parameters that affect the operating effect of the precision system, which are vividly called "hyperparameters" or "super parameters" because of their many types and wide range of values. Accordingly, the hyperparameter space, i.e., the hyperparameter space (Hyperparameter Space), refers to the set of all possible hyperparameter values. In the hyperparameter space, each dimension represents a hyperparameter, and different dimensions have different degrees of correlation, and each point represents a specific set of hyperparameter settings. For example, if there are four hyperparameters: wind direction, wind speed, air pressure, and temperature, then the hyperparameter space is a four-dimensional space. The size and complexity of the hyperparameter space depends on the number of hyperparameters and the range of their possible values. In view of the large number of hyperparameter types that affect the operating effect of the precision system, the hyperparameter space is usually a high-dimensional space, and a high-dimensional space or a hyperparameter with a large number of possible values ​​makes it more difficult to search for the optimal hyperparameter configuration, which is the so-called "dimensionality disaster". Therefore, choosing a suitable hyperparameter space is crucial for precision systems.

[0033] Hyperparameter space boundary: the boundary of the hyperparameter space, which refers to a complete set of possible value ranges of the hyperparameters. The boundary of the hyperparameter space has an important impact on the setting, testing, and optimization of precision systems. In the disclosed embodiment, the hyperparameter space boundary refers to the sampling interval where the other dimensions in the hyperparameter space have the greatest impact on a specified dimension.

[0034] Exemplary Methods

[0035] Figure 1 FIG. 1 shows a flow chart of a Transformer-based hyperparameter space analysis method provided in an embodiment of the present disclosure. Figure 1 , the method may include the following steps:

[0036] Step 101, obtaining hyperparameter data of a precision system, where the hyperparameter data has multiple dimensions, each dimension representing a hyperparameter, and the hyperparameter is a parameter that affects the operation effect of the precision system;

[0037] Step 102, using the data on the second dimension in the hyperparameter data to perform forward propagation of the Transformer model to obtain the predicted data of the first dimension in the hyperparameter data;

[0038] Step 103, performing back propagation of the Transformer model using the data on the first dimension in the prediction data and the hyperparameter data to obtain the gradient information of each dimension in the second dimension;

[0039] Step 104, determine the hyper-parameter space boundary based on the gradient information of each dimension in the second dimension, the hyper-parameter space boundary includes the first value range of each dimension in the second dimension, and the data on each dimension in the second dimension has the greatest impact on the value of the data on the first dimension when it is in the first value range. The hyper-parameter space boundary is used to set or optimize the precision system.

[0040] In the disclosed embodiment, the hyperparameter data is a data set of hyperparameters of a precision system, which may include all parameters that affect the operation of the precision system. The precision system may be any type of precision system. For example, aircraft, guided weapons, precision positioning systems, etc. in the aerospace field. Accordingly, the hyperparameters that affect the precision system may include, but are not limited to, environmental parameters, component parameters of the precision system, motion parameters of the precision system, performance parameters, etc.

[0041] Environmental parameters may include parameters of the production or manufacturing environment of precision systems, parameters of the transportation environment, parameters of the operating environment, etc. For example, environmental parameters for manufacturing aircraft include: manufacturing temperature, humidity, manufacturing equipment accuracy, etc. Aircraft operating environment parameters include wind direction, wind speed, air pressure, temperature, space altitude particles, space magnetic field, noise, vibration, etc.

[0042] The number of component parameters of precision systems is even greater. Taking aircraft as an example, they can include frame parameters, power system parameters, control system parameters, etc. Frame parameters can specifically include weight, wheelbase, materials, etc. Power system parameters can include motor parameters (such as nominal no-load KV value, maximum peak current, maximum peak power, etc.), battery parameters (such as capacity, voltage, discharge rate, etc.), propeller parameters (such as pitch, model, chord length, number of blades, safe speed, etc.). Control system parameters include receiver parameters (such as frequency, modulation mode, number of channels, remote control distance), autopilot parameters (such as GPS, IMU, barometer, etc.).

[0043] The motion parameters of a precision system may include speed, attitude, acceleration, etc. Taking an aircraft as an example, they may include take-off speed, cruising speed, maximum speed, glide speed, climb rate, turning radius, pitch angle, yaw angle, roll angle, airflow angle, angular velocity, acceleration, etc.

[0044] Taking an aircraft as an example, performance parameters may include: ceiling, range, endurance, activity radius, maneuverability parameters, take-off performance parameters, landing performance parameters, stability parameters, target hitting accuracy, etc.

[0045] As can be seen from the above description, there are usually a large number of types of hyperparameters that affect the operation of precision systems.

[0046] In some embodiments, if there are N hyperparameters of a precision system, and N is an integer greater than 1, the hyperparameter data of the model is N-dimensional data, and the hyperparameter space corresponding to the hyperparameter data is N-dimensional space. If the model has thousands or tens of thousands of hyperparameters, its hyperparameter data is high-dimensional data of thousands or ten thousand dimensions, and the corresponding hyperparameter space is a high-dimensional space such as thousands of dimensions or ten thousand dimensions. Exemplarily, the hyperparameter data can be represented as multidimensional feature data (X1, X2, ..., Xn), where n represents the total number of dimensions of the hyperparameter data, and n is an integer greater than 1. Xi (i = 1, 2, ..., n) represents the data on dimension i, that is, the possible values ​​of hyperparameter i.

[0047] In step 101, the hyper-parameter data may be input by a staff member or may come from other electronic devices. The specific method of obtaining the hyper-parameter data is not limited in the embodiments of the present disclosure.

[0048] In some implementations, before step 102, the following may also be included: performing data preprocessing on the hyperparameter data, where the data preprocessing includes one or more of the following: data cleaning and normalization. Data preprocessing can effectively improve the data quality of the hyperparameter data, reduce the risk of errors, and provide a reliable basis for subsequent processing.

[0049] In some examples, data cleaning can include checking, correcting, and screening the hyperparameter data to ensure the accuracy and completeness of the first hyperparameter data.

[0050] In some examples, the normalization process may include: unifying the units and scales of data of different dimensions in the hyperparameter data. By normalizing the hyperparameter data, the dimensional differences between different features may be eliminated.

[0051] In other implementations, data preprocessing may also include: inconsistency processing, missing value processing, etc. The specific process and implementation of data preprocessing are not limited in the embodiments of the present disclosure.

[0052] Before step 102, the method of the embodiment of the present application may further include: a training step of a Transformer model. Specifically, the training step of the Transformer model may include but is not limited to parameter initialization, hyperparameter selection, training and testing of the Transformer model. Among them, parameter initialization and hyperparameter selection of the Transformer model are optional operations.

[0053] Specifically, before step 102, the method may further include: by constructing a minimum mean square error loss, optimizing the Transformer model using a gradient descent algorithm so that the Transformer model can fit the mapping relationship between the first dimension and each dimension in the second dimension. Specifically, the constructed data set is loaded into a training environment, the Transformer model is trained using the loaded data set, and the model parameters of the Transformer model are continuously adjusted by the gradient descent algorithm to minimize the loss value of the loss function until convergence. After the training is completed, the model parameters of the Transformer model are saved as a file and the file is saved in a specified path. In this way, in step 102, only the file under the specified path needs to be loaded and the Transformer model needs to be run to perform forward propagation for the hyperparameter data.

[0054] The dataset used to train the Transformer model can be obtained from a reliable source to ensure the quality of the data in the dataset and the accuracy of the labels. Before training the Transformer model, the data in the dataset can be preprocessed to further improve the instructions of the data in the dataset.

[0055] After training, the Transformer model can characterize the mapping relationship between the first dimension and the second dimension in the hyperparameter space corresponding to the hyperparameter data. For example, if the hyperparameter data has n dimensions, the first dimension is selected as the third dimension of the hyperparameter data, and the first, second, and fourth to nth dimensions of the hyperparameter data all belong to the second dimension, then the Transformer model can characterize the mapping relationship between the third dimension in the hyperparameter space and the other dimensions in the second dimension, that is, the Transformer model can use the data in the first, second, and fourth to nth dimensions of the hyperparameter space as input features, and output the predicted value of the data in the first dimension in the hyperparameter space.

[0056] The first dimension may include one or more dimensions in the hyperparameter data, and the second dimension may include all dimensions in the hyperparameter except the first dimension. For example, if the hyperparameter data has n dimensions, and the first dimension is selected as the third dimension of the hyperparameter data, then the first dimension, the second dimension, and the fourth to the nth dimension of the hyperparameter data all belong to the second dimension.

[0057] For precision systems, one or more performance parameters of the precision system in the hyperparameter data can be used as the first dimension, and the remaining parameters can be used as the second dimension, so that the value range of other parameters that have the greatest impact on the change of performance parameters (referred to as the first value range of the second dimension in the embodiment of the present application) can be analyzed as the hyperparameter space boundary through the method provided in the embodiment of the present application. Taking an aircraft as an example, one or some specific performance parameters of the aircraft can be used as the first dimension, and other parameters such as various environmental parameters, motion parameters, component parameters, etc. of the aircraft can be used as the second dimension, so as to analyze the value range of other parameters that have the greatest impact on one or some specific performance changes.

[0058] By using the embodiments of the present invention and using the Transformer model to model the hyperparameter space, discrete hyperparameter data can be converted into continuous hyperparameter data, so that the granularity of the hyperparameter space is refined, making it easier to accurately find the hyperparameter data boundaries with higher precision through subsequent steps.

[0059] In step 102, a forward propagation process is performed, that is, the data on each dimension in the second dimension is used as input features through the encoder and decoder of the Transformer model layer by layer to generate predicted data of the first dimension.

[0060] In some implementations, the exemplary implementation process of step 102 may include the following steps a1 to a3:

[0061] Step a1, extracting data on each dimension in the second dimension of the hyper-parameter data to form a feature vector of each dimension in the second dimension;

[0062] Step a2, inputting the feature vector of each dimension in the second dimension into the encoder of the Transformer model to generate a coding information matrix;

[0063] Step a3, sending the encoded information matrix to the decoder of the Transformer model to generate an output feature vector, which is a vector representation of the predicted data of the first dimension in the hyperparameter data.

[0064] The Transformer model of the disclosed embodiment may include N encoders (Encoder block) and N decoders (Decoder block), where N is an integer greater than or equal to 1. A multi-head attention unit (Multi-Head Attention) and a feedforward neural network (Feed Forward) are provided in each encoder and decoder, and a residual normalization (Add&Norm) layer is provided above each multi-head attention unit.

[0065] The encoder contains one multi-head attention unit. The decoder contains two multi-head attention units, one of which uses a mask. In the Add&Norm layer, Add represents the residual connection, which is used to prevent network degradation, and Norm represents the layer normalization, which is used to normalize the activation values ​​of each layer.

[0066] The multi-head attention unit is composed of multiple self-attention units (Self-Attention). The query matrix Q, key value matrix K and value V are required for calculation of the self-attention unit. In the embodiment of the present disclosure, the query matrix Q, key value matrix K and value V required by the self-attention unit are obtained by linear transformation based on the data on each dimension in the second dimension of the hyperparameter data or the encoding information matrix output by the previous encoder.

[0067] In some implementations, assuming that the feature vectors in each dimension of the second dimension in the hyperparameter data or the encoding information matrix output by the previous encoder is a matrix X, the linear transformation matrix W can be used. Q、 W K , W V Multiplying by matrix X gives the query matrix Q, key-value matrix K, and value V required by the self-attention unit.

[0068] It should be noted that the architecture of the Tranformer model used in the embodiment of the present disclosure is not limited to the above-mentioned method. The embodiment of the present disclosure does not limit the specific structure of the Tranformer model.

[0069] Taking into account that the multi-dimensional hyper-parameter data often has complex internal data relationships, and different dimensions often have a certain correlation with each other, therefore, in order to solve the problem that the existing correlation method cannot model nonlinear complex features, the Transformer model is used in the embodiment of the present disclosure to model the multi-dimensional hyper-parameter data. Since the Transformer model is built based on the self-attention mechanism, the self-attention mechanism can enable the Transformer model to effectively capture the dependencies in the input features. This mechanism also determines that the Transformer model can better pay attention to the correlation between features. Therefore, the Transformer model can more accurately characterize the complex correlations between different dimensions in the hyper-parameter data, thereby determining which values ​​of the data in each dimension in the second dimension have a greater impact on the numerical changes of the data in the first dimension by calculating the gradient of the input features relative to the prediction results, thereby achieving the purpose of accurately locating the boundaries of the hyper-parameter space.

[0070] In step 103, it is difficult to directly obtain the gradient of the output feature of a complex Transformer model with respect to the input feature, so the embodiment of the present disclosure introduces a reverse chain derivation technique. That is, in some implementations, step 103 may include: using a reverse chain to derive the gradient of the output of the Transformer model relative to the input, which is the gradient of the data in the first dimension relative to the other dimensions in the second range.

[0071] Specifically, step 103 may include: using the data on the first dimension in the prediction data and the hyperparameter data to calculate the reverse gradient of each layer in the Transformer model, and using the chain rule to propagate the reverse gradient layer by layer in the Transformer model to the input layer of the Transformer model, so that the gradient of the data on the first dimension to the data on one dimension in the second dimension is obtained through gradient iteration. Among them, the nonlinear function part in the Transformer model is replaced by linear function approximation fitting. In this way, the gradient of the data on the first dimension to the data on each dimension in the second dimension can be obtained, that is, the gradient information is obtained.

[0072] Considering that the gradient will be distributed at various positions according to the degree of influence of the input in the Transformer model during the back-propagation process, the gradient information can be used to analyze the criticality and sensitivity of the input, so as to determine the impact of the specific input variable on the output, that is, to find the input interval corresponding to the fastest output change. It can be seen that the method of step 104 can efficiently, quickly and accurately find the first value range of each dimension in the second dimension.

[0073] In step 104, the boundary of the hyperparameter space can be expressed as Xi∈[a,b], where Xi represents the first value range of dimension i, and dimension i belongs to any dimension in the second dimension. When the data value on dimension i is in the corresponding range of Xi, it has the greatest impact on the numerical change of the data on the first dimension. a is the lower limit of the first value range, and b is the upper limit of the first value range.

[0074] There are many ways to use gradient information to determine the hyper-parameter space boundary in step 104. Figure 2 FIG. 1 shows an exemplary implementation flow chart of determining the hyper-parameter space boundary using gradient information in step 104. Figure 2 In some implementations, the exemplary implementation process in step 104 may include the following steps:

[0075] Step 201, determining a focus dimension in the second dimension;

[0076] Step 202, traverse all gradient information, obtain and store the gradient information corresponding to the data on the dimension of interest;

[0077] Step 203, in the data on the dimension of interest, find a numerical interval with a maximum absolute value of the gradient, and the numerical interval is the first value range of the dimension of interest.

[0078] The larger the absolute value of the gradient, the more sensitive the output of the Transformer model is to its changes. That is, within the numerical range of the data on the dimension of interest corresponding to the maximum absolute value of the gradient, the value of the data on the dimension of interest has the greatest impact on the change in the value of the data on the first dimension.

[0079] In practical applications, it only takes seconds to traverse tens of thousands of data in step 104 using the above method. Therefore, using gradient information can efficiently, quickly and accurately solve the value range of each dimension on the second dimension that has the greatest impact on the value change of the data on the first dimension, that is, using gradient information can obtain the first value range of each dimension on the second dimension, and this value range is the boundary of the hyperparameter space.

[0080] The disclosed embodiment performs gradient analysis through Tranformer, that is, hyper-parameter data is fitted through Tranformer model so that the Tranformer model learns the mapping relationship between each dimension in the second dimension and the first dimension in the hyper-parameter data, and uses the mapping relationship learned by the Tranformer model to seek the gradient, and then uses the gradient to find the value range of each dimension in the second dimension that has the greatest impact on the change of data value in the first dimension, so as to efficiently, quickly and accurately obtain the hyper-parameter space boundary, and at the same time improve the accuracy of the hyper-parameter space boundary.

[0081] After determining the boundary of the hyperparameter space, the precision system can be set or optimized based on the first value range of each hyperparameter of the second dimension. For example, through the above process, it is determined that the wind speed between 13 and 20 (m / s) has the greatest impact on the accuracy of the aircraft capturing the target, which means that the accuracy of the aircraft capturing the target is most sensitive to the wind speed in this interval. The aircraft can be further set or optimized accordingly. For example, the weight of the wind speed parameter within this interval can be increased.

[0082] Exemplary Devices

[0083] Figure 3 FIG. 1 shows a schematic diagram of the structure of a Transformer-based hyperparameter spatial analysis device provided in an embodiment of the present application. Figure 3 , the Transformer-based hyperparameter spatial analysis device 300 may include:

[0084] An acquisition unit 301 is used to acquire hyperparameter data of a precision system, where the hyperparameter data has multiple dimensions, each dimension represents a hyperparameter, and the hyperparameter is a parameter that affects the operation effect of the precision system;

[0085] A forward processing unit 302 is used to perform forward propagation of the Transformer model using the data on the second dimension in the hyperparameter data to obtain predicted data of the first dimension in the hyperparameter data;

[0086] A gradient analysis unit 303 is used to perform back propagation of the Transformer model using the data on the first dimension in the prediction data and the hyperparameter data to obtain gradient information of each dimension in the second dimension;

[0087] The boundary determination unit 304 is used to determine the boundary of the hyper-parameter space according to the gradient information of each dimension in the second dimension. The boundary of the hyper-parameter space includes the first value range of each dimension in the second dimension. When the data on each dimension in the second dimension is in the first value range, the value of the data on the first dimension has the greatest impact, and is used to set or optimize the precision system.

[0088] In some implementations, the Transformer-based hyperparameter space analysis device 300 may further include: a preprocessing unit 305 for performing data preprocessing on the hyperparameter data, where the data preprocessing includes one or more of the following: data cleaning, missing value processing, and consistency processing.

[0089] In some embodiments, the Transformer-based hyperparameter space analysis device 300 may also include: a model training unit 306, which is used to optimize the Transformer model by constructing a minimum mean square error loss and using a gradient descent algorithm so that the Transformer model can fit the mapping relationship between the first dimension and each dimension in the second dimension.

[0090] In some embodiments, the forward processing unit 302 can be specifically used to: extract data on each dimension in the second dimension of the hyperparameter data to form a feature vector for each dimension in the second dimension; input the feature vector of each dimension in the second dimension into the encoder of the Transformer model to generate an encoding information matrix; and, send the encoding information matrix into the decoder of the Transformer model to generate an output feature vector, which is a vector representation of the predicted data.

[0091] In some embodiments, the gradient analysis unit 303 can be specifically used to: calculate the reverse gradient of each layer in the Transformer model using the data on the first dimension in the prediction data and the hyperparameter data, and use the chain rule to backpropagate the reverse gradient layer by layer in the Transformer model to the input layer of the Transformer model, so that the gradient of the data on the first dimension with respect to the data on one dimension in the second dimension is obtained through gradient iteration.

[0092] In some implementations, the boundary determination unit 304 can be specifically used to: determine a focus dimension in the second dimension; traverse all gradient information, statistically obtain and store the gradient information corresponding to the data on the focus dimension; and find a numerical interval with a maximum gradient absolute value in the data on the focus dimension, where the numerical interval is the first value range of the focus dimension.

[0093] Other technical details of the Transformer-based hyperparameter spatial analysis device 300 in the embodiment of the present application can be found in the previous method section and will not be repeated here.

[0094] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0095] Electronic devices

[0096] Figure 4 FIG. 1 shows an example diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 4 As shown, the electronic device may include: one or more processors 401, and a memory 402 storing one or more programs, which are executed by the one or more processors 401 to implement the method flow shown in the above embodiments of the present disclosure and / or program units corresponding to each unit in the device.

[0097] The various components are interconnected using different buses and can be mounted on a common motherboard or otherwise as needed. The processor 401 can process instructions executed within the electronic device, including instructions stored in or on the memory to display graphical information of a user interface on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memories if desired.

[0098] The processor 401 may include one or more single-core processors or multi-core processors. The processor 401 may include any combination of general-purpose processors or dedicated processors (such as image processors, application processors, baseband processors, etc.).

[0099] The memory 402 is a computer-readable storage medium provided by the present disclosure, which can be used to store non-transient software programs, non-transient computer executable programs and units, such as the following in the embodiments of the present disclosure: Figure 1 The processor 401 executes the non-transient software programs, instructions and units stored in the memory 402, thereby executing the above method embodiments. Figure 1 The methods shown correspond to programs, instructions, and units.

[0100] The electronic device may further include: an input device 403 and an output device 404. The processor 401, the memory 402, the input device 403 and the output device 404 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0101] The input device 403 can receive input digital or character information, and generate signal input related to user settings and function control of the camera calibration device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, and other input devices. The output device 404 may include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0102] The above-mentioned programs (also referred to as software, software applications, or codes) include machine instructions for programmable processors, and these computer programs can be implemented using object-oriented programming languages, assembly or machine languages.

[0103] With the development of time and technology, the meaning of medium is becoming more and more extensive, and the propagation path of computer programs is no longer limited to tangible media, and can also be downloaded directly from the network, etc. Any combination of one or more computer-readable storage media can be used. Computer-readable storage media can be used but not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing programs, which can be used by or in combination with instruction execution systems, devices or devices.

[0104] In one implementation, the electronic device may be, but is not limited to, electronic devices with relatively high power consumption requirements, such as laptop computers and desktop computers commonly used by ARM chip system developers.

[0105] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. The program includes instructions. When the instructions are executed by one or more processors of a computing device, the steps of the method described in any one of the aforementioned method embodiments are executed.

[0106] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the steps of any one of the methods described in the aforementioned method embodiments.

[0107] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A Transformer-based hyperparameter space analysis method, comprising: Obtaining hyperparameter data of a precision system, wherein the hyperparameter data has multiple dimensions, each dimension representing a hyperparameter, and the hyperparameter is a parameter that affects the operation effect of the precision system; Perform forward propagation of the Transformer model using the data on the second dimension in the hyperparameter data to obtain predicted data of the first dimension in the hyperparameter data; Performing back propagation of a Transformer model using the predicted data and the data on the first dimension in the hyperparameter data to obtain gradient information of each dimension in the second dimension; A hyper-parameter space boundary is determined based on the gradient information of each dimension in the second dimension, and the hyper-parameter space boundary includes a first value range of each dimension in the second dimension. When the data on each dimension in the second dimension is in the first value range, the data on the first dimension has the greatest impact on the value of the data on the first dimension. The hyper-parameter space boundary is used to set or optimize the precision system.

2. The method according to claim 1, characterized in that The hyperparameter data includes at least one of the following hyperparameters: environmental parameters, component parameters of the precision system, motion parameters and performance parameters of the precision system.

3. The method according to claim 1, characterized in that Also includes: By constructing a minimum mean square error loss, the Transformer model is optimized using a gradient descent algorithm so that the Transformer model can fit the mapping relationship between the first dimension and each dimension in the second dimension.

4. The method according to claim 1, characterized in that: Executing forward propagation of a Transformer model using data on a second dimension in the hyperparameter data to obtain predicted data on a first dimension in the hyperparameter data includes: Respectively extracting data on each dimension in the second dimension of the hyperparameter data to form a feature vector of each dimension in the second dimension; Inputting the feature vector of each dimension in the second dimension into the encoder of the Transformer model to generate an encoding information matrix; The encoded information matrix is ​​fed into the decoder of the Transformer model to generate an output feature vector, which is the vector representation of the predicted data.

5. The method according to claim 2, characterized in that: The first dimension includes performance parameters of one or more dimensions in the hyperparameter data, and the second dimension includes all dimensions in the hyperparameter data except the first dimension.

6. The method according to claim 1, characterized in that The performing back propagation of the Transformer model using the predicted data and the data on the first dimension in the hyperparameter data to obtain gradient information of each dimension in the second dimension includes: The reverse gradient of each layer in the Transformer model is calculated using the predicted data and the data on the first dimension of the hyperparameter data, and the reverse gradient is back-propagated layer by layer in the Transformer model to the input layer of the Transformer model using the chain rule. In this way, the gradient of the data on the first dimension with respect to the data on one dimension in the second dimension is obtained through gradient iteration.

7. The method according to claim 6, characterized in that The step of determining the hyper-parameter space boundary according to the gradient information of each dimension in the second dimension includes: Determine the dimension of interest in the second dimension; Traversing all the gradient information, obtaining and storing the gradient information corresponding to the data on the dimension of interest; A numerical interval with a maximum gradient absolute value is found in the data on the focus dimension, and the numerical interval is the first value range of the focus dimension.

8. A hyperparameter space analysis device based on Transformer, characterized in that: include: An acquisition unit, used to acquire hyperparameter data of a precision system, wherein the hyperparameter data has multiple dimensions, each dimension represents a hyperparameter, and the hyperparameter is a parameter that affects the operation effect of the precision system; A forward processing unit, configured to perform forward propagation of a Transformer model using data on a second dimension in the hyperparameter data to obtain predicted data on a first dimension in the hyperparameter data; A gradient analysis unit, configured to perform back propagation of a Transformer model using the predicted data and the data on the first dimension in the hyperparameter data to obtain gradient information of each dimension in the second dimension; A boundary determination unit is used to determine the boundary of the hyper-parameter space according to the gradient information of each dimension in the second dimension, the hyper-parameter space boundary includes a first value range of each dimension in the second dimension, and the data on each dimension in the second dimension has the greatest impact on the value of the data on the first dimension when it is in the first value range. The hyper-parameter space boundary is used to set or optimize the precision system.

9. An electronic device, characterized in that: include: Processor and memory; Wherein, the memory is connected to the processor, and the memory is used to store a computer program; The processor is configured to implement the method according to any one of claims 1 to 7 by running the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images