Training method and device of neural network model, equipment and storage medium

Abstract

The application provides a neural network model training method and device, equipment and a storage medium. In the method, a hyperparameter set generated by a preset search strategy is used to update the configuration of an initial neural network model to obtain a first neural network model. The hyperparameter set includes state control parameters of multiple basic structures, and the state control parameters are used to indicate the structure state of the basic structures during model training. The first neural network model is trained using preset training data. After the model training is completed, the basic structures with a first state or a second state in the first neural network model are converted into basic structures with a third state to obtain a second neural network model. The second neural network model is a candidate neural network model suitable for a target processing task. By converting the states of part of the basic structures after the training is completed, the model structure is simplified, and the operation speed and accuracy of the model during inference are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, device and storage medium for a neural network model. Background Art

[0002] With the continuous advancement of terminal technology, users have higher requirements for real-time performance and imaging quality when taking photos on their mobile phones. On the one hand, they require low image output latency, and on the other hand, they demand high image quality. This places higher demands on the image processing models deployed on mobile phones. Different model architectures affect the quality and speed of image output. How to design and train more efficient image processing models is currently a pressing issue. Summary of the Invention

[0003] The embodiments of the present application provide a training method, apparatus, device, and storage medium for a neural network model to train a more efficient and accurate target model.

[0004] In a first aspect, an embodiment of the present application proposes a training method for a neural network model, the method comprising: obtaining an initial neural network model suitable for a target processing task, the initial neural network model comprising a plurality of basic structures having a preset connection relationship; updating the configuration of the initial neural network model based on a set of hyperparameters generated based on a preset search strategy to obtain a first neural network model; the hyperparameter set comprising state control parameters for the plurality of basic structures, the state control parameters being used to indicate the structural state of the basic structures during model training; performing model training on the first neural network model using preset training data; after the model training is completed, converting the basic structures in the first neural network model whose structural state is the first state or the second state into basic structures whose structural state is the third state to obtain a second neural network model, the second neural network model being a candidate neural network model suitable for the target processing task.

[0005] In this approach, before model training, the initial neural network model configuration is updated based on a set of hyperparameters to expand the model structure during training, improve the model's training results, and ultimately achieve a higher-performance target model. After model training, the model's infrastructure is converted from the first or second state to the third state, simplifying the model structure and improving the model's computational speed and accuracy during inference.

[0006] In an optional embodiment of the first aspect, the hyperparameter set further includes: channel numbers of the multiple basic structures and structure indexes of the multiple basic structures, where the structure indexes are used to indicate structure information of the basic structures.

[0007] Exemplarily, the hyperparameter set includes a plurality of arrays, which can refer to Table 1, and different arrays indicate different information of the model base structure, for example, indicating the number of channels of the base structure, the structure index of the base structure, the state control parameter, and the like.

[0008] In the above scheme, the hyperparameter set is generated based on a preset search strategy, and the application does not limit the search strategy.

[0009] In an optional embodiment of the first aspect, the hyperparameter set generated based on the preset search strategy updates the configuration of the initial neural network model, including: if the original number of channels of the first base structure in the initial neural network model is inconsistent with the number of channels of the first base structure indicated in the hyperparameter set, updating the number of channels of the first base structure to the number of channels of the first base structure indicated in the hyperparameter set; the first base structure is any one of the plurality of base structures.

[0010] The above scheme shows that the number of channels of part of the base structures in the initial neural network model is updated, and the model performance is optimized by changing the number of channels.

[0011] In an optional embodiment of the first aspect, the hyperparameter set generated based on the preset search strategy updates the configuration of the initial neural network model, including: if the original structure information of the second base structure in the initial neural network model is inconsistent with the structure information of the second base structure indicated in the hyperparameter set, updating the second base structure to the structure of the second base structure indicated in the hyperparameter set; the second base structure is any one of the plurality of base structures.

[0012] The above scheme shows that the structure information of part of the base structures in the initial neural network model is updated, for example, replacing a single-branch structure with a multi-branch structure, and the model performance is optimized by changing the structure information.

[0013] In an optional embodiment of the first aspect, the number of branches of the updated second base structure is greater than the original number of branches of the second base structure, or the number of layers of the updated second base structure is greater than the original number of layers of the second base structure.

[0014] The above scheme shows two examples of updating the base structure in the model, by increasing the number of layers or branches of the base structure, although the model structure becomes complex, the number of learnable parameters increases accordingly, but based on the updated model structure, the training effect of the model is improved.

[0015] In an optional embodiment of the first aspect, the configuration of the initial neural network model is updated based on a hyperparameter set generated by a preset search strategy, including: if the original structural information of the third basic structure in the initial neural network model is a single-branch structure, the structural information of the third basic structure indicated in the hyperparameter set is a multi-branch structure, and the structural state of the third basic structure indicated in the hyperparameter set is the first state, updating the third basic structure to a multi-branch structure, and configuring learnable weight parameters for the convolutional layers in each branch of the third basic structure; the third basic structure is any one of the multiple basic structures.

[0016] The above scheme shows that part of the basic structure in the initial neural network model is updated to a multi-branch structure, and weight parameters are configured for the convolutional layer in the multi-branch structure to optimize the model performance.

[0017] In an optional embodiment of the first aspect, based on a hyperparameter set generated by a preset search strategy, the configuration of the initial neural network model is updated, including: if the original structural information of the fourth basic structure in the initial neural network model is a single-branch structure, the structural information of the fourth basic structure indicated in the hyperparameter set is a multi-branch structure, and the structural state of the fourth basic structure indicated in the hyperparameter set is the second state, a learnable weight parameter is configured for the convolutional layer in each branch of the multi-branch structure, and a learnable weight parameter is configured for each channel of the convolutional layer in each branch of the multi-branch structure; based on the multi-branch structure, the learnable weight parameters of the convolutional layer in each branch of the multi-branch structure, and the learnable weight parameters of each channel of the convolutional layer in each branch, weight merging information is generated; based on the weight merging information, the weight configuration of the fourth basic structure is updated, and the updated fourth basic structure is still a single-branch structure; the fourth basic structure is any one of the multiple basic structures.

[0018] The above scheme shows the update of the weight configuration of some basic structures in the initial neural network model. The weight parameters of the basic structure are no longer single-dimensional weight parameters, but contain multi-dimensional weight parameters. During model training, for a certain basic structure, the weight parameters of the basic structure are first calculated, and then the operation is performed with the input of the basic structure, which can improve the speed of model training.

[0019] In an optional embodiment of the first aspect, based on a hyperparameter set generated by a preset search strategy, the configuration of the initial neural network model is updated, including: if the fifth basic structure in the initial neural network model is consistent with the structure of the fifth basic structure indicated in the hyperparameter set, the fifth basic structure is a multi-branch structure, and the structural state of the fifth basic structure indicated in the hyperparameter set is the second state, a learnable weight parameter is configured for the convolutional layer in each branch of the multi-branch structure, and a learnable weight parameter is configured for each channel of the convolutional layer in each branch of the multi-branch structure; based on the multi-branch structure, the learnable weight parameters of the convolutional layer in each branch of the multi-branch structure, and the learnable weight parameters of each channel of the convolutional layer in each branch, weight merging information is generated; based on the weight merging information, the weight configuration of the fifth basic structure is updated, and the updated fifth basic structure is a single-branch structure; the fifth basic structure is any one of the multiple basic structures.

[0020] The above scheme shows that part of the basic structure in the updated initial neural network model is a single-branch structure. The weight configuration of the single-branch structure is generated based on the weight configuration of the multi-branch structure indicated by the hyperparameter set. The weight parameters of the single-branch structure are no longer single-dimensional weight parameters, but contain multi-dimensional weight parameters. During model training, for the updated single-branch structure, the weight parameter operation of the single-branch structure is performed first, and then the operation is performed with the input of the branch structure, which can improve the speed of model training.

[0021] In a second aspect, an embodiment of the present application provides a training method for a neural network model, comprising: obtaining an initial neural network model suitable for a target processing task, the initial neural network model comprising a plurality of basic structures having a preset connection relationship; updating the configuration of the initial neural network model based on a plurality of hyperparameter sets generated based on a preset search strategy to obtain a plurality of updated neural network models; the hyperparameter sets comprise state control parameters for the plurality of basic structures, the state control parameters being used to indicate the structural state of the basic structures during model training; performing model training on the plurality of updated neural network models respectively using the same training data to obtain a plurality of candidate neural network models; selecting a neural network model having a model loss value less than a first threshold value, and / or a number of floating-point operations less than a second threshold value from the plurality of candidate neural network models as a target neural network model.

[0022] In the above scheme, before model training, the configuration of the initial neural network model is updated based on the hyperparameter set to expand the model structure during training and improve the model training effect, so that the target model that meets the performance requirements can be screened out from the multiple models searched and trained.

[0023] In a third aspect, an embodiment of the present application provides a training device for a neural network model, comprising: an acquisition module for acquiring an initial neural network model suitable for a target processing task, the initial neural network model comprising a plurality of basic structures having a preset connection relationship; a processing module for updating the configuration of the initial neural network model based on a set of hyperparameters generated by a preset search strategy to obtain a first neural network model; the hyperparameter set comprises state control parameters for the plurality of basic structures, the state control parameters being used to indicate the structural state of the basic structure during model training; the first neural network model is trained using preset training data; after the model training is completed, the basic structure in the first neural network model whose structural state is the first state or the second state is converted into a basic structure whose structural state is the third state to obtain a second neural network model, the second neural network model being a candidate neural network model suitable for the target processing task.

[0024] In a fourth aspect, an embodiment of the present application provides a training device for a neural network model, comprising: an acquisition module for acquiring an initial neural network model suitable for a target processing task, the initial neural network model comprising a plurality of basic structures having a preset connection relationship; a processing module for updating the configuration of the initial neural network model based on a plurality of hyperparameter sets generated by a preset search strategy, to obtain a plurality of updated neural network models; the hyperparameter sets comprise state control parameters for a plurality of basic structures, the state control parameters being used to indicate the structural state of the basic structures during model training; using the same training data, model training is performed on the plurality of updated neural network models respectively to obtain a plurality of candidate neural network models; a neural network model having a model loss value less than a first threshold value, and / or a number of floating-point operations less than a second threshold value, is selected from the plurality of candidate neural network models as the target neural network model.

[0025] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the electronic device executes any method as in the first aspect.

[0026] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the method as described in any one of the first aspects.

[0027] In a seventh aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the electronic device executes any method as described in the first aspect.

[0028] In an eighth aspect, an embodiment of the present application provides a chip, the chip including a processor, the processor being used to call a computer program in a memory to execute any method as described in the first aspect.

[0029] It should be understood that the third to eighth aspects of the present application correspond to the technical solutions of the first or second aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding optional embodiments are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of several basic structures in the neural network model provided in the embodiments of the present application;

[0031] Figure 2 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0032] Figure 3 A schematic diagram of another application scenario provided by an embodiment of the present application;

[0033] Figure 4 A schematic diagram of a system architecture provided in an embodiment of the present application;

[0034] Figure 5 A schematic diagram of a training method for a neural network model provided in an embodiment of the present application;

[0035] Figure 6 A schematic diagram of the structure of an image classification model provided in an embodiment of the present application;

[0036] Figure 7 A schematic diagram of an infrastructure search space provided in an embodiment of the present application;

[0037] Figure 8 A schematic diagram of the structure of the infrastructure in the first state provided by an embodiment of the present application;

[0038] Figure 9 A schematic diagram of the structure of the infrastructure in the second state provided by an embodiment of the present application;

[0039] Figure 10 A schematic diagram of the structure of the infrastructure in the third state provided by an embodiment of the present application;

[0040] Figure 11 A schematic diagram of the entire process state of the basic structure in a neural network model provided in an embodiment of the present application;

[0041] Figure 12 A schematic diagram of another neural network model training method provided in an embodiment of the present application;

[0042] Figure 13A schematic diagram of the structure of a training device for a neural network model provided in an embodiment of the present application;

[0043] Figure 14 A schematic diagram of the structure of another neural network model training device provided in an embodiment of the present application;

[0044] Figure 15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] To facilitate understanding, the following first introduces the professional terms and concepts involved in the embodiments of this application:

[0046] 1) Neural Architecture Search (NAS): It is a new method for automatically learning network structures to reduce the heavy network design costs.

[0047] In an embodiment of the present application, a neural network architecture search is performed to obtain multiple neural network architectures (or neural network models). Multiple neural network architectures can be traversed to train each neural network architecture to obtain a target neural network architecture with good performance. Good performance means, for example, that the network has a faster processing speed and more accurate processing results.

[0048] 2) Search Space: This defines all optional structures and operations in the network. Optional structures include, for example, convolutional layers and pooling layers, and operations include, for example, convolution, pooling, addition, and concatenation. Different hyperparameters can be selected based on the different operations, such as the number of convolution kernels, their sizes, and stride. Generally speaking, a good search space should be able to eliminate human bias and be flexible enough to cover a wide range of model architectures.

[0049] In the embodiment of the present application, the optional structure defined in the search space can also be described as a basic structure. The basic structure can be a convolution layer, such as Figure 1 As shown in (a), it can also be a pooling layer, such as Figure 1 As shown in (b), there can also be multiple convolutional layers, and multiple convolutional layers can be in one branch, such as Figure 1 The two convolutional layers shown in (c) are in one branch, or multiple convolutional layers are in different branches, such as Figure 1 The two convolutional layers shown in (d) are in two branches. The basic structure can also be a multi-branch structure with a more complex layout, such as Figure 1 As shown in (e), the basic structure has at least three branches, branch 1 has two convolutional layers, branch 2 has one convolutional layer and the first pooling layer, and branch 3 has one pooling layer. It should be noted that Figure 1The multiple infrastructure structures shown are only examples. The embodiments of the present application do not specifically limit the infrastructure of the search space. Researchers can expand the style of the infrastructure in the search space according to actual needs.

[0050] 3) Search Strategy: After defining the search space, the search strategy guides the search for high-performance model architectures. On the one hand, we want to quickly find a high-performing architecture, while on the other hand, we need to avoid premature convergence to a suboptimal architecture. Search strategies include reinforcement learning, evolutionary algorithms, surrogate models, and gradient-based methods.

[0051] In the embodiments of the present application, the search strategy is not specifically limited. A hyperparameter set can be generated through the search strategy. Based on the hyperparameter set, the configuration parameters of each infrastructure in the neural network model (such as the initial weight parameters of the infrastructure) can be determined to facilitate subsequent model training, thereby obtaining the optimized weight parameters of the infrastructure. For details about the hyperparameter set, please refer to the embodiments below and will not be elaborated on here.

[0052] 4) Convolutional layer: This is the fundamental operation in convolutional neural networks and can include multiple convolution operators, also known as convolution kernels. In image processing, a convolution layer acts as a filter that extracts specific information from the input image matrix. A convolution operator is essentially a predefined weight matrix. During the convolution operation, the weight matrix is ​​typically applied horizontally across the input image with a preset stride (stride value) to extract features from the input image.

[0053] 5) Pooling layer: It extracts the key information of a certain area. It usually appears after the convolution layer and plays the role of reducing the number of features output by the convolution layer, thereby reducing model parameters and improving overfitting. In the image processing process, the purpose of the pooling layer is to sample the input image to obtain an image of smaller size. The pooling layer can include an average pooling operator and / or a maximum pooling operator. The average pooling operator can calculate the pixel values ​​in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator can take the pixel with the largest value within a specific range as the result of maximum pooling.

[0054] The solutions of the embodiments of the present application can be applied to many specific fields in the field of artificial intelligence, such as smart manufacturing, smart transportation, smart home, smart security, autonomous driving, etc. The embodiments of the present application can be specifically applied to fields such as image classification, image retrieval, image semantic segmentation, and image super-resolution.

[0055] A possible application scenario: image classification scenario, the training method provided in the embodiment of this application can be used to train a neural network model for album image classification. For example, Figure 2 As shown, through neural network model search, multiple neural network models for album image classification are configured, and the same training data is used to train these multiple neural network models. The training data includes multiple groups of training data, each group of training data includes sample images and annotation information of the sample images, and the annotation information is used to indicate the image category. Based on the training effect, a target neural network model whose performance meets the preset requirements is selected from multiple neural network models for image classification in practical applications. Meeting the preset requirements includes, for example, that the model loss value is less than a first threshold, and / or that the number of model floating-point operations is less than a second threshold. In one example, when a user stores a large number of pictures on a mobile phone, the pictures in the album can be identified through the target neural network model for image classification pre-installed on the mobile phone, and pictures of different categories can be labeled to facilitate the user or the system to classify and manage the album, thereby improving the user experience.

[0056] A possible application scenario: image super-resolution scenario, the training method provided in the embodiment of the present application can be used to train a neural network model for improving image resolution. For example, Figure 3 As shown, through neural network model search, multiple neural network models for improving image resolution are configured, and the same training data is used to perform model training on the multiple neural network models. The training data includes multiple groups of training data, and each group of training data includes a group of sample images with the same image content. Based on the training effect, a target neural network model whose performance meets the preset requirements is selected from multiple neural network models for improving image resolution in practical applications. Meeting the preset requirements includes, for example, a model loss value less than a first threshold, and / or the number of model floating-point operations less than a second threshold. In one example, a user takes an image in a scene with insufficient ambient light. The image is a low-resolution image. The image can be processed by a target neural network model for improving image resolution preset on the mobile phone to obtain a high-resolution image with improved image quality.

[0057] Figure 4 This is a schematic diagram of a system architecture provided in an embodiment of the present application. Figure 4 As shown, the data acquisition device 401 is used to collect training data for model training, such as Figure 2In the image classification task shown, the training data includes sample images and their annotated information, where the annotated information may be manually pre-annotated. After collecting multiple sets of training data, data acquisition device 401 may store this data in database 402. Neural network model training device 403 may then perform model training on the multiple neural network models found through neural network model search based on the multiple sets of training data in database 402, thereby selecting a target neural network model whose performance meets preset requirements.

[0058] It should be noted that, in actual applications, the training data in the database 402 may also come from other devices, for example, the training data comes from a cloud device, and this embodiment of the present application does not impose any limitation on this.

[0059] After the neural network model training device 403 screens out a target neural network model whose performance meets the preset requirements, the target neural network model can be preset in different systems or devices, such as Figure 4 The processor 405 of the electronic device 404 is shown. The electronic device 404 can be a terminal, such as a smartphone, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, in-vehicle terminal, etc., or a server or cloud server. The electronic device 404 is configured with an input / output (I / O) interface 406 for exchanging data with external devices, for example, obtaining image data to be processed through the I / O interface 406.

[0060] It should be noted that Figure 4 The system architecture shown is only an example, and the positional relationships between the devices, apparatuses, and modules shown in the figure do not constitute any limitation. Figure 4 The illustrated structure does not constitute a specific limitation on the electronic device 404. In some embodiments, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0061] like Figure 4 As shown, the target neural network model searched and trained by the neural network model training device 403 can be a convolutional neural network CNN, a deep convolutional neural network (DCNN), an encoder-decoder model, etc., and this embodiment of the application does not impose any restrictions on this.

[0062] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be implemented independently or in combination with each other. For the same or similar concepts or processes, some embodiments may not be described in detail.

[0063] Figure 5 A schematic diagram of a training method for a neural network model provided in an embodiment of the present application. The training method for the neural network model of this embodiment can be applied to Figure 4 The training device 403 of the neural network model shown (hereinafter referred to as the training device) includes:

[0064] S501. Obtain an initial neural network model suitable for a target processing task, where the initial neural network model includes a plurality of basic structures with preset connection relationships.

[0065] The target processing tasks include, but are not limited to, image processing tasks, such as image classification and image super-resolution. For ease of understanding, the following explanation will take the image classification task as an example to illustrate the solution.

[0066] The initial neural network model suitable for the image classification task can be an existing image classification model, and the model structure of the image classification model can be, for example, CNN, DCNN, encoder-decoder, etc.

[0067] For example, Figure 6 An image classification model is shown, which includes n basic structures, where n is 20, for example. The basic structures can be Figure 1 Any of the structures shown, such as Figure 6 In the example, base structure 1 can be a 3×3 convolutional layer, base structure 2 can be a 5×5 convolutional layer, and base structure 3 can include a 3×3 convolutional layer and a pooling layer. The 3×3 and 5×5 refer to the kernel size of the convolutional layer. It should be understood that the number of base structures in a model is related to the complexity of the processing task: the more complex the processing task, the larger the number of base structures.

[0068] It should be understood that the processing performance of the initial neural network model in this embodiment usually cannot meet the requirements of specific scenarios. The training device can generate a set of hyperparameters based on the model structure of the initial neural network model through an automated NAS search strategy, so as to configure an optimized neural network model based on the model structure of the initial neural network model and perform model training to obtain a neural network model with better performance.

[0069] S502. Based on the hyperparameter set generated by the preset search strategy, update the configuration of the initial neural network model to obtain a first neural network model.

[0070] In some embodiments, the search strategy is also called a search method. The search strategy can be any existing search strategy, such as reinforcement learning, evolutionary algorithm, agent-based model, gradient-based method, etc. The embodiments of this application do not impose any limitations on the search strategy.

[0071] In an optional embodiment, the hyperparameter set includes state control parameters of multiple basic structures, which can be used to update the weight configuration of the basic structures in the model.

[0072] In an optional embodiment, the hyperparameter set includes the number of channels of multiple basic structures, the structure index of multiple basic structures, and the state control parameters of multiple basic structures, which can be used to update the number of channels, structure type (single-branch structure or multi-branch structure, etc.), weight configuration, etc. of the basic structures in the model.

[0073] The number of channels of the infrastructure usually refers to the number of input channels of the infrastructure. For example, Figure 6 In the example, the channel counts of base structure 1, base structure 2, and base structure 3 are 32, 64, and 64, respectively, indicating that base structure 1 has 32 input channels and 64 output channels, base structure 2 has both 64 input channels and 64 output channels, and base structure 3 has 64 output channels. In some embodiments, the channel count of a base structure may also refer to the output channel count of the base structure. It should be understood that the output channel count of the previous base structure is the input channel count of the next base structure.

[0074] The structure index of the basic structure is used to indicate the structure information of the basic structure. It should be understood that Figure 1 The infrastructure shown are all predefined infrastructures. These infrastructures can be divided into different infrastructure sets according to different step sizes. In order to facilitate the search of infrastructures, different infrastructures in the same infrastructure set can be numbered. The number of each infrastructure is the structural index of the infrastructure, which is used to uniquely identify the infrastructure.

[0075] It should be noted that the stride of the convolution layer in the base architecture refers to the step size of the convolution kernel moving on the image. The size of the stride directly affects the result of the convolution operation and the size of the feature map. Similarly, the stride of the pooling layer in the base architecture refers to the step size of the pooling kernel moving on the image.

[0076] For example, Figure 7 shows the basic structure set with a step size of 1 and the basic structure set with a step size of 2, Figure 7It can be seen that there may be some basic structures in the basic structure set with a step size of 1, which are the same as the basic structures in the basic structure set with a step size of 2, for example Figure 7 The basic structure 4 is a 3×3 convolution layer, and the step size of the convolution layer can be set to 1 or 2. The basic structure 4 is a basic structure of the intersection of two basic structure sets.

[0077] In some embodiments, the set of infrastructures may also be referred to as a search space of infrastructures.

[0078] In some embodiments, the step length of the basic structure may not be distinguished, all basic structures may be included in one search space, and all basic structures in the search space may be numbered uniformly.

[0079] The state control parameter of the basic structure is used to indicate the structural state of the basic structure during model training. This parameter is newly introduced in the structure search process. The structural state of the basic structure during model training includes the first state, the second state, and the third state. The structural state of each basic structure in the neural network model during model training is one of these three states. The state control parameter of the basic structure in the hyperparameter set can indicate the structural state of each basic structure during model training.

[0080] For example, the three structural states are characterized by the following formulas:

[0081] 1) First state (also called offline re-parameterization state, state1): z = w1*x + w2*x + w3*x;

[0082] 2) Second state (also called online re-parameter state, state2): z = (a1*w1+a2*w2+a3*w3)x;

[0083] 3) The third state (also called the normal structure state, state3): z=w*x.

[0084] For 1), refer to Figure 8 , Figure 8 The basic structure shown on the left is a multi-branch structure, for example, it includes three branches, the first branch includes a convolution layer, the second branch includes a convolution layer, the third branch includes a convolution layer and a pooling layer, the weight configuration of the convolution layer in the first branch is w1, the weight configuration of the convolution layer in the second branch is w2, the weight configuration of the convolution layer in the third branch is w3, the input of the basic structure is x, and the output of the basic structure is z.

[0085] If the state control parameter of a multi-branch structure indicates that its structural state during model training is the first state, it can be Figure 8The principle shown on the left side is to configure weight parameters for the multi-branch structure. After the end of model training (at inference time), the multi-branch structure can be converted into the basic structure in the third state, such as Figure 8 The basic structure shown on the right side is z = w * x, where w = w1' + w2' + w3', w1' is the optimized (post-training) weight parameter of w1, w2' is the optimized weight parameter of w2, and w3' is the optimized weight parameter of w3. Based on the basic structure obtained through the foregoing structure configuration and training, the model structure and computational complexity can be simplified.

[0086] For 2), refer to Figure 9 , Figure 9 The basic structure shown on the top is a multi-branch structure, for example, including three branches, the first branch including one convolutional layer, the second branch including one convolutional layer, and the third branch including one convolutional layer and one pooling layer, which is similar to Figure 8 The weight parameter w1 is configured for the convolutional layer in the first branch, the weight parameter w2 is configured for the convolutional layer in the second branch, and the weight parameter w3 is configured for the convolutional layer in the third branch. If the structure state of the foregoing multi-branch structure is indicated as the second state, in addition to the foregoing configuration, the multi-channel weight parameters are also configured for the convolutional layers in the branches, such as the multi-channel weight parameter a1 is configured for the convolutional layer in the first branch, the multi-channel weight parameter a2 is configured for the convolutional layer in the second branch, and the multi-channel weight parameter a3 is configured for the convolutional layer in the third branch, where a1, a2, and a3 are vector parameters, and the dimension of the vector parameter is related to the number of channels, for example, the input channel numbers of the first branch, the second branch, and the third branch are all 32, and a1, a2, and a3 are 32-dimensional vectors.

[0087] If the state control parameter of a certain multi-branch structure indicates that its structure state during model training is the second state, the model training can be performed according to Figure 9 the basic structure shown on the left side below, instead of Figure 9 the multi-branch structure shown on the top, Figure 9 the basic structure shown on the left side below is based on Figure 9 the multi-branch structure shown on the top. That is, although the hyperparameter set indicates that a certain basic structure in the model is a multi-branch structure, if it is also indicated that the structure state of the basic structure is the second state, the model operation logic during model training is different from that of the foregoing multi-branch structure in the first state. Figure 8 and Figure 9For example, in the second state, the multi-branch structure first merges the weight parameters and then performs operations on the input during model training. In contrast, in the first state, the multi-branch structure first performs operations on each branch and then sums the results. It should be understood that for the same multi-branch structure, the computational speed of model training using the first state is slower than that using the second state.

[0088] Based on the second state infrastructure (such as Figure 9 After model training, the updated weight parameter w' can be obtained. The corresponding basic structure is as follows Figure 9 The infrastructure shown on the right side below is a third-state infrastructure, where w'=a1'*w1'+a2'*w2'+a3'*w3', w1' is the optimized weight parameter for w1, w2' is the optimized weight parameter for w2, w3' is the optimized weight parameter for w3, a1' is the optimized weight parameter for a1, a2' is the optimized weight parameter for a2, and a3' is the optimized weight parameter for a3. Based on the aforementioned structural configuration and the infrastructure obtained through training, the model structure and computational complexity can be simplified.

[0089] It should be pointed out that the multi-channel weight parameters (or weight vectors) a1, a2 and a3 added in the second state can be regarded as: adding a learnable scaling factor to the weight of the convolutional layer of each branch in the multi-branch structure. The scaling factor corresponds to each channel of the convolutional layer of each branch. The benefit is to expand the learning ability of the model during training, avoid the degeneration of multiple branches into a single branch, and thus obtain better model performance.

[0090] It should be understood that Figure 8 and Figure 9 The multi-branch structure shown is the same as Figure 9 The weight vectors a1, a2, and a3 configured in the multi-branch structure shown are all unit vectors (ie, the values ​​of each dimension in the vector are all 1), and the two have equivalent functions in the neural network model.

[0091] For 3), refer to Figure 10 , Figure 10 The basic structure shown is a single-branch structure, for example, it can be a common convolution layer, and the structural state of the convolution layer remains unchanged during and after model training, such as Figure 10 The basic structure of z=w*x in .

[0092] In an optional embodiment, the state control parameter includes two state control bits, illustratively, 10 represents the first state, 01 represents the second state, and 11 or 00 represents the third state. The basic structure of the first state or the second state can be converted to the basic structure of the third state after training.

[0093] In some embodiments, the infrastructure of any state after training can be synchronized to the search space of the infrastructure to expand the search space of the infrastructure, increase the diversity of the model structure composition, and facilitate the subsequent construction of a better quality model structure.

[0094] Based on the above description of the parameters in the hyperparameter set, several possible implementation methods for updating the configuration of the initial neural network model are described below. It should be noted that during a round of model structure search, several of the following embodiments may be executed simultaneously, or some of the following embodiments may be executed, depending on how the hyperparameter set indicates.

[0095] In an optional embodiment, the hyperparameter set indicates the number of channels of a first base structure, where the first base structure is any one of multiple base structures in the initial neural network model. In one example, if the original number of channels of the first base structure in the initial neural network model is inconsistent with the number of channels of the first base structure indicated in the hyperparameter set, the number of channels of the first base structure is updated to the number of channels of the first base structure indicated in the hyperparameter set.

[0096] For example, refer to Figure 6 , if the second basic structure in the original neural network model (i.e. Figure 6 The basic structure 2) in is a 5×5 convolutional layer with 64 channels. If the hyperparameter set indicates that the number of channels of the second basic structure is 32, the number of channels of the second basic structure is updated from 64 to 32.

[0097] In an optional embodiment, the hyperparameter set indicates structural information of a second base structure, where the second base structure is any one of a plurality of base structures in the initial neural network model. In one example, if the original structural information of the second base structure in the initial neural network model is inconsistent with the structural information of the second base structure indicated in the hyperparameter set, the second base structure is updated to the structure of the second base structure indicated in the hyperparameter set.

[0098] For example, refer to Figure 6 , if the third basic structure in the original neural network model (i.e. Figure 6 The basic structure 3) in includes a 3×3 convolution layer and a pooling layer. The basic structure is a single-branch structure. If the hyperparameter set indicates that the third basic structure is a multi-branch structure, such as Figure 8The multi-branch structure shown on the left will Figure 6 The infrastructure 3 in is replaced by Figure 8 The multi-branch structure shown on the left. In this example, the number of branches in the updated base structure is greater than the number of branches in the base structure before the update.

[0099] For example, refer to Figure 6 , if the first basic structure in the original neural network model (i.e. Figure 6 The basic structure 1) in is a 3×3 convolution layer. If the first basic structure is indicated in the hyperparameter set Figure 1 The single-branch structure shown in (c) in the figure includes two convolutional layers. Figure 6 The infrastructure 1 in is replaced by Figure 1 The single-branch structure shown in (c) in the figure. In this example, the basic structures before and after the update are both single-branch structures, and the number of layers in the updated single-branch structure is greater than that in the previous structure.

[0100] Based on the two aforementioned examples, in this embodiment, if the original structural information of the second basic structure in the initial neural network model is inconsistent with the structural information of the second basic structure indicated in the hyperparameter set, the second basic structure is updated to the structure of the second basic structure indicated in the hyperparameter set, and the number of branches of the updated second basic structure is greater than the original number of branches of the second basic structure, or the number of layers of the updated second basic structure is greater than the original number of layers of the second basic structure. By changing the structural information of part of the basic structure in the model, the model performance is optimized.

[0101] In some embodiments, if the original structural information of a certain basic structure in the initial neural network model is consistent with the structural information of the basic structure indicated in the hyperparameter set, for example, the original structure of a certain basic structure is a convolutional layer, and the structure of the basic structure indicated in the hyperparameter set is still a convolutional layer, then the basic structure remains unchanged.

[0102] It should be understood that by optimizing the basic structure in the neural network model, such as increasing the number of basic structure layers or branches, although the model structure becomes complex and the number of learnable parameters increases accordingly, the training effect of the model training based on the optimized neural network model is improved.

[0103] In an optional embodiment, the hyperparameter set indicates the structural information and structural state of the third basic structure, and the third basic structure is any one of the multiple basic structures in the initial neural network model. In one example, if the original structural information of the third basic structure in the initial neural network model is a single-branch structure, the structural information of the third basic structure indicated in the hyperparameter set is a multi-branch structure, and the structural state of the third basic structure indicated in the hyperparameter set is the first state (such as the state control parameter corresponding to the third basic structure is 10), the third basic structure is updated to a multi-branch structure, and the learnable weight parameters are configured for the convolutional layers in each branch of the third basic structure.

[0104] For example, refer to Figure 6 , if the third basic structure in the original neural network model (i.e. Figure 6 The basic structure 3) in includes a 3×3 convolution layer and a pooling layer. The basic structure is a single-branch structure. If the hyperparameter set indicates that the third basic structure is a multi-branch structure, such as Figure 8 The multi-branch structure shown on the left, and the hyperparameter set indicates that the structural state of the third basic structure is the first state (for example, the state control parameter corresponding to the third basic structure is 10), then Figure 6 The infrastructure 3 in is replaced by Figure 8 The multi-branch structure shown on the left configures learnable weight parameters for the convolutional layers in each branch, such as Figure 8 w1, w2 and w3 in .

[0105] The above embodiment shows that part of the basic structure in the initial neural network model is updated to a multi-branch structure, and weight parameters are configured for the convolutional layers in the multi-branch structure to optimize the model performance.

[0106] In an optional embodiment, the hyperparameter set indicates the structural information and structural state of the fourth basic structure, and the fourth basic structure is any one of the multiple basic structures in the initial neural network model. In one example, if the original structural information of the fourth basic structure in the initial neural network model is a single-branch structure, the structural information of the fourth basic structure indicated in the hyperparameter set is a multi-branch structure, and the structural state of the fourth basic structure indicated in the hyperparameter set is the second state (such as the state control parameter corresponding to the fourth basic structure is 01), it can be executed: configuring learnable weight parameters for the convolutional layer in each branch of the multi-branch structure, and configuring learnable weight parameters for each channel of the convolutional layer in each branch of the multi-branch structure; based on the multi-branch structure, the learnable weight parameters of the convolutional layer in each branch of the multi-branch structure, and the learnable weight parameters of each channel of the convolutional layer in each branch, generate weight merging information; based on the weight merging information, update the weight configuration of the fourth basic structure, and the updated fourth basic structure is still a single-branch structure.

[0107] For example, refer to Figure 6 , if the third basic structure in the original neural network model (i.e. Figure 6 The basic structure 3) in includes a 3×3 convolution layer and a pooling layer. The basic structure is a single-branch structure. If the hyperparameter set indicates that the third basic structure is a multi-branch structure, such as Figure 9 The multi-branch structure shown above, and the hyperparameter set indicates that the structural state of the third basic structure is the second state (such as the state control parameter corresponding to the third basic structure is 01), can be Figure 9 The multi-branch structure shown above configures the weight parameters (such as Figure 9 After a1, w1, a2, w2, a3, w3) in, generate weight merging information (such as Figure 9 w=a1*w1+a2*w2+a3*w3) shown in the figure, based on the weighted information, Figure 6 The infrastructure 3 in is replaced by Figure 9 The single-branch structure shown on the left below is used for subsequent model training.

[0108] The above embodiment shows the updating of the weight configuration of some basic structures in the initial neural network model. The weight parameters of the basic structure are no longer single-dimensional weight parameters, but include multi-dimensional weight parameters. During model training, for a certain basic structure, the weight parameters of the basic structure are first calculated, and then the operation is performed with the input of the basic structure, which can improve the speed of model training.

[0109] In an optional embodiment, the hyperparameter set indicates the structural information and structural state of the fifth basic structure, and the fifth basic structure is any one of the multiple basic structures in the initial neural network model. If the fifth basic structure in the initial neural network model is consistent with the structure of the fifth basic structure indicated in the hyperparameter set, the fifth basic structure is a multi-branch structure, and the structural state of the fifth basic structure indicated in the hyperparameter set is the second state (such as the state control parameter corresponding to the fifth basic structure is 01), it can be executed: configuring learnable weight parameters for the convolutional layer in each branch of the multi-branch structure, and configuring learnable weight parameters for each channel of the convolutional layer in each branch of the multi-branch structure; based on the multi-branch structure, the learnable weight parameters of the convolutional layer in each branch of the multi-branch structure, and the learnable weight parameters of each channel of the convolutional layer in each branch, generate weight merging information; based on the weight merging information, update the weight configuration of the fifth basic structure, and the updated fifth basic structure is a single-branch structure;.

[0110] For example, if the third basic structure in the original neural network model is Figure 9 The multi-branch structure shown above, if the hyperparameter set indicates the multi-branch structure of the third basic structure Figure 9 The multi-branch structure shown above indicates that the structural state of the third basic structure is the second state (such as the state control parameter corresponding to the third basic structure is 01). Then, a learnable weight parameter is configured for the convolution layer of each branch in the third basic structure, and a learnable weight parameter is configured for each channel of the convolution layer of each branch. For example, Figure 9 Based on the configurations w1, w2 and w3 of the convolutional layers of each branch shown in the figure, a1, a2 and a3 are further configured for the convolutional layers of each branch. After weight merging, the third basic structure is updated as Figure 9 The single-branch structure shown on the left below has a weight parameter expressed as w=a1*w1+a2*w2+a3*w3.

[0111] The above embodiment shows that part of the basic structure in the updated initial neural network model is a single-branch structure. The weight configuration of the single-branch structure is generated based on the weight configuration of the multi-branch structure indicated by the hyperparameter set. The weight parameters of the single-branch structure are no longer single-dimensional weight parameters, but include multi-dimensional weight parameters. During model training, for the updated single-branch structure, the weight parameter operation of the single-branch structure is first performed, and then the operation is performed with the input of the branch structure, which can improve the speed of model training.

[0112] It should be noted that the descriptions of the first basic structure and the second basic structure respectively indicate the first basic structure and the second basic structure in the initial neural network model. The "first" and "second" in the description indicate the order. Figure 6 The above descriptions of the first basic structure, the second basic structure, etc. are only used to distinguish the basic structures of different embodiments, and the "first", "second", etc. in the description do not indicate the order.

[0113] Through one or more of the above optional embodiments, the configuration update of the initial neural network model is completed to obtain the first neural network model of this model search. It should be understood that in the subsequent model search process, by executing this step, more neural network models with updated configurations based on different hyperparameter sets can be obtained. Subsequently, these models can be used for model training to obtain the trained model effect, and then a target neural network model whose performance meets the preset requirements can be selected.

[0114] S503. Perform model training on the first neural network model using preset training data.

[0115] Model training uses different training data depending on the target processing task. For example, in image classification, the training data includes a certain number of sample images and their annotations. The annotations indicate the category of the sample images, such as whether the sample images are natural scenery, portraits, buildings, etc.

[0116] In one example, the training process of the first neural network model includes: taking the sample image as the input of the first neural network model, obtaining the prediction information of the sample image after processing by the first neural network model, combining the annotation information of the sample image to obtain the loss value of the first neural network model, updating the learnable weight parameters in the first neural network model according to the loss value, and stopping the model training after traversing the preset training data (i.e., the number of training times is fixed).

[0117] In one example, a backpropagation algorithm (BP) is used to update the parameters of the first neural network model. In another example, a gradient descent method is used to calculate the gradient values ​​corresponding to the optimized parameters of the network model layer by layer, which are used as the basis for updating the network model parameters.

[0118] S504. After the model training is completed, the basic structure in the first neural network model whose structural state is the first state or the second state is converted into a basic structure in the third state to obtain a second neural network model.

[0119] Among them, the second neural network model is a candidate neural network model suitable for the target processing task.

[0120] For example, Figure 8 The multi-branch structure shown on the left is a basic structure of the first neural network model during model training. The convolutional layer of each branch of the multi-branch structure is configured with learnable weight parameters, which is the basic structure of the first state. After the model training is completed, Figure 8 The multi-branch structure shown on the left is transformed into Figure 8 The basic structure shown on the right, Figure 8 The basic structure shown on the right is the basic structure of the third state, which can simplify the model structure and improve the calculation speed of the model.

[0121] For example, Figure 9 The multi-branch structure shown above is an intermediate state infrastructure in the process of updating the initial neural network model configuration. After configuring the learnable weight parameters for the convolutional layers of each branch of the intermediate state infrastructure and configuring the learnable weight parameters for each channel of the convolutional layer of each branch, the weights are merged to obtain Figure 9 The infrastructure shown on the left below is used for model training. After the model training is completed, you can Figure 9 The infrastructure shown below on the left translates to Figure 9 The basic structure shown below on the right, Figure 9 The infrastructure shown on the left below is the infrastructure in the second state. Figure 9The basic structure shown on the right side below is the basic structure of the third state, which can simplify the model structure during model training (convert the indicated multi-branch structure into a single-branch structure for model training) and improve the calculation speed of the model during training.

[0122] Based on the foregoing description, during the NAS search process, a portion of the original single-branch infrastructure in the initial neural network model can be replaced with a multi-branch infrastructure indicated by the structural index based on the searched infrastructure. Furthermore, the weight parameters of the multi-branch infrastructure can be configured using the structural state indicated by the state control parameter, thereby changing a portion of the infrastructure in the original neural network model and its structural state. After model training is completed, the infrastructure with a changed structural state can be converted to an infrastructure with a normal structural state (i.e., the third state), simplifying the model structure and computational complexity without affecting the performance of the trained model.

[0123] It should be noted that after training is completed, the total number of model layers of the model used for inference (such as the second neural network model obtained in S504) is consistent with that of the original neural network model and will not increase. In other words, the solution provided in the embodiment of the present application does not involve the search of model depth.

[0124] In the training method of the neural network model shown in this embodiment, for a set of hyperparameters searched by the training device, the structure and weight configuration of the initial neural network model suitable for the target processing task are first updated based on the set of hyperparameters to obtain a first neural network model, and then the first neural network model is trained based on the preset training data. The above scheme introduces a state control parameter in the hyperparameter to indicate the structural state of the basic structure in the model during model training, so as to expand the model structure and improve the training effect of the model. After the model training is completed, the basic structure of the first state or the second state in the model is converted to the basic structure of the third state to simplify the model structure and improve the calculation speed of the model during reasoning.

[0125] Figure 11 This is a schematic diagram of the entire process state of the basic structure in a neural network model provided in an embodiment of the present application. Figure 11 As shown in the figure, during the neural network model search phase, the structural state of each basic structure in the neural network model may be any of the three states defined above. After obtaining a set of hyperparameters based on the search strategy, the configuration of the neural network model can be updated based on this set of hyperparameters. After the neural network model configuration is completed and the model training phase begins, the structural state of each basic structure in the neural network model is fixed to a certain state. During the model inference phase after model training, the structural state of each basic structure in the neural network model is converted to the third state, which can reduce the amount of model calculations during inference and improve the model calculation speed.

[0126] Figure 12 A schematic diagram of another neural network model training method provided in an embodiment of the present application. The neural network model training method of this embodiment can be applied to Figure 4 The training device 403 of the neural network model shown, the training method, includes:

[0127] S1201. Obtain an initial neural network model suitable for a target processing task, where the initial neural network model includes a plurality of basic structures with preset connection relationships.

[0128] S1202. Based on multiple hyperparameter sets generated by a preset search strategy, the configuration of the initial neural network model is updated respectively to obtain multiple updated neural network models.

[0129] The multiple hyperparameter sets generated based on the preset search strategy can be represented in the form of Table 1, where each row in Table 1 corresponds to a hyperparameter set. The training device can update the configuration of the initial neural network model based on any hyperparameter set in Table 1 to obtain an updated neural network model.

[0130] Table 1

[0131]

[0132] The following describes the various hyperparameters using the first row of hyperparameters in Table 1 as an example.

[0133] 1) In the channel number array in the first column, the values ​​in the array sequentially indicate the number of output channels of the basic structures in the neural network model. For example, the first value 32 indicates the number of output channels of the first basic structure in the neural network model. It should be understood that the number of output channels of the first basic structure is the number of input channels of the second basic structure, and so on.

[0134] 2) In the basic structure index array of the second column, the numerical values ​​in the array indicate the structural information of the basic structure with a step size of 1 in the neural network model in sequence. For example, the first numerical value 12 indicates the structural information of the first basic structure with a step size of 1 in the neural network model.

[0135] 3) In the basic structure index array of the third column, the numerical values ​​in the array indicate the structural information of the basic structure with a step size of 2 in the neural network model in sequence. For example, the first numerical value 1 indicates the structural information of the first basic structure with a step size of 2 in the neural network model.

[0136] 4) the state control parameters in the fourth column and the fifth column, used to jointly indicate the structure state of each base structure in the neural network model. For example, the first value in the fourth column array is 0, the first value in the fifth column array is 0, combined as 00 (i.e., two state control bits of the state control parameter), used to indicate that the structure state of the first base structure in the neural network model is the third state. For another example, the second value in the fourth column array is 1, the second value in the fifth column array is 0, combined as 10, used to indicate that the structure state of the second base structure in the neural network model is the first state.

[0137] It should be noted that the parameters in Table 1 are only an example, the length of each array is related to the depth of the neural network model, and it should be understood that the greater the model depth, the longer the length of each array. In some embodiments, the array of base structure indexes in Table 1 can be combined into one array, i.e., without distinguishing the step length of the base structure, and the structure information of each base structure in the neural network model is indicated by one array.

[0138] The specific implementation principles of S1201 and S1202 of the embodiment are the same as those of S501 and S502 of the foregoing embodiment, and can be referred to in the foregoing, which will not be expanded here.

[0139] S1203. Using the same training data, the plurality of updated neural network models are respectively subjected to model training to obtain a plurality of candidate neural network models.

[0140] S1204. From the plurality of candidate neural network models, a neural network model with a model loss value less than a first threshold value and / or a floating point operation number less than a second threshold value is selected as a target neural network model.

[0141] Based on the plurality of hyperparameter sets shown in Table 1, the initial neural network model is respectively updated, the same training data is used, and the updated neural network model is subjected to model training, and the training effect of the plurality of candidate neural network models can be obtained. The evaluation parameters of the training effect include the loss value of the training (train loss), the floating point operation number (GFlops), and the parameter number (ParamNum). The parameter number is positively correlated with the floating point operation number, and the more the parameter number, the greater the floating point operation number.

[0142] In an optional embodiment, a condition meeting the model performance requirement is set, for example, the model loss value is less than the first threshold value and / or the floating point operation number is less than the second threshold value. When the condition is met, the search training is ended. That is, in the process of searching and training the model, if the model loss value of a candidate neural network model is less than the first threshold value and / or the floating point operation number is less than the second threshold value, the search training is stopped, and the candidate neural network model is taken as the target neural network model.

[0143] In an optional embodiment, by setting a search number N, after completing a preset number of model searches and training, a target neural network model having a model loss value less than a first threshold and / or a number of floating-point operations less than a second threshold is selected from the N candidate neural network models. Where N is a positive integer.

[0144] In the neural network model training method illustrated in this embodiment, under a determined set of hyperparameters, the state of each basic structure in the neural network model is fixed. The neural network model is then trained using the same training data to obtain the training effect of the model. The above steps are repeated to select a target neural network model that meets the model performance requirements from the multiple trained neural network models. In this method, since each neural network model has its structure and weight configuration updated based on the hyperparameter set before training, the training effect of each neural network model is improved, which facilitates the rapid screening of target models that meet the performance requirements.

[0145] Figure 13 This is a schematic diagram of the structure of a training device for a neural network model provided in an embodiment of the present application. Figure 13 As shown, the training device 1300 of the neural network model of this embodiment includes: an acquisition module 1301 and a processing module 1302.

[0146] An acquisition module 1301 is configured to acquire an initial neural network model suitable for a target processing task, wherein the initial neural network model includes a plurality of basic structures having a preset connection relationship;

[0147] Processing module 1302 is used to update the configuration of the initial neural network model based on a hyperparameter set generated by a preset search strategy to obtain a first neural network model; the hyperparameter set includes state control parameters for the multiple basic structures, and the state control parameters are used to indicate the structural state of the basic structure during model training; the first neural network model is trained using preset training data; after the model training is completed, the basic structure in the first neural network model whose structural state is the first state or the second state is converted into a basic structure whose structural state is the third state to obtain a second neural network model, and the second neural network model is a candidate neural network model suitable for the target processing task.

[0148] In an optional embodiment, the hyperparameter set further includes: the number of channels of the multiple basic structures and a structure index of the multiple basic structures, where the structure index is used to indicate the structure information of the basic structure.

[0149] In an optional embodiment, the processing module 1302 is used to update the number of channels of the first basic structure in the initial neural network model to the number of channels of the first basic structure indicated in the hyperparameter set if the original number of channels of the first basic structure in the initial neural network model is inconsistent with the number of channels of the first basic structure indicated in the hyperparameter set; the first basic structure is any one of the multiple basic structures.

[0150] In an optional embodiment, the processing module 1302 is used to update the second basic structure to the structure of the second basic structure indicated in the hyperparameter set if the original structural information of the second basic structure in the initial neural network model is inconsistent with the structural information of the second basic structure indicated in the hyperparameter set; the second basic structure is any one of the multiple basic structures.

[0151] In an optional embodiment, the number of branches of the updated second basic structure is greater than the original number of branches of the second basic structure, or the number of layers of the updated second basic structure is greater than the original number of layers of the second basic structure.

[0152] In an optional embodiment, the processing module 1302 is used to update the third basic structure to the multi-branch structure if the original structural information of the third basic structure in the initial neural network model is a single-branch structure, the structural information of the third basic structure indicated in the hyperparameter set is a multi-branch structure, and the structural state of the third basic structure indicated in the hyperparameter set is the first state, and at the same time configure learnable weight parameters for the convolutional layers in each branch of the third basic structure; the third basic structure is any one of the multiple basic structures.

[0153] In an optional embodiment, the processing module 1302 is used to configure learnable weight parameters for the convolutional layers in each branch of the multi-branch structure, and configure learnable weight parameters for each channel of the convolutional layers in each branch of the multi-branch structure if the original structural information of the fourth basic structure in the initial neural network model is a single-branch structure, the structural information of the fourth basic structure indicated in the hyperparameter set is a multi-branch structure, and the structural state of the fourth basic structure indicated in the hyperparameter set is the second state; based on the multi-branch structure, the learnable weight parameters of the convolutional layers in each branch of the multi-branch structure, and the learnable weight parameters of each channel of the convolutional layers in each branch, generate weight merging information; based on the weight merging information, update the weight configuration of the fourth basic structure, and the updated fourth basic structure is still the single-branch structure; the fourth basic structure is any one of the multiple basic structures.

[0154] In an optional embodiment, the processing module 1302 is used to configure learnable weight parameters for the convolutional layers in each branch of the multi-branch structure, and configure learnable weight parameters for each channel of the convolutional layers in each branch of the multi-branch structure if the fifth basic structure in the initial neural network model is consistent with the structure of the fifth basic structure indicated in the hyperparameter set, the fifth basic structure is a multi-branch structure, and the structural state of the fifth basic structure indicated in the hyperparameter set is the second state; based on the multi-branch structure, the learnable weight parameters of the convolutional layers in each branch of the multi-branch structure, and the learnable weight parameters of each channel of the convolutional layers in each branch, generate weight merging information; based on the weight merging information, update the weight configuration of the fifth basic structure, and the updated fifth basic structure is a single-branch structure; the fifth basic structure is any one of the multiple basic structures.

[0155] The training device of the neural network model of the embodiment of the present application has the same implementation principle and technical effect as the Figure 5 The method embodiments shown are similar and will not be described again here.

[0156] Figure 14 This is a schematic diagram of the structure of another neural network model training device provided in an embodiment of the present application. Figure 14 As shown, the training device 1400 of the neural network model of this embodiment includes: an acquisition module 1401 and a processing module 1402.

[0157] An acquisition module, configured to acquire an initial neural network model suitable for a target processing task, wherein the initial neural network model includes a plurality of basic structures having a preset connection relationship;

[0158] A processing module is used to update the configuration of the initial neural network model based on multiple hyperparameter sets generated by a preset search strategy to obtain multiple updated neural network models; the hyperparameter sets include state control parameters for the multiple basic structures, and the state control parameters are used to indicate the structural state of the basic structures during model training; using the same training data, the multiple updated neural network models are trained to obtain multiple candidate neural network models; from the multiple candidate neural network models, a neural network model with a model loss value less than a first threshold and / or a number of floating-point operations less than a second threshold is selected as the target neural network model.

[0159] The training device of the neural network model of the embodiment of the present application has the same implementation principle and technical effect as the Figure 12 The method embodiments shown are similar and will not be described again here.

[0160] It should be noted that the embodiments of the present application do not particularly limit the specific structure of the execution subject of the training method of a neural network model. As long as the code storing the training method of a neural network model of the embodiments of the present application can be run to perform processing according to the training method of a neural network model provided by the embodiments of the present application. For example, the execution subject of the training method of a neural network model provided by the embodiments of the present application can be a functional module in an electronic device that can call and execute a program, or a processing device applied to an electronic device, for example, the processing device is a chip.

[0161] In the above embodiments, a "module" may be a software program, a hardware circuit, or a combination of the two that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a group processor) and memory for executing one or more software or firmware programs, combined logic circuits, and / or other suitable components that support the described functions.

[0162] Therefore, the modules of each example described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0163] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 15 As shown, the electronic device 1500 includes: a processor 1501 and a memory 1502; the processor 1501 is used to execute the computer execution instructions stored in the memory 1502, so that the electronic device executes the technical solution of the above-mentioned method embodiment, and its implementation principle and technical effect are similar to those of the above-mentioned related embodiments, which will not be repeated here.

[0164] The memory may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0165] The memory can be independent and connected to the processor via a communication line, or it can be integrated with the processor.

[0166] The processor may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0167] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar to those of the above-mentioned related embodiments and will not be repeated here.

[0168] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is run, the electronic device executes the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar to those of the above-mentioned related embodiments and will not be repeated here.

[0169] An embodiment of the present application provides a chip, which includes a processor. The processor is used to call a computer program in a memory to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar to those of the above-mentioned related embodiments and will not be repeated here.

[0170] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. A training method for a neural network model, characterized in that: include: Acquire an initial neural network model suitable for a target processing task, wherein the initial neural network model includes a plurality of basic structures having a preset connection relationship; Based on a set of hyperparameters generated by a preset search strategy, the configuration of the initial neural network model is updated to obtain a first neural network model; the set of hyperparameters includes the number of channels of the multiple basic structures, the structural indexes of the multiple basic structures, and state control parameters for the multiple basic structures, the state control parameters are used to indicate the structural state of the basic structures during model training, and when the weight parameters of the multi-branch basic structure are configured based on the structural state indicated by the state control parameters, a portion of the basic structures in the initial neural network model and their structural states are changed; Performing model training on the first neural network model using preset training data, wherein the training data includes sample images; After the model training is completed, the basic structure in the first neural network model whose structural state is the first state or the second state is converted into a basic structure whose structural state is the third state to obtain a second neural network model, wherein the second neural network model is a candidate neural network model suitable for the target processing task, wherein the first state is the offline re-parameter state: z=w1 x+w2 x+w3 x, the second state is the online re-parameter state: z=(a1 w1+a2 w2+a3 w3)x, the third state is the common structural state: z=w x, where z is the output of the base structure, x is the input of the base structure, w1 is the weight configuration of the convolutional layer in the first branch, w2 is the weight configuration of the convolutional layer in the second branch, w3 is the weight configuration of the convolutional layer in the third branch, a1 is the multi-channel weight parameter of the convolutional layer configuration in the first branch, a2 is the multi-channel weight parameter of the convolutional layer configuration in the second branch, and a3 is the multi-channel weight parameter of the convolutional layer configuration in the third branch; Based on the set of hyperparameters generated by the preset search strategy, updating the configuration of the initial neural network model includes: If the original structural information of the third basic structure in the initial neural network model is a single-branch structure, the structural information of the third basic structure indicated in the hyperparameter set is a multi-branch structure, and the structural state of the third basic structure indicated in the hyperparameter set is the first state, updating the third basic structure to the multi-branch structure, and configuring learnable weight parameters for the convolutional layer in each branch of the third basic structure; The third basic structure is any one of the plurality of basic structures.

2. The method according to claim 1, characterized in that Based on the set of hyperparameters generated by the preset search strategy, updating the configuration of the initial neural network model also includes: If the original number of channels of the first basic structure in the initial neural network model is inconsistent with the number of channels of the first basic structure indicated in the hyperparameter set, updating the number of channels of the first basic structure to the number of channels of the first basic structure indicated in the hyperparameter set; The first basic structure is any one of the plurality of basic structures.

3. The method according to claim 1 or 2, characterized in that Based on the set of hyperparameters generated by the preset search strategy, updating the configuration of the initial neural network model also includes: If the original structure information of the second basic structure in the initial neural network model is inconsistent with the structure information of the second basic structure indicated in the hyperparameter set, updating the second basic structure to the structure of the second basic structure indicated in the hyperparameter set; The second basic structure is any one of the plurality of basic structures.

4. The method according to claim 3, characterized in that The number of branches of the updated second basic structure is greater than the original number of branches of the second basic structure, or, The updated number of layers of the second basic structure is greater than the original number of layers of the second basic structure.

5. The method according to claim 1 or 2, characterized in that Based on the set of hyperparameters generated by the preset search strategy, updating the configuration of the initial neural network model also includes: If original structural information of the fourth basic structure in the initial neural network model is a single-branch structure, the structural information of the fourth basic structure indicated in the hyperparameter set is a multi-branch structure, and the structural state of the fourth basic structure indicated in the hyperparameter set is the second state, configuring a learnable weight parameter for a convolutional layer in each branch of the multi-branch structure, and configuring a learnable weight parameter for each channel of the convolutional layer in each branch of the multi-branch structure; Generate weight merging information based on the multi-branch structure, the learnable weight parameters of the convolutional layers in each branch of the multi-branch structure, and the learnable weight parameters of each channel of the convolutional layers in each branch; Based on the weight merging information, updating the weight configuration of the fourth basic structure, wherein the updated fourth basic structure is still the single-branch structure; The fourth basic structure is any one of the plurality of basic structures.

6. The method according to claim 1 or 2, characterized in that Updating the configuration of the initial neural network model based on a set of hyperparameters generated by a preset search strategy also includes: If the fifth basic structure in the initial neural network model is consistent with the structure of the fifth basic structure indicated in the hyperparameter set, the fifth basic structure is a multi-branch structure, the structural state of the fifth basic structure indicated in the hyperparameter set is the second state, and a learnable weight parameter is configured for the convolutional layer in each branch of the multi-branch structure, and a learnable weight parameter is configured for each channel of the convolutional layer in each branch of the multi-branch structure; Generate weight merging information based on the multi-branch structure, the learnable weight parameters of the convolutional layers in each branch of the multi-branch structure, and the learnable weight parameters of each channel of the convolutional layers in each branch; Based on the weight merging information, updating the weight configuration of the fifth basic structure, wherein the updated fifth basic structure is a single-branch structure; The fifth basic structure is any one of the plurality of basic structures.

7. A training method for a neural network model, characterized in that: include: Acquire an initial neural network model suitable for a target processing task, wherein the initial neural network model includes a plurality of basic structures having a preset connection relationship; Based on multiple hyperparameter sets generated by a preset search strategy, the configuration of the initial neural network model is updated respectively to obtain multiple updated neural network models; the hyperparameter sets include the number of channels of the multiple basic structures, the structural indexes of the multiple basic structures, and state control parameters for the multiple basic structures, the state control parameters are used to indicate the structural state of the basic structures during model training, and when the weight parameters of the multi-branch basic structure are configured based on the structural state indicated by the state control parameters, a part of the basic structures in the initial neural network model and their structural state are changed; Using the same training data, respectively performing model training on the multiple updated neural network models to obtain multiple candidate neural network models, wherein the training data includes sample images; Selecting, from the multiple candidate neural network models, a neural network model whose model loss value is less than a first threshold and / or whose number of floating-point operations is less than a second threshold as a target neural network model; After the model training is completed, the basic structure in the target neural network model whose structural state is the first state or the second state is converted into a basic structure whose structural state is the third state, to obtain a third neural network model, wherein the third neural network model is a candidate neural network model suitable for the target processing task, wherein the first state is the offline re-parameter state: z=w1 x+w2 x+w3 x, the second state is the online re-parameter state: z=(a1 w1+a2 w2+a3 w3)x, the third state is the common structural state: z=w x, where z is the output of the base structure, x is the input of the base structure, w1 is the weight configuration of the convolutional layer in the first branch, w2 is the weight configuration of the convolutional layer in the second branch, w3 is the weight configuration of the convolutional layer in the third branch, a1 is the multi-channel weight parameter of the convolutional layer configuration in the first branch, a2 is the multi-channel weight parameter of the convolutional layer configuration in the second branch, and a3 is the multi-channel weight parameter of the convolutional layer configuration in the third branch; Based on multiple hyperparameter sets generated by a preset search strategy, the configuration of the initial neural network model is updated respectively, including: If the original structural information of the third basic structure in the initial neural network model is a single-branch structure, the structural information of the third basic structure indicated in the hyperparameter set is a multi-branch structure, and the structural state of the third basic structure indicated in the hyperparameter set is the first state, updating the third basic structure to the multi-branch structure, and configuring learnable weight parameters for the convolutional layer in each branch of the third basic structure; The third basic structure is any one of the plurality of basic structures.

8. A training device for a neural network model, characterized in that: include: An acquisition module, configured to acquire an initial neural network model suitable for a target processing task, wherein the initial neural network model includes a plurality of basic structures having a preset connection relationship; a processing module, configured to update the configuration of the initial neural network model based on a hyperparameter set generated by a preset search strategy to obtain a first neural network model; the hyperparameter set includes the number of channels of the multiple basic structures, the structural indexes of the multiple basic structures, and state control parameters for the multiple basic structures, the state control parameters being used to indicate the structural states of the basic structures during model training, and when weight parameters of the multi-branch basic structures are configured based on the structural states indicated by the state control parameters, a portion of the basic structures in the initial neural network model and their structural states are changed; Performing model training on the first neural network model using preset training data, wherein the training data includes sample images; After the model training is completed, the basic structure in the first neural network model whose structural state is the first state or the second state is converted into a basic structure whose structural state is the third state to obtain a second neural network model, wherein the second neural network model is a candidate neural network model suitable for the target processing task, wherein the first state is the offline re-parameter state: z=w1 x+w2 x+w3 x, the second state is the online re-parameter state: z=(a1 w1+a2 w2+a3 w3)x, the third state is the common structural state: z=w x, where z is the output of the base structure, x is the input of the base structure, w1 is the weight configuration of the convolutional layer in the first branch, w2 is the weight configuration of the convolutional layer in the second branch, w3 is the weight configuration of the convolutional layer in the third branch, a1 is the multi-channel weight parameter of the convolutional layer configuration in the first branch, a2 is the multi-channel weight parameter of the convolutional layer configuration in the second branch, and a3 is the multi-channel weight parameter of the convolutional layer configuration in the third branch; The processing module is specifically used to: If the original structural information of the third basic structure in the initial neural network model is a single-branch structure, the structural information of the third basic structure indicated in the hyperparameter set is a multi-branch structure, and the structural state of the third basic structure indicated in the hyperparameter set is the first state, updating the third basic structure to the multi-branch structure, and configuring learnable weight parameters for the convolutional layer in each branch of the third basic structure; The third basic structure is any one of the plurality of basic structures.

9. A training device for a neural network model, characterized in that: include: An acquisition module, configured to acquire an initial neural network model suitable for a target processing task, wherein the initial neural network model includes a plurality of basic structures having a preset connection relationship; a processing module, configured to update the configuration of the initial neural network model based on multiple hyperparameter sets generated by a preset search strategy, respectively, to obtain multiple updated neural network models; the hyperparameter sets include the number of channels of the multiple basic structures, the structural indexes of the multiple basic structures, and state control parameters for the multiple basic structures, the state control parameters being used to indicate the structural state of the basic structures during model training, and when configuring the weight parameters of the multi-branch basic structure based on the structural state indicated by the state control parameters, causing a portion of the basic structures in the initial neural network model and their structural states to change; Using the same training data, respectively performing model training on the multiple updated neural network models to obtain multiple candidate neural network models, wherein the training data includes sample images; Selecting, from the multiple candidate neural network models, a neural network model whose model loss value is less than a first threshold and / or whose number of floating-point operations is less than a second threshold as a target neural network model; After the model training is completed, the basic structure in the target neural network model whose structural state is the first state or the second state is converted into a basic structure whose structural state is the third state, to obtain a third neural network model, wherein the third neural network model is a candidate neural network model suitable for the target processing task, wherein the first state is the offline re-parameter state: z=w1 x+w2 x+w3 x, the second state is the online re-parameter state: z=(a1 w1+a2 w2+a3 w3)x, the third state is the common structural state: z=w x, where z is the output of the base structure, x is the input of the base structure, w1 is the weight configuration of the convolutional layer in the first branch, w2 is the weight configuration of the convolutional layer in the second branch, w3 is the weight configuration of the convolutional layer in the third branch, a1 is the multi-channel weight parameter of the convolutional layer configuration in the first branch, a2 is the multi-channel weight parameter of the convolutional layer configuration in the second branch, and a3 is the multi-channel weight parameter of the convolutional layer configuration in the third branch; The processing module is specifically used to: If the original structural information of the third basic structure in the initial neural network model is a single-branch structure, the structural information of the third basic structure indicated in the hyperparameter set is a multi-branch structure, and the structural state of the third basic structure indicated in the hyperparameter set is the first state, updating the third basic structure to the multi-branch structure, and configuring learnable weight parameters for the convolutional layer in each branch of the third basic structure; The third basic structure is any one of the plurality of basic structures.

10. An electronic device, characterized in that: include: processor and memory; The processor is configured to execute the computer-executable instructions stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images