A neural network generation method and device, computer equipment and storage medium

Abstract

The present disclosure provides a neural network generation method and device, computer equipment and a storage medium, and the method comprises the following steps: determining a network layer to be adjusted based on the operator types corresponding to each network layer in a neural network to be deployed; adjusting at least one of the operator types and network layer parameters of the network layer to be adjusted based on the data frame number of the data to be processed of the neural network to be deployed, to obtain a target network layer; and generating a target neural network based on the target network layer.

Description

Technical Field

[0001] This disclosure relates to the field of neural network technology, and more specifically, to a method, apparatus, computer device, and storage medium for generating a neural network. Background Technology

[0002] In certain image processing tasks, such as action recognition and action prediction, multiple frames of data need to be input into a neural network. The neural network extracts features over time and within each single frame to obtain the inference result. Neural networks have different requirements when deployed on different platforms. When deploying a convolutional neural network on a Neural Network Inference Engine (NNIE) platform, due to the NNIE platform's limitations on input data tensors, the neural network needs to be retrained to meet the NNIE platform's deployment requirements. This retraining method leads to low efficiency. Summary of the Invention

[0003] This disclosure provides at least one method, apparatus, computer device, and storage medium for generating a neural network.

[0004] In a first aspect, embodiments of this disclosure provide a method for generating a neural network, comprising: determining a network layer to be adjusted based on the operator types corresponding to each network layer in the neural network to be deployed; adjusting at least one of the operator type and network layer intrinsic parameters of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed, thereby obtaining a target network layer; and generating a target neural network based on the target network layer.

[0005] In one optional implementation, the network layer to be adjusted includes a convolutional layer; the intrinsic parameters of the network layer include a first number of input channels and a first number of output channels; adjusting the operator type and / or intrinsic parameters of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain a target network layer, includes: changing the operator type of the convolutional layer to grouped convolution based on the number of data frames of the data to be processed in the neural network to be deployed; the first number of groups in the grouped convolution is the same as the number of data frames; and adjusting the first number of input channels and the first number of output channels of the convolutional layer based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain a first target number of input channels and a first target number of output channels; and constructing the target network layer based on the changed operator type of grouped convolution, the first target number of input channels, and the first target number of output channels.

[0006] In an optional implementation, the network layer intrinsic parameters further include: a first convolutional kernel corresponding to each first output channel; after adjusting the number of first input channels and the number of first output channels of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain the number of first target input channels and the number of first target output channels, the method further includes: amplifying the first convolutional kernel based on the number of first groups of the grouped convolution, to obtain a first target convolutional kernel corresponding to each first target output channel; the method of constructing the target network layer based on the operator type changed to grouped convolution, the number of first target input channels, and the number of first target output channels includes: constructing the target network layer based on the operator type changed to grouped convolution, the number of first target input channels, the number of first target output channels, and the first target convolutional kernel.

[0007] In one optional implementation, the step of constructing the target network layer based on the operator type changed to grouped convolution, the first target number of input channels, the first target number of output channels, and the first target convolution kernel includes: determining the operator corresponding to the target network layer based on the operator type changed to grouped convolution; and determining the input channels and output channels for the operator corresponding to the target network layer based on the first target number of input channels and the first target number of output channels, and determining the corresponding first target convolution kernel for each output channel and the corresponding input data for each input channel to obtain the target network layer.

[0008] In one optional implementation, the network layer to be adjusted further includes: a batch normalization layer connected to the output of the convolutional layer; before adjusting the operator type and / or network layer intrinsic parameters of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed to obtain the target network layer, the method further includes: determining a third relationship information between the first input data of the convolutional layer and the second output data of the batch normalization layer based on a first relationship information between the first input data and the first output data of the convolutional layer, and a second relationship information between the second input data and the second output data of the batch normalization layer; merging the batch normalization layer into the convolutional layer based on the third relationship information to obtain a new convolutional layer; adjusting at least one of the operator type and network layer intrinsic parameters of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed to obtain the target network layer, which includes: adjusting at least one of the operator type and network layer intrinsic parameters of the new convolutional layer based on the number of data frames of the data to be processed in the neural network to be deployed to obtain the target network layer.

[0009] In one optional implementation, merging the batch normalization layer into the convolutional layer based on the third relation information to obtain a new convolutional layer includes: adjusting at least one of the first convolutional kernel and bias term coefficients of the convolutional layer based on the third relation information to obtain a new first convolutional kernel and / or a new bias term coefficient; and obtaining the new convolutional layer based on the new first convolutional kernel and / or the new bias term coefficient.

[0010] In one optional implementation, obtaining the new convolutional layer based on the new first convolutional kernel and / or the new bias term coefficients includes: replacing the first convolutional kernel in the operator corresponding to the convolutional layer with the new first convolutional kernel, and / or replacing the bias term coefficients in the operator corresponding to the convolutional layer with the new bias term coefficients to obtain the new convolutional layer.

[0011] In one optional implementation, the network layer to be adjusted further includes: a fully connected layer; the network layer intrinsic parameters include: fully connected weights; adjusting at least one of the operator type and network layer intrinsic parameters of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain a target network layer, includes: changing the operator type of the fully connected layer to grouped convolution based on the number of data frames of the data to be processed in the neural network to be deployed, and changing the fully connected weights of the fully connected layer to the convolution kernels of the convolution operator, to obtain the target network layer; wherein, the number of the second group to which the operator type of the fully connected layer is changed to grouped convolution is the same as the number of data frames.

[0012] In one optional implementation, the step of changing the operator type of the fully connected layer to grouped convolution and changing the fully connected weights of the fully connected layer to the convolution kernels of the convolution operator based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain the target network layer, includes: adjusting the number of second input channels and the number of second output channels of the fully connected layer based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain the second target number of input channels and the second target number of output channels after changing the operator type of the fully connected layer to grouped convolution; and generating a second convolution kernel after changing the operator type of the fully connected layer to grouped convolution based on the second number of groups; the number of the second convolution kernels is the same as the second target number of output channels.

[0013] In one optional implementation, the method further includes: adding an input data fusion layer at the input end of the neural network to be deployed; the input data fusion layer is used to fuse multiple frames of data to be processed into a target number of frames of data to be processed.

[0014] In one optional implementation, the target neural network includes an image recognition network, which is used to perform behavior recognition on an image containing a target human body to obtain a behavior recognition result of the target human body in the image.

[0015] In one optional implementation, after generating the target neural network, the method further includes: deploying the target neural network on a neural network inference engine (NNIE).

[0016] Secondly, embodiments of this disclosure also provide a neural network generation apparatus, comprising: a determining module, configured to determine a network layer to be adjusted based on the operator types corresponding to each network layer in the neural network to be deployed; an adjusting module, configured to adjust at least one of the operator type and network layer intrinsic parameters of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain a target network layer; and a generating module, configured to generate a target neural network based on the target network layer.

[0017] In one optional implementation, the network layer to be adjusted includes a convolutional layer; the intrinsic parameters of the network layer include a first number of input channels and a first number of output channels; when the adjustment module adjusts the operator type and / or intrinsic parameters of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain a target network layer, it is used to: change the operator type of the convolutional layer to grouped convolution based on the number of data frames of the data to be processed in the neural network to be deployed; the first group number of the grouped convolution is the same as the number of data frames; and adjust the first number of input channels and the first number of output channels of the convolutional layer based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain a first target number of input channels and a first target number of output channels; and construct the target network layer based on the operator type changed to grouped convolution, the first target number of input channels, and the first target number of output channels.

[0018] In an optional implementation, the network layer intrinsic parameters further include: a first convolutional kernel corresponding to each first output channel; after adjusting the number of first input channels and the number of first output channels of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain the number of first target input channels and the number of first target output channels, the adjustment module is further configured to: amplify the first convolutional kernel based on the number of first groups of the grouped convolution to obtain the first target convolutional kernel corresponding to each first target output channel; when the adjustment module constructs the target network layer based on the operator type changed to grouped convolution, the number of first target input channels, and the number of first target output channels, it is configured to: construct the target network layer based on the operator type changed to grouped convolution, the number of first target input channels, the number of first target output channels, and the first target convolutional kernel.

[0019] In one optional implementation, when the adjustment module constructs the target network layer based on the operator type changed to grouped convolution, the first target number of input channels, the first target number of output channels, and the first target convolution kernel, it is configured to: determine the operator corresponding to the target network layer based on the operator type changed to grouped convolution; and, based on the first target number of input channels and the first target number of output channels, determine the input channels and output channels for the operator corresponding to the target network layer, and determine the corresponding first target convolution kernel for each output channel and the corresponding input data for each input channel to obtain the target network layer.

[0020] In an optional implementation, the network layer to be adjusted further includes a batch normalization layer connected to the output of the convolutional layer; before the adjustment module adjusts the operator type and / or network layer intrinsic parameters of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed to obtain the target network layer, it is further configured to: determine a third relationship information between the first input data of the convolutional layer and the second output data of the batch normalization layer based on the first relationship information between the first input data and the first output data of the convolutional layer, and the second relationship information between the second input data and the second output data of the batch normalization layer; merge the batch normalization layer into the convolutional layer based on the third relationship information to obtain a new convolutional layer; when the adjustment module adjusts at least one of the operator type and network layer intrinsic parameters of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed to obtain the target network layer, it is configured to: adjust at least one of the operator type and network layer intrinsic parameters of the new convolutional layer based on the number of data frames of the data to be processed in the neural network to be deployed to obtain the target network layer.

[0021] In one optional implementation, when the adjustment module merges the batch normalization layer into the convolutional layer based on the third relation information to obtain a new convolutional layer, it is configured to: adjust at least one of the first convolutional kernel and the bias term coefficient of the convolutional layer based on the third relation information to obtain a new first convolutional kernel and / or a new bias term coefficient; and obtain the new convolutional layer based on the new first convolutional kernel and / or the new bias term coefficient.

[0022] In one optional implementation, when the adjustment module obtains the new convolutional layer based on the new first convolutional kernel and / or the new bias term coefficients, it is configured to: replace the first convolutional kernel in the operator corresponding to the convolutional layer with the new first convolutional kernel, and / or replace the bias term coefficients in the operator corresponding to the convolutional layer with the new bias term coefficients, thereby obtaining the new convolutional layer.

[0023] In one optional implementation, the network layer to be adjusted further includes: a fully connected layer; the network layer intrinsic parameters include: fully connected weights; when the adjustment module adjusts at least one of the operator type and network layer intrinsic parameters of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain the target network layer, it is used to: change the operator type of the fully connected layer to grouped convolution based on the number of data frames of the data to be processed in the neural network to be deployed, and change the fully connected weights of the fully connected layer to the convolution kernel of the convolution operator, to obtain the target network layer; wherein, the number of the second groups to change the operator type of the fully connected layer to grouped convolution is the same as the number of data frames.

[0024] In one optional implementation, when the adjustment module changes the operator type of the fully connected layer to grouped convolution and changes the fully connected weights of the fully connected layer to the convolution kernels of the convolution operator based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain the target network layer, it is used to: adjust the number of second input channels and the number of second output channels of the fully connected layer based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain the second target number of input channels and the second target number of output channels after the operator type of the fully connected layer is changed to grouped convolution; and generate a second convolution kernel after the operator type of the fully connected layer is changed to grouped convolution based on the second number of groups; the number of the second convolution kernels is the same as the second target number of output channels.

[0025] In one optional implementation, the adjustment module is further configured to: add an input data fusion layer at the input end of the neural network to be deployed; the input data fusion layer is configured to fuse multiple frames of data to be processed into a target number of frames of data to be processed.

[0026] In one optional implementation, the target neural network includes an image recognition network, which is used to perform behavior recognition on an image containing a target human body to obtain a behavior recognition result of the target human body in the image.

[0027] In one optional implementation, after generating the target neural network, the generation module is further configured to: deploy the target neural network on a neural network inference engine (NNIE).

[0028] Thirdly, an optional implementation of this disclosure also provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is configured to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the steps of the first aspect above, or any possible implementation of the first aspect, are performed.

[0029] Fourthly, an optional implementation of this disclosure also provides a computer-readable storage medium storing a computer program that, when run, performs the steps of the first aspect or any possible implementation of the first aspect.

[0030] For a description of the effects of the aforementioned neural network generation apparatus, computer equipment, and computer-readable storage medium, please refer to the description of the aforementioned neural network generation method; it will not be repeated here.

[0031] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0032] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0033] Figure 1 A flowchart illustrating a method for generating a neural network according to an embodiment of this disclosure is shown;

[0034] Figure 2a This diagram illustrates a method for processing data using a first convolution kernel, as provided in an embodiment of this disclosure.

[0035] Figure 2b This diagram illustrates a method for processing data using a first target convolution kernel, as provided in an embodiment of this disclosure.

[0036] Figure 3 A schematic diagram of a neural network generation apparatus provided in an embodiment of the present disclosure is shown;

[0037] Figure 4 A schematic diagram of a computer device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown herein can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0039] Research has found that in image processing tasks such as behavior recognition and behavior prediction, neural networks are typically used to perform feature analysis on multiple frames of data to obtain inference results. Feature analysis can include analyzing the temporal features of multiple frames and / or analyzing the image features of a single frame. Taking human behavior recognition as an example, human behavior usually involves multiple consecutive actions, each forming a frame of image data. To recognize human behavior, the images corresponding to each action need to be used as input data. Simultaneously, the temporal relationships between these actions must be analyzed to determine the final behavior recognition result. Various platforms are available for deploying such neural networks, including the NNIE platform and the NVIDIA platform. Different platforms have different requirements for the data tensors of the neural network when deployed. For example, NVIDIA's inherent characteristics allow for inputting multiple consecutive frames of data into the neural network and outputting the final result; however, the NNIE platform requires only one frame of data to be input at a time, making it impossible for neural networks deployed on the NNIE platform to perform temporal analysis using multiple frames of data. To deploy neural networks for behavior recognition and prediction on the NNIE platform, multiple frames of data need to be fused into a single frame. The fused data is then used to train the neural network, adjusting its parameters so that it can extract temporal features from the fused frame, thus enabling behavior recognition and prediction. However, for the NNIE platform, directly training the neural network to extract temporal features from a single input frame typically involves adding grouped convolutional layers. These layers perform grouped convolutions on the input frame to extract its temporal features. However, maintaining consistent weights across the groups during training is difficult; large differences in weights between groups can lead to performance degradation. To reduce these weight differences, current approaches pre-train the neural network using non-grouped convolutional operators, then adjust the operator type and fuse multiple input frames into a single frame. The neural network (i.e., the neural network to be deployed in this embodiment) that has undergone operator type adjustment is retrained using the fused input frame to obtain a neural network that can be deployed on the NNIE platform. However, this method of generating a neural network that can be deployed on the NNIE platform is inefficient because it requires a lot of time to retrain the neural network to be deployed.

[0040] Based on the above research, this disclosure provides a method for generating neural networks. By utilizing the number of data frames of the data to be processed in the neural network to be deployed, the method adjusts the operator type and at least one of the network layer intrinsic parameters on a tensor basis. This eliminates the need to retrain the neural network to be deployed. Instead, it directly adjusts the operator type and / or the network layer intrinsic parameters on a tensor basis, thereby converting the neural network to be deployed into a target neural network that can be deployed on the NNIE platform. This improves the performance of the neural network and is also highly efficient because it does not require retraining.

[0041] The shortcomings of the above solutions are the result of the inventor's practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below should be considered as the inventor's contribution to this disclosure.

[0042] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0043] To facilitate understanding of this embodiment, a method for generating a neural network disclosed in this disclosure will first be described in detail. The execution entity of the neural network generation method provided in this disclosure is generally a computer device with certain computing capabilities. This computer device may include, for example, a terminal device, a server, or other processing devices. The terminal device may be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the neural network generation method can be implemented by a processor calling computer-readable instructions stored in memory.

[0044] The method for generating neural networks provided in the embodiments of this disclosure will now be described.

[0045] See Figure 1 The diagram shows a flowchart of a neural network generation method provided in this embodiment of the present disclosure. The method includes steps S101 to S103, wherein:

[0046] S101: Determine the network layers to be adjusted based on the operator types corresponding to each network layer in the neural network to be deployed;

[0047] S102: Based on the number of data frames of the data to be processed in the neural network to be deployed, adjust at least one of the operator type and intrinsic parameters of the network layer to be adjusted to obtain the target network layer;

[0048] S103: Generate the target neural network based on the target network layer.

[0049] This embodiment of the disclosure utilizes the number of data frames of the data to be processed in the neural network to be deployed to adjust the operator type and / or adjust the tensor of the network layer intrinsic parameters. In this way, without training the neural network to be deployed, but directly adjusting the operator type and / or adjusting the tensor of the network layer intrinsic parameters, the purpose of generating a target neural network that can be deployed on the NNIE platform is achieved, thereby improving the performance of the neural network.

[0050] The above S101 to S103 are explained in detail below.

[0051] Regarding S101 above, when the neural network generation method provided in this embodiment is applied to a scenario of behavior recognition or behavior prediction, the neural network to be deployed may include, for example, a convolutional neural network (CNN). As can be seen from the above description of the background technology, the platform on which the neural network to be deployed is actually deployed may include, for example, the NNIE platform.

[0052] When the neural network to be deployed includes a convolutional neural network, the convolutional neural network includes an input layer, convolutional layers, activation layers, fully connected layers, batch normalization layers, pooling layers, etc. The types of operators corresponding to different network layers are different. For example, if the network layer includes convolutional layers, the corresponding operators are convolutional operators; if the network layer includes fully connected layers, the corresponding operators are fully connected operators.

[0053] In the data processing of scenarios involving behavior recognition or prediction using a neural network to be deployed, the data to be processed may include video frame images. In one possible case, due to the specific requirements of behavior recognition or prediction, when processing the data, temporal feature extraction is performed on multiple consecutive video frame images, for example, temporal feature extraction is performed on 8 video frame images; and for each video frame image, image feature extraction is performed to determine the action and behavior characteristics of the human figure in that frame image.

[0054] When deploying a neural network on the NNIE platform, the NNIE platform requires that the first dimension of the tensor shape of the input data to the neural network cannot be changed during forward propagation. However, in scenarios such as action recognition, the shape of the tensor of the input data will change between the number of data frames (also known as the batch number) and the number of input channels. Therefore, to address the problems mentioned in the background, for example, some network layers in the neural network to be deployed can be adjusted to solve the problem of low neural network performance. In this embodiment, the network layer that needs adjustment (i.e., some network layers in the neural network to be deployed) is referred to as the network layer to be adjusted.

[0055] In specific implementations, when determining the network layer to be adjusted, it can be determined, for example, based on the operator types corresponding to each network layer in the neural network to be deployed. In one possible implementation, since the operators corresponding to the convolutional layer, fully connected layer, and batch normalization layer in the convolutional neural network may change due to the change in the tensor dimension of the input data to that network layer, specifically including changes in the number of operators, the determined network layer to be adjusted in this embodiment of the disclosure may include at least one of the convolutional layer, fully connected layer, and batch normalization layer with operators that may change.

[0056] Regarding S102 above, once the network layer to be adjusted is determined, the operator type and / or intrinsic parameters of the network layer can be adjusted so that the neural network to be deployed can be deployed on the NNIE platform.

[0057] Based on S101 above, the network layer to be adjusted can be determined to include at least one of convolutional layers, fully connected layers, and batch normalized layers. The specific methods for adjusting the operator types and / or intrinsic parameters of convolutional layers, fully connected layers, and batch normalized layers are described below.

[0058] (A): The network layers to be adjusted include convolutional layers.

[0059] In the case where the network layer to be adjusted includes a convolutional layer, the intrinsic parameters of the network layer may include, for example, the number of first input channels and the number of first output channels. Additionally, the intrinsic parameters of the network layer may also include, for example, the height and width of the convolutional kernel. For example, for ease of description, the number of first input channels of the convolutional layer is represented as... The number of the first output channels of the convolutional layer is expressed as... The height of the convolution kernel is represented as And the kernel width is expressed as The number of groups is represented by "group". Since the network layer to be adjusted is not a grouped convolution, the value of "group" here is 1.

[0060] When adjusting the intrinsic parameters of a network layer in a convolutional layer, one can, for example, adjust them using the number of data frames in the data to be processed by the neural network to be deployed. For example, the number of data frames to be processed may be 8 frames, 10 frames, 15 frames, etc., which may vary depending on the actual neural network input; correspondingly, for ease of description, the number of data frames is represented as a batch.

[0061] Specifically, when adjusting the operator type of a convolutional layer using the data frame batch size, the operator type can be changed from a single convolutional operator to a grouped convolutional operator. Correspondingly, when adjusting the intrinsic parameters of the network layer on a tensor basis, a first group size `group1` can be added to the intrinsic parameters of the convolutional layer. Here, the first group size `group1` is the same as the data frame batch size, i.e., `group1 = batch`. In other words, when a convolutional layer is adjusted to grouped convolution, each group of convolutions actually performs the data processing task of the previous convolutional layer before adjustment.

[0062] Accordingly, since the convolutional layer is adjusted to grouped convolution, the number of the first input channels of the convolutional layer as a whole is... and the number of first output channels There will also be some changes. Specifically, by determining the corresponding number of input channels and output channels for each group of convolutions, the first target number of input channels for the adjusted convolutional layer can be determined. The number of first input channels is a multiple of the number of data frames per batch. First target output channel number The number of first output channels is a multiple of the number of data frames per batch. That is, after determining the first target number of input channels When, specifically, the following formula (1-1) is satisfied:

[0063]

[0064] Determine the first target number of output channels When, specifically, the following formula (1-2) is satisfied:

[0065]

[0066] For the convolution kernel in the convolutional layer, the height of the convolution kernel (e.g., denoted as...) is... ), and the kernel width (e.g., expressed as As for the convolutional layer, its size does not change accordingly when it is adjusted to a grouped convolution, so its value remains unchanged.

[0067] In another embodiment of this disclosure, in the convolutional layer, the corresponding network layer intrinsic parameters further include, for example, a first convolutional kernel corresponding to each first output channel. The convolutional kernel height and width of the first convolutional kernel are as described above. as well as The same applies. Correspondingly, before transforming the convolutional layer, the number of output channels and input channels corresponding to the first convolutional kernel are also the same as the network layer intrinsic parameters of the convolutional layer.

[0068] Determine the first target number of input channels and the number of output channels of the first target In this case, since the convolutional layer is modified to grouped convolution, the first convolutional kernel will also change accordingly based on the multiple groups.

[0069] Specifically, when adjusting the first convolution kernel, for example, the first convolution kernel can be amplified according to the number of the first groups in the grouped convolution to obtain the first target convolution kernel corresponding to each first target output channel.

[0070] For example, see Figure 2a The diagram shown is a schematic representation of processing data using a first convolutional kernel, according to an embodiment of this disclosure. Figure 2a This diagram illustrates the convolutional layer processing data without adjusting the first convolutional kernel. For clarity, a single video frame is used as the data to be processed. This data includes three channels, such as the common red, green, and blue (RGB) color channels, which are the first input channels described above. The case where it is 3. In this example, the number of the first output channels. The number is 4, corresponding to 4 convolutional kernels for processing the data. The number of layers corresponding to each first convolutional kernel is equal to the number of first input channels. The same applies. That is, the first convolutional kernels that process the data consist of four kernels, each with three convolutional layers. As explained above, the size of the first convolutional kernels and the values ​​of each element are not affected by the transformation of the convolutional layers into grouped convolutions; therefore, the size of each first convolutional kernel is not specified here. Figure 2a In this example, for ease of representation, the input and output data before and after convolution are represented by the same color, and different shades of gray are added to multiple first convolution kernels for differentiation.

[0071] exist Figure 2bThe diagram illustrates the processing of the image using a first-target convolutional kernel; in this diagram, the number of first-target input channels is shown after the convolutional layer is transformed into a grouped convolution. and the number of output channels of the first target And the corresponding first target convolutional kernel. Where the data frame batch is 2, then the corresponding number of first groups, group1, is 2. Based on the above explanation, it can be determined that the number of first input channels... The first target number of input channels obtained after the transformation The number is 6, for the first input channel. The first target number of output channels obtained after the transformation It is 8. In Figure 2b To clearly illustrate this, the input and output channels belonging to the same video frame image will be given the same color. And for the first target input channel number... When processing the two corresponding video frame images, the first convolution kernel needs to be amplified so that the first convolution kernel, which is used to convolve one video frame image, is amplified into the first target convolution kernel required to convolve two video frame images respectively.

[0072] Among them, according to Figure 2b The diagram clearly shows that, from the perspective of the number of input channels, when amplifying the first convolutional kernel, it actually involves amplifying each of the individual first convolutional kernels within the first convolutional kernel, and then merging the resulting convolutional kernel layers belonging to different convolutional kernels to obtain a new convolutional kernel, which constitutes the first target convolutional kernel. From the perspective of the output channels, when amplifying the first convolutional kernel, it actually involves amplifying the first convolutional kernels corresponding to different output channels to obtain multiple convolutional kernels under one channel, which is the first target convolutional kernel.

[0073] In this embodiment of the disclosure, if a convolutional layer is selected as the network layer to be adjusted, the target network layer can be determined based on, for example, the operator type of the grouped convolution, the first target number of input channels, the first target number of output channels, and the first target convolution kernel.

[0074] In specific implementation, the following method can be used to generate the target network layer: Based on the operator type changed to grouped convolution, determine the operator corresponding to the target network layer; and based on the first target number of input channels and the first target number of output channels, determine the input channels and output channels for the operator corresponding to the target network layer, and determine the corresponding first target convolution kernel for each output channel and the corresponding input data for each input channel to obtain the target network layer. In this way, the target network layer can be constructed accordingly using the determined operator type and network layer intrinsic parameters.

[0075] (B): The network layer to be adjusted includes a batch normalization layer connected to the output of the convolutional layer.

[0076] In cases where the network layer to be adjusted includes a batch normalization layer connected to the output of the convolutional layer, before adjusting the batch normalization layer, the batch normalization layer and the convolutional layer can be merged in the following manner: based on the first relationship information between the first input data and the first output data of the convolutional layer, and the second relationship information between the second input data and the second output data of the batch normalization layer, a third relationship information is determined between the first input data of the convolutional layer and the second output data of the batch normalization layer; based on the third relationship information, the batch normalization layer is merged into the convolutional layer to obtain a new convolutional layer.

[0077] The first relationship information between the first input data and the first output data of the convolutional layer can be represented by, for example, the following formula (2-1):

[0078] y = wx + b (2-1)

[0079] Where x represents the first input data, y represents the first output data, w represents the weight term determined according to the first convolution kernel, and b represents the bias term coefficient.

[0080] For the batch normalization layer, the second relationship information between the second input data and the second output data of the batch normalization layer can be represented by, for example, the following formula (2-2):

[0081]

[0082] Where y represents the second input data and y' represents the second output data; here, the second input data is also the first output data described in formula (2-1) above. Since formula (2-1) expresses the first relational information of the convolutional layer and formula (2-2) expresses the second relational information of the normalization layer, y is named differently in different network layers. The remaining values ​​γ represent the scaling factor, σ represent the variance of the second input data, μ represent the mean of the second input data, and β represent the translation factor.

[0083] Using the first and second relational information described above, a third relational information can be determined accordingly between the first input data x of the convolutional layer and the batch-normalized second output data y'.

[0084] Specifically, for example, the first output data y in formula (2-1) can be substituted into formula (2-2) to obtain the following formula (2-3):

[0085]

[0086] By rearranging the obtained formula (2-3), we can obtain the third relationship information between the first input data x and the second output data y', which is expressed by the following formula (2-4):

[0087]

[0088] After determining the third relation information according to the above formula (2-4), based on the third relation information i, it can be determined that when merging the batch normalization layer into the convolutional layer, the following method can be adopted: based on the third relation information, the first convolution kernel and / or bias term coefficients of the convolutional layer are adjusted to obtain a new first convolution kernel and / or a new bias term coefficient.

[0089] Specifically, by comparing formula (2-4) with formula (2-1), it can be seen that when batch normalized layers are merged into convolutional layers, the weight term corresponding to batch normalization is adjusted to γw / σ, and the bias term coefficient is adjusted to γb / σ-γμ / σ+β. Thus, batch normalized layers can be merged into convolutional layers. When merging batch normalized layers into convolutional layers, since the batch normalized layers have their corresponding weight terms and bias term coefficients, the weight terms and bias term coefficients determined for merging batch normalized layers into convolutional layers can be used to adjust the first convolutional kernel and bias term coefficients of the convolutional layer, resulting in a new first convolutional kernel and new bias term coefficients, thereby obtaining a new convolutional layer.

[0090] In this embodiment of the disclosure, if a batch normalization layer is selected as the network layer to be adjusted, then when determining the target network layer, for example, the operator type and / or network layer intrinsic parameters of the new convolutional layer can be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed, so as to obtain the target network layer.

[0091] Based on the above description of merging batch normalization layers into convolutional layers, it can be understood that merging batch normalization layers essentially involves determining a new convolutional layer based on the architecture of the convolutional layers. Thus, the new convolutional layer determined by the batch normalization layers can be adjusted according to the adjustment method described in (A) above. In specific implementation, the first convolution kernel in the operator corresponding to the convolutional layer can be replaced with the new first convolution kernel, and / or the bias term coefficients in the operator corresponding to the convolutional layer can be replaced with the new bias term coefficients to obtain the new convolutional layer.

[0092] (C): The network layers to be adjusted include fully connected layers.

[0093] In cases where the network layer to be adjusted includes a fully connected layer, the network layer intrinsic parameters may include fully connected weights.

[0094] When adjusting the intrinsic parameters of the fully connected layer, for example, the following method can be used: based on the number of data frames of the data to be processed in the neural network to be deployed, change the operator type of the fully connected layer to grouped convolution, and change the fully connected weights of the fully connected layer to the convolution kernels of the convolution operator to obtain the target network layer; wherein, the number of the second group when changing the operator type of the fully connected layer to grouped convolution is the same as the number of data frames.

[0095] Following the example in (A) above, the number of data frames to be processed in the neural network to be deployed is, for example, batch. When adjusting the fully connected layer, similar to the batch normalization layer in (B) above, the fully connected layer is converted into a convolutional layer, specifically adjusting the operator type and weights of the fully connected layer. After changing the operator type in the fully connected layer to grouped convolution, the number of the second group in the grouped convolution (denoted as group2) is the same as the number of data frames, that is, group2 = batch.

[0096] In specific implementation, the fully connected layer can be adjusted in the following way to obtain the target network layer corresponding to the fully connected layer: based on the number of data frames of the data to be processed in the neural network to be deployed, the number of the second input channels and the number of the second output channels of the fully connected layer are adjusted to obtain the second target number of input channels and the second target number of output channels after the operator type of the fully connected layer is changed to grouped convolution; and based on the second number of groups, a second convolution kernel is generated after the operator type of the fully connected layer is changed to grouped convolution; the number of the second convolution kernels is the same as the second target number of output channels.

[0097] The number of second input channels in a fully connected layer is, for example, expressed as: The number of second output channels is, for example, expressed as: Using the number of data frames to determine the number of the second input channels When making adjustments to obtain the second target number of input channels, the following formula (3-1) can be used, for example:

[0098]

[0099] in, This indicates the number of input channels for the second target.

[0100] Regarding the second output channel number When making adjustments to obtain the second target number of output channels, the following formula (3-2) can be used, for example:

[0101]

[0102] in, This indicates the number of output channels for the second target.

[0103] Furthermore, the number of data frames to be processed is batch. Using the batch data frame number, the operator type of the fully connected layer can be changed from a fully connected operator to a grouped convolution operator. Correspondingly, when merging fully connected layers into grouped convolutions, for example, a second convolution kernel corresponding to the grouped convolution can be generated based on the second number of groups. Similar to (A) above, the number of second convolution kernels is the same as the number of second target output channels. For a detailed explanation of the process, please refer to the explanation in (A) above; it will not be repeated here.

[0104] In addition to the target network layers described in (A), (B) and (C) above, another embodiment of this disclosure further includes an input data fusion layer added to the input end of the neural network to be deployed; wherein the input data fusion layer is used to fuse multiple frames of data to be processed into a target data to be processed of a preset number of frames.

[0105] In practical implementation, the parameters of the multiple frames of data to be processed may include, for example, the number of data frames and the number of channels, height, and width of each frame of data to be processed, as described above. When the input data fusion layer fuses the multiple frames of data to be processed, it can, for example, transform the number of channels of the multiple frames of data to be processed into a multiple of the number of data frames. This is equivalent to transforming the number of data frames into 1, thereby expanding the dimension of the number of data frames to include the number of channels, thus achieving the fusion of multiple frames of data to be processed into 1 frame of data to be processed. Accordingly, since the first dimension of the data to be processed can be fixed, it can be applied to the NNIE platform.

[0106] This allows us to determine the target network layer.

[0107] Regarding S103 above, given the target network layer, the neural network can be reconstructed based on the obtained target network layer to obtain the target neural network. Since each network layer is designed according to the deployment requirements on the NNIE platform, the resulting target neural network is more suitable for deployment on the NNIE platform.

[0108] In another embodiment of this disclosure, a specific embodiment of a method for generating a neural network is also provided. In this embodiment, the various network layers mentioned above are illustrated by way of example.

[0109] In this embodiment, for the convolutional layer in (A) above, the intrinsic parameters of the convolutional layer may include, for example, defined parameters including the number of first input channels and the number of first output channels, such as the number of first input channels. Number of first output channels Kernel height kernel width And the number of groups. An exemplary definition of the parameter's value can be expressed as shown in the following formula (4-1):

[0110]

[0111] When adjusting the intrinsic parameters of the convolutional layer, the number of the first input channels is adjusted using the number of data frames per batch. Number of first output channels The number of groups is adjusted accordingly. This adjustment can be simplified by multiplying each group by the number of data frames (batch) to obtain the first target number of input channels. First target output channel number And the number of the first group, group1. For example, with a data frame batch of 8, the transformed network layer intrinsic parameters can be expressed as the following formula (4-2):

[0112]

[0113] Additionally, the convolutional layer includes adjustments to the first convolutional kernel. The parameters corresponding to the first convolutional kernel may include, for example, the number of output channels, the number of input channels, the kernel height, and the kernel width. When the definition parameters of the convolutional layer are determined, the number of output channels of the convolutional kernel and the number of the first output channels are also determined. The same, therefore also expressed as The number of input channels of the convolution kernel and the number of the first input channels And related to the number of groups, if the convolutional layer is not transformed into a grouped convolution, and the number of groups is 1, then the number of input channels of the convolutional kernel is equal to the number of the first input channels. The kernel height and kernel width are the same as those expressed in formula (4-1) above. kernel width Consistent. For example, when specifically defining the first convolution kernel, it can be expressed as the following formula (4-3):

[0114]

[0115] When adjusting the first convolution kernel, similar to the adjustment of the first output channel number described above, the kernel number determined when defining the first convolution kernel can be adjusted accordingly. Perform the transformation, and accordingly convert the transformed... Represented as For example, the first target convolutional kernel obtained after the transformation can be expressed as the following formula (4-4):

[0116]

[0117] The batch normalization layer in (B) above has been explained in detail in the above description, and will not be further illustrated here.

[0118] Regarding the fully connected layer in (C) above, the intrinsic parameters of the fully connected layer may include, for example, defined parameters containing the second number of input channels and the second number of output channels, such as the first number of input channels. and the number of second output channels For example, the intrinsic parameters of a fully connected layer network can be expressed as follows (4-5):

[0119]

[0120] When adjusting the intrinsic parameters of the fully connected layer, for example, we can first expand the tensors based on the intrinsic parameters of the convolutional layer and adjust them to the form of the following formula (4-6):

[0121]

[0122] in, as well as This parameter is determined solely for adjusting fully connected layers to correspond with the intrinsic parameters of convolutional layers. It may not represent any specific meaning, but the subscript "FC" is added for distinction, indicating that the parameter corresponds to a fully connected layer. Furthermore, before the transformation, the corresponding number of groups is the same as that of the convolutional layer, denoted as "group," with a value of 1.

[0123] When adjusting the intrinsic parameters of the fully connected layer, since the fully connected layer has been transformed into a convolutional layer (i.e., a new convolutional layer is obtained), the adjustment process is the same as that for the convolutional layer, and will not be repeated here. After adjusting the fully connected layer, the transformed network layer intrinsic parameters can be expressed as, for example, in the form of the following formula (4-7):

[0124]

[0125] For the fully connected weights corresponding to the fully connected layer, they are correspondingly replaced with the convolution kernels of the convolution operator. Before the transformation, the parameters of the fully connected weights can be expressed in the form of the following formula (4-8):

[0126]

[0127] The reason why formula (4-8) is consistent with the form of (4-5) above is explained in the description of (4-3) above, and will not be repeated here. After adjusting the parameters of the fully connected weights, it can be expressed, for example, in the form of formula (4-9):

[0128]

[0129] Regarding the input data fusion layer described above, when processing multiple frames of objects to be processed, it will affect the feature maps corresponding to the convolutional layer and the fully connected layer.

[0130] For the feature map of a convolutional layer, the corresponding parameters may include, for example, the number of data frames, the number of channels, the height, and the width. For instance, the parameters corresponding to the feature map of a convolutional layer can be expressed in the form of the following formula (4-10):

[0131] [b Conv ,c Conv ,h Conv ,w Conv ]=[8,6,64,64] (4-10)

[0132] When the input data fusion layer fuses multiple frames of data to be processed, the feature maps of the convolutional layers are transformed into the form of the following formula (4-11):

[0133] [B Conv C Conv ,h Conv ,w Conv ]=[1,48,64,64] (4-11)

[0134] That is, the dimension of the data frame number is transformed to 1 to meet the actual needs of the NNIE platform, and the data in the first dimension is transformed to the second dimension channel.

[0135] Correspondingly, for the feature map of a fully connected layer, the corresponding parameters include, for example, the number of data frames and the number of channels. For instance, the parameters corresponding to the feature map of a fully connected layer, including the number of data frames and the number of channels, can be expressed in the form of the following formula (4-12):

[0136] [b FC ,c FC ]=[8,6] (4-12)

[0137] After transforming the feature map of the fully connected layer, the resulting feature map can be represented in the form of the following formula (4-13):

[0138] [B FC C FC ,h FC ,w FC ]=[1,48,1,1] (4-13)

[0139] In this way, for different network layers, the corresponding target network layer can be determined by transformation, so as to generate the target neural network using the target network layer.

[0140] In another embodiment of this disclosure, a specific implementation of applying the target neural network obtained by the neural network generation method provided in this disclosure to behavior recognition is also provided. Specifically, for scenarios requiring behavior recognition, such as safety detection of workers' behavior on a factory assembly line, in order to obtain behavior recognition results more accurately, it is generally necessary to perform behavior recognition on multiple consecutive video frame images to obtain the behavior recognition results of the target human body in the time dimension. In this case, the generation method provided in this disclosure can be used to obtain a target neural network for behavior recognition, such as an image recognition network. The image recognition network obtained by the generation method provided in this disclosure, on the one hand, does not affect the detection accuracy of the neural network, thus it can obtain behavior recognition results more accurately; on the other hand, it can be directly deployed on the NNIE platform to reduce equipment costs.

[0141] The above embodiments only provide one possible application scenario. Other application scenarios that require feature analysis in the time dimension and / or image feature analysis on images, such as behavior prediction scenarios, are also within the scope of protection of this application.

[0142] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0143] Based on the same inventive concept, this disclosure also provides a neural network generation device corresponding to the neural network generation method. Since the principle of the device in this disclosure for solving the problem is similar to the neural network generation method described above in this disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0144] Reference Figure 3 The diagram shown is a schematic representation of a neural network generation apparatus according to an embodiment of this disclosure. The apparatus includes: a determining module 31, an adjusting module 32, and a generating module 33; wherein,

[0145] The determination module 31 is used to determine the network layer to be adjusted based on the operator type corresponding to each network layer in the neural network to be deployed.

[0146] The adjustment module 32 is used to adjust at least one of the operator type and network layer intrinsic parameters of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed, so as to obtain the target network layer.

[0147] The generation module 33 is used to generate a target neural network based on the target network layer.

[0148] In one optional implementation, the network layer to be adjusted includes a convolutional layer; the intrinsic parameters of the network layer include a first number of input channels and a first number of output channels; when the adjustment module 32 adjusts the operator type and / or intrinsic parameters of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain a target network layer, it is used to: change the operator type of the convolutional layer to grouped convolution based on the number of data frames of the data to be processed in the neural network to be deployed; the first group number of the grouped convolution is the same as the number of data frames; and adjust the first number of input channels and the first number of output channels of the convolutional layer based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain a first target number of input channels and a first target number of output channels; and construct the target network layer based on the operator type changed to grouped convolution, the first target number of input channels, and the first target number of output channels.

[0149] In an optional implementation, the intrinsic parameters of the network layer further include: a first convolutional kernel corresponding to each first output channel; after adjusting the number of first input channels and the number of first output channels of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed, the adjustment module 32 is further configured to: amplify the first convolutional kernel based on the number of first groups of the grouped convolution to obtain a first target convolutional kernel corresponding to each first target output channel; when the adjustment module 32 constructs the target network layer based on the operator type changed to grouped convolution, the number of first target input channels, and the number of first target output channels, the target network layer is constructed based on the operator type changed to grouped convolution, the number of first target input channels, the number of first target output channels, and the first target convolutional kernel.

[0150] In one optional implementation, when the adjustment module 32 constructs the target network layer based on the operator type changed to grouped convolution, the first target number of input channels, the first target number of output channels, and the first target convolution kernel, it is configured to: determine the operator corresponding to the target network layer based on the operator type changed to grouped convolution; and, based on the first target number of input channels and the first target number of output channels, determine the input channels and output channels for the operator corresponding to the target network layer, and determine the corresponding first target convolution kernel for each output channel and the corresponding input data for each input channel to obtain the target network layer.

[0151] In an optional implementation, the network layer to be adjusted further includes a batch normalization layer connected to the output of the convolutional layer; before the adjustment module 32 adjusts the operator type and / or network layer intrinsic parameters of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed to obtain the target network layer, it is further configured to: determine a third relationship information between the first input data of the convolutional layer and the second output data of the batch normalization layer based on the first relationship information between the first input data and the first output data of the convolutional layer, and the second relationship information between the second input data and the second output data of the batch normalization layer; merge the batch normalization layer into the convolutional layer based on the third relationship information to obtain a new convolutional layer; when the adjustment module 32 adjusts at least one of the operator type and network layer intrinsic parameters of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed to obtain the target network layer, it is configured to: adjust at least one of the operator type and network layer intrinsic parameters of the new convolutional layer based on the number of data frames of the data to be processed in the neural network to be deployed to obtain the target network layer.

[0152] In one optional implementation, when the adjustment module 32 merges the batch normalization layer into the convolutional layer based on the third relation information to obtain a new convolutional layer, it is configured to: adjust at least one of the first convolutional kernel and the bias term coefficient of the convolutional layer based on the third relation information to obtain a new first convolutional kernel and / or a new bias term coefficient; and obtain the new convolutional layer based on the new first convolutional kernel and / or the new bias term coefficient.

[0153] In one optional implementation, when the adjustment module 32 obtains the new convolutional layer based on the new first convolutional kernel and / or the new bias term coefficients, it is configured to: replace the first convolutional kernel in the operator corresponding to the convolutional layer with the new first convolutional kernel, and / or replace the bias term coefficients in the operator corresponding to the convolutional layer with the new bias term coefficients, thereby obtaining the new convolutional layer.

[0154] In one optional implementation, the network layer to be adjusted further includes: a fully connected layer; the network layer intrinsic parameters include: fully connected weights; when the adjustment module 32 adjusts at least one of the operator type and network layer intrinsic parameters of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain the target network layer, it is used to: change the operator type of the fully connected layer to grouped convolution based on the number of data frames of the data to be processed in the neural network to be deployed, and change the fully connected weights of the fully connected layer to the convolution kernel of the convolution operator, to obtain the target network layer; wherein, the number of the second group to which the operator type of the fully connected layer is changed to grouped convolution is the same as the number of data frames.

[0155] In one optional implementation, when the adjustment module 32 changes the operator type of the fully connected layer to grouped convolution and changes the fully connected weights of the fully connected layer to the convolution kernels of the convolution operator based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain the target network layer, it is configured to: adjust the number of second input channels and the number of second output channels of the fully connected layer based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain the second target number of input channels and the second target number of output channels after the operator type of the fully connected layer is changed to grouped convolution; and generate a second convolution kernel after the operator type of the fully connected layer is changed to grouped convolution based on the second number of groups; the number of the second convolution kernels is the same as the second target number of output channels.

[0156] In an optional implementation, the adjustment module 32 is further configured to: add an input data fusion layer at the input end of the neural network to be deployed; the input data fusion layer is configured to fuse multiple frames of data to be processed into a target number of frames of data to be processed.

[0157] In one optional implementation, the target neural network includes an image recognition network, which is used to perform behavior recognition on an image containing a target human body to obtain a behavior recognition result of the target human body in the image.

[0158] In one optional implementation, after generating the target neural network, the generation module 33 is further configured to: deploy the target neural network on the neural network inference engine NNIE.

[0159] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0160] This disclosure also provides a computer device, such as... Figure 4 The diagram shown is a schematic representation of a computer device structure provided in an embodiment of this disclosure, comprising at least:

[0161] A processor 10 and a memory 20 are connected via a bus; the memory 20 stores machine-readable instructions executable by the processor 10, and the processor 10 executes the machine-readable instructions stored in the memory 20. When the machine-readable instructions are executed by the processor 10, the processor 10 performs the following steps:

[0162] Based on the operator types corresponding to each network layer in the neural network to be deployed, the network layer to be adjusted is determined; based on the number of data frames of the data to be processed in the neural network to be deployed, at least one of the operator types and intrinsic parameters of the network layer to be adjusted is adjusted to obtain the target network layer; based on the target network layer, the target neural network is generated.

[0163] The aforementioned memory 20 includes a main memory 210 and an external memory 220; the main memory 210, also known as internal memory, is used to temporarily store the computational data in the processor 10, as well as the data exchanged with external memory 220 such as a hard disk. The processor 10 exchanges data with the external memory 220 through the main memory 210.

[0164] The specific execution process of the above instructions can be referred to the steps of the neural network generation method described in the embodiments of this disclosure, and will not be repeated here.

[0165] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the neural network generation method described in the above method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0166] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the neural network generation method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0167] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0168] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0169] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0170] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0171] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0172] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A method for generating a neural network, characterized in that, include: Based on the operator types corresponding to each network layer in the neural network to be deployed, determine the network layers to be adjusted; Based on the number of data frames of the data to be processed in the neural network to be deployed, at least one of the operator type and intrinsic parameters of the network layer to be adjusted is obtained to obtain the target network layer; based on the target network layer, the target neural network is generated. The target neural network includes an image recognition network, which is used to perform behavior recognition on an image containing a target human body to obtain a behavior recognition result of the target human body in the image. The network layer to be adjusted includes a convolutional layer; the intrinsic parameters of the network layer include a first number of input channels and a first number of output channels; adjusting the operator type and / or intrinsic parameters of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain a target network layer, includes: changing the operator type of the convolutional layer to grouped convolution based on the number of data frames of the data to be processed in the neural network to be deployed; the first group number of the grouped convolution is the same as the number of data frames; and adjusting the first number of input channels and the first number of output channels of the convolutional layer based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain a first target number of input channels and a first target number of output channels; the target network layer is constructed based on the operator type changed to grouped convolution, the first target number of input channels, and the first target number of output channels.

2. The method according to claim 1, characterized in that, The network layer intrinsic parameters also include: Each first output channel corresponds to a first convolutional kernel; after adjusting the number of first input channels and the number of first output channels of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain the first target number of input channels and the first target number of output channels, the method further includes: Based on the first number of groups in the grouped convolution, the first convolution kernel is amplified to obtain a first target convolution kernel corresponding to each first target output channel; the formation of the target network layer based on the operator type changed to grouped convolution, the number of first target input channels, and the number of first target output channels includes: The target network layer is constructed based on the operator type changed to grouped convolution, the first target number of input channels, the first target number of output channels, and the first target convolution kernel.

3. The method according to claim 2, characterized in that, The method of constructing the target network layer based on the operator type changed to grouped convolution, the first target number of input channels, the first target number of output channels, and the first target convolution kernel includes: determining the operator corresponding to the target network layer based on the operator type changed to grouped convolution; and determining the input channels and output channels for the operator corresponding to the target network layer based on the first target number of input channels and the first target number of output channels, and determining the corresponding first target convolution kernel for each output channel and the corresponding input data for each input channel to obtain the target network layer.

4. The method according to claim 3, characterized in that, The network layer to be adjusted further includes: a batch normalization layer connected to the output of the convolutional layer; before adjusting the operator type and / or network layer intrinsic parameters of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed, to obtain the target network layer, the following steps are also included: Based on the first relationship information between the first input data and the first output data of the convolutional layer, and the second relationship information between the second input data and the second output data of the batch normalization layer, a third relationship information is determined between the first input data of the convolutional layer and the second output data mapped to the batch normalization layer; based on the third relationship information, the batch normalization layer is merged into the convolutional layer to obtain a new convolutional layer. The process of adjusting at least one of the operator type and intrinsic parameters of the network layer to be adjusted based on the number of data frames of the data to be processed from the neural network to be deployed, to obtain the target network layer, includes: Based on the number of data frames of the data to be processed in the neural network to be deployed, at least one of the operator type and network layer intrinsic parameters of the new convolutional layer is adjusted to obtain the target network layer.

5. The method according to claim 4, characterized in that, The step of merging the batch normalization layer into the convolutional layer based on the third relation information to obtain a new convolutional layer includes: adjusting at least one of the first convolutional kernel and bias term coefficients of the convolutional layer based on the third relation information to obtain a new first convolutional kernel and / or a new bias term coefficient; and obtaining the new convolutional layer based on the new first convolutional kernel and / or the new bias term coefficient.

6. The method according to claim 5, characterized in that, The step of obtaining the new convolutional layer based on the new first convolutional kernel and / or the new bias term coefficients includes: replacing the first convolutional kernel in the operator corresponding to the convolutional layer with the new first convolutional kernel, and / or replacing the bias term coefficients in the operator corresponding to the convolutional layer with the new bias term coefficients, thereby obtaining the new convolutional layer.

7. The method according to claim 6, characterized in that, The network layer to be adjusted further includes: a fully connected layer; the intrinsic parameters of the network layer include: fully connected weights; the number of data frames of the data to be processed based on the neural network to be deployed, and the adjustment of the network layer to be adjusted. The target network layer is obtained by adjusting at least one of the operator types and intrinsic parameters of the entire network layer, including: Based on the number of data frames of the data to be processed in the neural network to be deployed, the operator type of the fully connected layer is changed to grouped convolution, and the fully connected weights of the fully connected layer are changed to the convolution kernels of the convolution operator to obtain the target network layer; wherein, the number of the second groups to which the operator type of the fully connected layer is changed to grouped convolution is the same as the number of data frames.

8. The method according to claim 7, characterized in that, The number of data frames of the data to be processed based on the neural network to be deployed, changing the operator type of the fully connected layer to grouped convolution, and changing the fully connected weights of the fully connected layer to the convolution kernels of the convolution operator, to obtain the target network layer, includes: Based on the number of data frames of the data to be processed in the neural network to be deployed, the number of the second input channels and the number of the second output channels of the fully connected layer are adjusted to obtain the number of the second target input channels and the number of the second target output channels after the operator type of the fully connected layer is changed to grouped convolution; and, based on the number of the second groups, a second convolution kernel after the operator type of the fully connected layer is changed to grouped convolution is generated; the number of the second convolution kernels is the same as the number of the second target output channels.

9. The method according to claim 8, characterized in that, Also includes: An input data fusion layer is added to the input end of the neural network to be deployed; the input data fusion layer is used to fuse multiple frames of data to be processed into a target number of frames of data to be processed.

10. The method according to claim 9, characterized in that, After generating the target neural network, the method further includes: The target neural network is deployed on the Neural Network Inference Engine (NNIE).

11. A neural network generation device, characterized in that, The apparatus includes a module for implementing the neural network generation method as described in any one of claims 1 to 10, comprising: The determination module is used to determine the network layer to be adjusted based on the operator type corresponding to each network layer in the neural network to be deployed; the adjustment module is used to adjust at least one of the operator type and network layer intrinsic parameters of the network layer to be adjusted based on the number of data frames of the data to be processed in the neural network to be deployed, so as to obtain the target network layer; the generation module is used to generate the target neural network based on the target network layer. The target neural network includes an image recognition network, which is used to perform behavior recognition on images containing target human bodies to obtain behavior recognition results of the target human bodies in the images.

12. A computer device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, the processor executing the machine-readable instructions stored in the memory, wherein when the machine-readable instructions are executed by the processor, the processor performs the steps of the neural network generation method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer device, performs the steps of the neural network generation method as described in any one of claims 1 to 10.

Images