Hardware constraint oriented target detection backbone network search method and device

Abstract

The application belongs to the technical field of image detection, and discloses a hardware constraint-oriented target detection backbone network searching method and equipment. The structure of the backbone network comprises an initial convolution layer and a plurality of feature extraction layers connected in series. Each feature extraction layer comprises a context extraction module connected in series. The searching method comprises the following steps: under the premise of meeting constraint conditions, taking the maximum multi-scale differential entropy as a searching target, and searching for a backbone network with optimal hyperparameters in a network searching space. The constraint conditions comprise that the calculation amount of the backbone network and the time consumption of the backbone network on a hardware platform are both less than corresponding preset thresholds. The multi-scale differential entropy is a weighted result of differential entropies of output feature maps of the plurality of feature extraction layers located at the last end of the backbone network. The above method can quickly search for a target detection backbone network which is adapted to different hardware platforms and can guarantee algorithm real-time performance and feature extraction effectiveness.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image detection, and more particularly relates to a hardware constraint-oriented target detection backbone network search method and device. BACKGROUND

[0002] In a target detection algorithm, a backbone network, as a core component of image feature extraction, is responsible for constructing a multi-scale feature representation, providing an input with rich semantic information for a subsequent feature fusion module and a detection head, and plays a decisive role in the final performance of a downstream task such as target detection. It is of great significance to construct a backbone network with strong context information aggregation capability to improve the accuracy and robustness of detection.

[0003] A target detection algorithm is usually required to be deployed on an edge computing device in actual application. Such a device usually faces problems such as limited computing resources, memory bandwidth bottlenecks, and strict power consumption constraints. Therefore, when designing a backbone network, the hardware constraints of the hardware platform need to be considered to ensure the real-time performance of the algorithm and the effectiveness of feature extraction under the condition of meeting the hardware constraints.

[0004] The model inference time efficiency of the same hardware platform is highly coupled with hardware computing resources, storage architecture, operator implementation, driver version, and operating system version. This high degree of software and hardware coupling makes the algorithm design process of directly optimizing for a specific hardware extremely complex and inefficient.

[0005] Therefore, it is urgent to propose an automatic search method for a target detection backbone network, which can quickly search for a target detection backbone network that is adapted to the hardware resources of different hardware platforms and can ensure the real-time performance of the algorithm and the effectiveness of feature extraction. SUMMARY

[0006] In view of the above defects or improvement needs of the prior art, the present application provides a hardware constraint-oriented target detection backbone network search method and device, which aims to quickly search for a target detection backbone network that is adapted to the hardware resources of different hardware platforms and can ensure the real-time performance of the algorithm and the effectiveness of feature extraction.

[0007] To achieve the above-mentioned purpose, the present application is proposed.

[0008] According to a first aspect of the present application, a hardware constraint-oriented target detection backbone network search method is provided, the structure of the backbone network comprises an initial convolutional layer and a plurality of feature extraction layers connected in series to realize pyramid-type multi-scale feature extraction, each feature extraction layer comprises a context extraction module connected in series, the hyperparameters to be optimized of the backbone network include the number of channels of each layer feature extraction layer, the number of context extraction modules contained in each layer feature extraction layer, and the convolution size in each context extraction module.

[0009] The search method comprises:

[0010] Under the premise of meeting the constraint condition, the multi-scale differential entropy maximum is taken as the search target, and the backbone network with optimal hyperparameters is searched in the network search space;

[0011] The constraint condition includes that the computational complexity of the backbone network and the time consumption thereof on the hardware platform are both less than the corresponding preset threshold; and the multi-scale differential entropy is the weighted result of the differential entropy of each output feature map of the plurality of feature extraction layers located at the last end of the backbone network.

[0012] According to a second aspect of the present application, a hardware constraint-oriented target detection backbone network search device is provided, comprising a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method according to any one of the above when executing the computer program.

[0013] Overall, compared with the prior art, the above technical solutions conceived by the present application mainly have the following beneficial effects.

[0014] 1. The target detection backbone network search method proposed in the present application, on the one hand, the present application takes the number of channels, the number of context extraction modules and the convolution size as the hyperparameters to be optimized, these three types of hyperparameters are the key factors affecting the network complexity and feature extraction, therefore, optimization of the above three types of hyperparameters can improve the search speed and quickly search the backbone network meeting the conditions; on the other hand, the present application constrains the computational complexity of the backbone network and the time consumption thereof on the hardware platform, through the constraint condition, it can be ensured that the backbone network meets the hardware resource limitation and can guarantee the real-time performance of the network on the hardware platform; moreover, the present application takes the maximum multi-scale differential entropy of the backbone network as the search target, which can balance the performance of each scale feature of the backbone network in the network search process, improve the performance of different scale target detection, guarantee the effectiveness of feature extraction, and then improve the detection accuracy of the target detection network.

[0015] 2. Further, in some embodiments, the context extraction module is a strip context extraction module, which can efficiently extract the extended features of long strip targets such as bridges, seaports, and airport runways, and has stronger adaptability to the changes in the aspect ratio of the target.

[0016] 3. Further, in some embodiments, the channel number search space of the feature extraction layer and the number search space of the context extraction module are both set according to the hardware resource characteristics, further improving the matching degree of the backbone network and the hardware platform.

[0017] 4. Further, in some embodiments, a specific calculation method of the multi-scale differential entropy is provided, which can quickly and accurately calculate the multi-scale differential entropy of different backbone networks.

[0018] 5. Further, in some embodiments, a genetic algorithm is used as the search method, which can quickly search for the best backbone network. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a step flowchart of the hardware constraint-oriented target detection backbone network search method in an embodiment of the present application.

[0020] Figure 2 is a structural schematic diagram of the target detection backbone network in an embodiment of the present application.

[0021] Figure 3 is a comparison diagram of the strip convolution operator and the standard convolution operator in an embodiment.

[0022] Figure 4 is a structural schematic diagram of the strip context extraction module in an embodiment of the present application.

[0023] Figure 5 is a search space schematic diagram of three to-be-optimized hyperparameters in an embodiment of the present application.

[0024] Figure 6 is a schematic diagram of the multi-scale differential entropy of the backbone network obtained by selecting the differential entropy of the output feature map of the feature extraction layer C2-C5 and performing weighted summation in an embodiment of the present application.

[0025] Figure 7 is a flowchart of the search using a genetic algorithm in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the accompanying drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and not used to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0027] Embodiment 1

[0028] The application discloses a hardware constraint-oriented target detection backbone network search method.

[0029] As Figure 1 The step flow chart of the hardware constraint-oriented target detection backbone network search method in the embodiment of the present application is shown in the figure, and the steps of the method will be described in detail below. Figure 1

[0030] S1, determine the structure of the target detection backbone network, the structure of the target detection backbone network includes an initial convolutional layer and a plurality of feature extraction layers connected in series to realize pyramid-type multi-scale feature extraction, each feature extraction layer includes a context extraction module connected in series, the hyperparameters of the backbone network to be optimized include the number of channels of each layer feature extraction layer, the number of context extraction modules contained in each layer feature extraction layer and the convolution size in each context extraction module.

[0031] Specifically, in the image target detection model, the backbone network is responsible for constructing multi-scale feature representation, providing input with rich semantic information for the subsequent feature fusion module (neck network) and detection head, and plays a decisive role in the final performance of the downstream tasks such as target detection. In the present application, as Figure 2 shown, the structure of the backbone network is determined in advance, i.e. the network architecture is determined, which includes an initial convolutional layer and a plurality of feature extraction layers connected in series, each feature extraction layer includes a context extraction module connected in series, therefore, the core operator of this network structure is the context extraction module, this network structure can extract multi-scale feature representation of the input image, then, the hyperparameters of the network structure are determined by the neural network search method, in the present application, the hyperparameters to be searched include the number of context extraction modules contained in each layer feature extraction layer, the number of channels of each layer feature extraction layer and the convolution size in each context extraction module.

[0032] Specifically, the initial convolutional layer can not participate in the hyperparameter search to reduce the search complexity, and its convolution size is directly set to preliminarily downsample its input feature map, for example, the downsample ratio of the initial convolutional layer is 1 / 2, and the initial feature dimension reduction is completed by a convolution with a size of 3x3 and a stride of 2.

[0033] ​Specifically, the down-sampling ratios of the plurality of feature extraction layers gradually decrease, and the down-sampling ratio between adjacent layers is 2 times, so that the backbone network realizes pyramid multi-scale feature extraction.

[0034] Specifically, the number of feature extraction layers can be 4, the down-sampling ratio of the initial convolution layer is 1 / 2, and the down-sampling ratios of the first to fourth feature extraction layers are 1 / 4, 1 / 8, 1 / 16 and 1 / 32, respectively. According to experience, 4 layers of feature extraction layers are set, the down-sampling ratio can reach 1 / 32, and better feature extraction can be realized. Increasing the number of feature extraction layers will increase the computational complexity and the feature extraction effect will not be significantly improved. Therefore, in general, 4 layers of feature extraction layers can be set. Therefore, the number of feature extraction layers and the down-sampling ratio of each layer can not participate in the hyperparameter search, and the parameters can be set in advance to reduce the search complexity.

[0035] In the present application, the hyperparameters to be searched include the number of context extraction modules included in each layer of feature extraction layer, the number of channels of each layer of feature extraction layer, and the convolution size in each context extraction module.

[0036] In specific embodiments, in order to regulate the down-sampling ratio and the number of channels, each layer of feature extraction layer is also provided with a convolution located at the front end of the context extraction module. The features input into the feature extraction layer are first subjected to down-sampling and channel adjustment by the convolution at the front end, and then input into the context extraction module to obtain the output features of the corresponding feature extraction layer, wherein the context extraction module performs feature extraction and the input and output channel numbers remain unchanged. At this time, the search for the number of channels of each layer of feature extraction layer is actually a search for the number of convolution kernels of the front-end convolution in each layer of feature extraction layer.

[0037] For the design of the context extraction module, it can be adjusted according to specific scenarios. In a conventional scenario, a conventional context extraction module can be used.

[0038] For application scenarios that require identification of high aspect ratio targets, in some embodiments, the context extraction module is also designed as follows.

[0039] Specifically, the context extraction module is designed as a strip context extraction module (SCM for short), i.e., the embodiment takes the strip context extraction module as a core operator. The strip context extraction module includes a batch normalization layer (BN layer) and a mixed convolution block connected in series. The mixed convolution block includes an orthogonal strip convolution and a depth separable square convolution connected in series. The BN layer is used to normalize the feature distribution and improve the training stability. The orthogonal strip convolution is used to expand the receptive field in a specific direction, thereby enhancing the stability of feature expression for different aspect ratio targets. The square convolution is responsible for capturing local feature information and optimizing spatial feature expression. More specifically, the orthogonal strip convolution is composed of a 1xk horizontal strip convolution and a kx1 vertical strip convolution, and k is a hyperparameter to be searched.

[0040] As a special convolution operator, the strip convolution (StripConv) is usually divided into horizontal strip convolution and vertical strip convolution. Its advantage lies in that it can capture long-distance spatial information in a specific direction. For example, in remote sensing images, it can efficiently extract the extension features of long strip targets such as bridges, seaports, and airport runways, and has stronger adaptability to the aspect ratio change of targets.

[0041] The strip convolution operator and the standard convolution operator are compared as shown in FIG. 1. Figure 3 As shown in FIG. 1, (a) is a standard convolution with a kernel size of 9, (b) is a vertical direction strip convolution with a kernel size of 9, and (c) is a horizontal direction convolution with a kernel size of 9. For a 9x9 standard convolution operator, after it is decomposed into 9x1 and 1x9 operators, the calculation complexity is reduced by 4.5 times while maintaining the long-distance feature extraction capability, which significantly improves the calculation efficiency. For targets with obvious direction attributes, the strip convolution can more effectively identify and extract the detailed features of these targets, thereby improving the overall detection and recognition performance. Compared with a square convolution kernel of the same size, the strip convolution has lower calculation amount and is suitable for scenarios with real-time or lightweight requirements.

[0042] For application scenarios such as remote sensing target detection that need to identify high aspect ratio targets, the embodiment introduces an orthogonal strip convolution as a core operator for context information extraction, and designs the above strip context extraction module SCM. The SCM can deeply fuse local and global, same direction and different direction spatial information, which is beneficial to improve the detection accuracy of high aspect ratio targets.

[0043] Optionally, in the hybrid convolutional block, the features input to the hybrid convolutional block are first expanded to the channel dimension by a 1×1 convolution at its front end, then extracted by concatenated orthogonal strip convolutions and depthwise separable square convolutions, and then multiplied by the output features of the front 1×1 convolution. Finally, they are compressed back to a lower channel number by a 1×1 convolution at its back end to obtain the output features of the corresponding hybrid convolutional block.

[0044] Optionally, the strip context extraction module can also introduce residual connections to alleviate the gradient vanishing problem and enhance the network's expressive power.

[0045] Optionally, the strip context extraction module can also introduce an inverse residual structure connected to the back end of the hybrid convolutional block. The inverse residual structure is used to first increase the channel dimension of the input features before extracting spatial features, and finally reduce the channel dimension to maintain the input and output channels unchanged. For example, in the inverse residual structure, the features input to the inverse residual structure are first expanded in channel dimension by a 1×1 convolution, then a 3×3 depthwise convolution is performed to extract spatial features, and finally, another 1×1 pointwise convolution projects the features from the high-dimensional space back to the lower output channel dimension. Through this inverse residual design of first increasing and then decreasing dimension, the inverse residual structure reduces computational complexity while preserving the expressive power of high-dimensional features.

[0046] The following is one specific design of the bar-shaped context extraction module: Figure 4 As shown, the features of the input strip context extraction module are first passed through a BN layer to normalize the feature distribution, and then extracted through a hybrid convolutional block. The features output by the hybrid convolutional block are connected to the features of the input strip context extraction module through a first residual connection, and then passed through a BN layer again to perform a second calibration of the feature distribution. They are then fed into an inverse residual network structure to perform feature transformation in the channel dimension, and then connected to the output features of the first residual connection through a second residual connection to obtain the output features of the strip context extraction module.

[0047] S2. Under the premise of satisfying the constraints, with the maximum multi-scale differential entropy as the search objective, search for the backbone network with the best hyperparameters in the network search space; wherein, the constraints include that the computational cost of the backbone network and its time consumption on the hardware platform do not exceed the corresponding preset threshold; the multi-scale differential entropy is the weighted result of the differential entropy of each output feature map of multiple feature extraction layers located at the end of the backbone network.

[0048] Specifically, the network search space is a preset search space, that is, each hyperparameter to be searched has a pre-set search space. By combining different hyperparameter values, candidate backbone networks that meet the constraints can be obtained. The backbone network that meets the search requirements needs to be selected from the candidate backbone networks according to the search objective.

[0049] Specifically, the search space for the number of channels in each feature extraction layer can be set according to the characteristics of the hardware platform to match the number of channels with the hardware platform. For example, the hardware architecture of the Ascend 310 chip has optimized support for multiples of 16 channels in its computing units. This hardware uses a parallel processing mechanism with 16 channels as a basic batch of operations. When the number of channels in the input data is less than 16, it will automatically perform zero-padding to round it to a multiple of 16. Although this zero-padding mechanism can ensure the execution of operations, it will waste computing resources. To fully utilize the hardware performance of the Ascend 310, the number of all candidate channels can be configured to be multiples of 16, reducing the additional overhead caused by automatic zero-padding and significantly improving inference efficiency. More specifically, the initial number of channels in the four feature extraction layers can be set to 64, 128, 256, and 512, respectively. Except for the initial convolutional layer with fixed input and output channels, the input and output channels of the other layers are scaled and expanded according to four multiples of the initial channels: {0.5, 0.75, 1, 1.5}.

[0050] Specifically, the search range for the number of context extraction modules included in each feature extraction layer can be set to a maximum value for the total number of context extraction modules in the backbone network based on hardware resource limitations. The search space for the number of context extraction modules contained in each feature extraction layer satisfies the following: the number of context extraction modules contained in each feature extraction layer is not less than 1 and the total number of context extraction modules contained in all feature extraction layers does not exceed 1. For example, it can be Set it to 20.

[0051] Specifically, the search space for the convolution size in each context extraction module is a trade-off between computational complexity and expanding the receptive field. Too large a size results in high computational complexity, while too small a size results in a small receptive field. This trade-off is between real-time performance and accuracy, and the search space can be determined based on experience or multiple experiments. For example, for the design of the strip context extraction module, the search space for the hyperparameter k of the orthogonal strip convolution can be set to {7, 11, 13}.

[0052] like Figure 5 The diagram shown is a search space diagram of the three hyperparameters to be optimized in a specific embodiment.

[0053] In this invention, it is also necessary to set constraints based on the resource limitations of the hardware platform, mainly including the computational load of the backbone network and its time consumption on the hardware platform. That is, the upper limit of computational load and the upper limit of time consumption are set according to the resource limitations of the hardware platform. The computational load of the backbone network cannot exceed the upper limit of computational load, and the time consumption of the backbone network on the hardware platform cannot exceed its upper limit of time consumption.

[0054] Specifically, the calculation amount of the model is an important indicator for measuring the complexity of the model, and the calculation amount of the backbone network can be measured by floating point operations per second (FLOP S ), which is related to the number of layers of the backbone network, the feature map size of each layer, the convolution kernel size, the number of channels, etc.

[0055] The calculation formula of FLOP S of different structures is given below.

[0056] The calculation formula of the floating point calculation amount FLOPs of the standard convolution is:

[0057] FLOPs conv = output feature map size x input channel number x output channel number x standard convolution size.

[0058] The calculation formula of the floating point calculation amount FLOPs StripConv of the strip convolution is:

[0059] FLOPs StripConv = output feature map size x input channel number x output channel number x strip convolution size.

[0060] The calculation formula of the floating point calculation amount FLOPs DSConv of the depth separable convolution is:

[0061] FLOPs DSConv = output feature map size x input channel number x (convolution size + output channel number).

[0062] Based on the above calculation formula, the calculation amount of different candidate backbone networks can be calculated.

[0063] For computer vision tasks such as object detection, a training and deployment separated manner is usually adopted, that is, network training is completed on a high-performance computing server, and the model obtained by training is converted into an inference model suitable for a hardware platform. In the neural network architecture search process, if only the calculation amount and other theoretical indicators are used as optimization constraints, and the actual inference performance of the hardware platform is ignored, the inference time of the network model obtained by searching may not meet the real-time requirement when the network model is deployed on the hardware platform. Based on this, the hardware platform inference time consumption constraint is introduced in the network architecture search, so that the final network has timeliness.

[0064] Specifically, since the core structure of the backbone network is actually a stacked context extraction module, the time consumption of each context extraction module (operator) in the search space can be estimated first, and then the time consumption of all context extraction modules in the candidate backbone network is added in the search process to obtain the time consumption of the candidate backbone network on the hardware platform.

[0065] Taking the hardware architecture of the Ascend 310 chip as an example, the inference time consumption statistical analysis is carried out on the Ascend 310 platform. The tool used is the Profiling performance analysis tool of the Ascend 310 platform, which can collect and analyze the key performance indicators of each stage running on the Ascend 310 processor, that is, the Profiling performance analysis tool is used for inference time consumption statistical analysis to obtain the actual running time consumption of the operators in the search space on the Ascend 310. The specific process of the tool collecting the time consumption of the operator (single context extraction module) is as follows: first, complete the installation of the CANN software, configure the corresponding environment, then compile the operator to be estimated into an executable program, call the msprof tool through the command line, specify the Profiling performance data landing path, and then obtain the Profiling performance data of the model after the model inference; finally, use the msprof command to parse and export the data, and the actual running time consumption of the operator on the Ascend 310 can be obtained.

[0066] In the present application, the search target is set to be the maximum multi-scale differential entropy of the backbone network, and the multi-scale differential entropy is the weighted result of the differential entropy of each output feature map of the plurality of feature extraction layers located at the last end in the backbone network.

[0067] The present application uses differential entropy as the network evaluation standard is described below.

[0068] In the field of machine learning, entropy (Entropy) as the core concept of information theory is an index for quantifying the uncertainty degree of the distribution of random variables. For discrete random variables, its entropy is defined as:

[0069]

[0070] wherein H(X) represents the entropy of the random variable X, , , represents the ith value of random variable X and its corresponding probability. The essence of entropy is to statistically calculate the expected value of the information amount of a random variable. When the research object is extended to continuous random variables, the discrete entropy needs to be continuously generalized by integral form, thus obtaining the concept of differential entropy. The maximum entropy principle, as an important modeling criterion, its core idea is that among all the probability models that meet the known constraints, the model with the maximum entropy is regarded as the optimal model.

[0071] For neural network models, the calculation process can be modeled as , i.e., mapping the input image to its label . And for network structure, it can be abstracted as a directed acyclic graph , where represents each neuron (i.e., each feature map) in the network, and represents the edges connecting the neurons, which are used to transmit information. For any and , let and represent the information value of each neuron and each edge , respectively, then the network continuous state space can be described by the set , and the feature extraction ability of the neural network model can be measured by the entropy of the set . including feature information entropy and network parameter information entropy , since in the backbone network search process, the main concern is the feature extraction ability of different candidate networks, therefore, the feature information is taken as the standard to measure the performance of different candidate networks, and only the calculation of is considered.

[0072] According to the Gaussian entropy upper bound theorem, among all continuous probability distributions with the same mean and variance , the differential entropy of Gaussian distribution is the largest. For a variable satisfying Gaussian distribution , the calculation formula of its differential entropy is as follows:

[0073]

[0074] From the formula, it can be analyzed that since the first term in the formula is a fixed value, the differential entropy of the variable ​ only depends on the second term, i.e. variance By controlling the value of variance, the differential entropy of the variable can be maximized.

[0075] Therefore, for a neural network , all the network weights of the operators can be initialized using a standard Gaussian distribution , the network input is an image filled with randomly generated standard Gaussian distribution noise, and after forward inference, the differential entropy of the feature map generated by the neural network can be obtained.

[0076] That is, in the search process, the process of calculating the multi-scale differential entropy of the current candidate backbone network includes:

[0077] The network weights in each context extraction module in the current candidate backbone network are initialized using a standard Gaussian distribution;

[0078] Randomly generated images filled with standard Gaussian distribution noise are input into the current candidate backbone network;

[0079] The differential entropy of the output feature map of the last multiple feature extraction layers in the current candidate backbone network is calculated and weighted to obtain the multi-scale differential entropy of the current candidate backbone network.

[0080] Further, the differential entropy of the output feature map of any feature extraction layer in the backbone network can be calculated as follows:

[0081] First, the differential entropy of each pixel position in the output feature map is calculated:

[0082]

[0083] In the formula, represents the output feature map, represents the variance of the output feature map , D represents the Dth feature extraction layer in the current candidate network, and is the Euclidean norm of the feature vector composed of the feature values of all output channels of the feature extraction layer at the pixel position.

[0084] Then, the differential entropy of all pixel positions in the output feature map is summed to obtain the differential entropy of the output feature map.

[0085] According to the above formula, by calculating the differential entropy of the output feature map corresponding to each candidate network, the differential entropy is used as the evaluation standard of network performance, and the candidate network with the maximum differential entropy searched can be used as the optimal network. ​

[0086] Considering the actual scene of image target detection, there are problems such as uneven target distribution and significant scale difference, and the network has high requirements for multi-scale feature representation capability. Therefore, the search method which simply maximizes the differential entropy of the final output layer of the network fails to fully consider the demand for cross-scale feature extraction of the detection task, and the backbone network searched by the search method may not be the optimal solution.

[0087] Therefore, the present application proposes to take the maximum multi-scale differential entropy as the search target, and the multi-scale differential entropy is the weighted result of the differential entropy of the output feature maps of the multiple feature extraction layers located at the last end in the backbone network. Taking the setting of four feature extraction layers as an example, the feature extraction layers of the backbone network can be denoted as As shown in Figure 6 , if only the differential entropy of the last feature C5 is calculated and the features of C3 and C4 are ignored, the backbone network will only focus on large-scale expression capability, resulting in that the finally searched backbone network is a suboptimal solution, so the differential entropy of each layer of the backbone network in the search process needs to be comprehensively processed to balance the feature expression capability of different scales and improve the performance of different scale target detection, therefore, the differential entropy of the output feature maps of the second to fourth feature extraction layers C2~C5 can be selected for weighted summation to obtain the multi-scale differential entropy of the backbone network.

[0088] In specific operation, for the different scale feature layers of the backbone network , the corresponding weights are , by weighting the differential entropy of the different scale feature layers, the performance of each scale feature of the backbone network in the network search process is balanced. Since in the calculation of the multi-scale entropy of the backbone network, the shallow part of the backbone network in the feature fusion process is discarded, that is, the weight is set to 0, thereby reducing the overall calculation amount of the network model. Specifically, can be respectively set to 1, 1, 6, and generally the weight of the last layer is the largest.

[0089] In an embodiment, a genetic algorithm can be used to search for a backbone network with optimal hyperparameters, and the execution process of the genetic algorithm includes:

[0090] Initialization: randomly generate multiple backbone networks satisfying the constraint conditions in the network search space as the initial population, and each backbone network is taken as a different individual;

[0091] Iteration: taking the multi-scale differential entropy as the fitness, the fitness of each individual in the current population is calculated, and under the premise of satisfying the constraint condition, the next generation population is obtained through selection, crossover and mutation operations according to the fitness; the iteration step is repeatedly executed until the iteration period reaches the upper limit of the iteration, and then the individual with the optimal fitness in the last generation population is output as the backbone network with optimal hyperparameters.

[0092] More specifically, when performing the crossover mutation operation, different operations are performed according to the iteration period:

[0093] When the iteration period is less than the switching threshold, a coarse mutation is performed, and the coarse mutation is a crossover mutation operation on all to-be-optimized hyperparameters;

[0094] When the iteration period is greater than or equal to the switching threshold, a fine mutation is performed, and the fine mutation is a crossover mutation on only the convolution size and the number of channels.

[0095] Specifically, as shown in Figure 7 the first N initial networks are randomly generated based on a preset search space, and each network meets a pre-set constraint condition, including the computational load of the network and the network estimated time consumption. In each iteration period, individuals are randomly selected from the population for mutation. Network mutation is divided into two stages. When the iteration period is before the set threshold period, a coarse mutation method is used for the network, and a crossover mutation operation is performed on the convolution kernel size, the number of network layers, and the module channel number. The crossover operation generates offspring by exchanging the architecture information of the two parent networks, and the mutation operation is performed on the candidate modules of the initial network. When the iteration period is greater than the set value, a fine mutation method is used for the network, and only the convolution kernel size and the channel number are mutated. For the mutated network, it is judged whether it meets the constraint condition. If it does not meet the constraint condition, the mutation operation is discarded. If the network meets the constraint condition, the mutated network is added to the next generation population, and the network with the smallest multi-scale entropy value in the population is discarded.

[0096] Based on the above method, the search of the hardware constraint-oriented target detection backbone network can be completed, and the best backbone network with hyperparameters is obtained.

[0097] It should be noted that the network search method only determines the hyperparameters of the backbone network. If target detection is required, a neck network that fuses the output feature maps of the feature extraction layer and a detection head located in the neck network need to be set to form a complete target detection network. In addition, the network needs to be trained using data in a specific scenario to determine the network weight parameters, and finally a target detection network that can realize target detection is obtained.

[0098] Embodiment 2

[0099] The application also relates to a hardware constraint-oriented target detection backbone network search device, which comprises a memory and a processor. The memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0100] The device can be a desktop computer, a notebook, a palm computer, a cloud server and the like. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. The memory can be used to store computer programs and / or modules, and the processor can realize various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory.

[0101] Overall, the above target detection backbone network search method, on the one hand, the application takes the number of channels, the number of context extraction modules and the convolution size as the super parameters to be optimized. These three types of super parameters are key factors affecting network complexity and feature extraction. Therefore, optimizing the above three types of super parameters can improve search speed and quickly search for a backbone network that meets the conditions. On the other hand, the application constrains the computational complexity of the backbone network and the time consumption on the hardware platform. Through this constraint condition, it can be ensured that the backbone network meets the hardware resource limit and can guarantee the real-time performance of the network on the hardware platform. Moreover, the application takes the maximum multi-scale differential entropy of the backbone network as the search target, which can balance the performance of the features of each scale of the backbone network in the network search process, improve the performance of target detection of different scales, guarantee the effectiveness of feature extraction, and then improve the detection accuracy of the target detection network. Through experiments on the hardware platform of the Ascend 310 chip, compared with manual design, the backbone network searched by the method proposed by the application is significantly reduced to 5.249M in parameter quantity, and the computational complexity is reduced to 13.913 GFLOPs. At the same time, the inference time on the Ascend 310 platform is also shortened to 34.45ms. The AP50 of the model on the DOTA v1 dataset reaches 79.25%, and the overall forward inference time of the network on the Ascend 310 platform is 60.16ms.

[0102] Any combination of the technical features in the above-described embodiments can be made, and for the sake of brevity, not all possible combinations are described, however, as long as there is no conflict, any combination of the technical features should be considered within the scope of the present disclosure. It should be noted that "in an embodiment of the present disclosure", "for example", "for instance", and the like are intended to serve as examples of the present disclosure, and are not intended to limit the present disclosure.

[0103] The above-described embodiments only express several implementation manners of the present disclosure, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the patent application. It should be noted that for those skilled in the art, without departing from the concept of the present disclosure, a number of modifications and improvements can be made, which are within the scope of protection of the present disclosure.

Claims

1. A hardware constraint oriented target detection backbone network search method, characterized in that, The structure of the backbone network comprises an initial convolutional layer and a plurality of feature extraction layers connected in series to realize pyramid multi-scale feature extraction, each of the feature extraction layers comprises a context extraction module connected in series, and the hyperparameters to be optimized of the backbone network comprise the number of channels of each layer of the feature extraction layer, the number of context extraction modules contained in each layer of the feature extraction layer, and the convolution size in each context extraction module; The search method comprises: Under the premise of meeting the constraint condition, a backbone network with optimal hyperparameters is searched in a network search space with multi-scale differential entropy maximization as the search target; The constraint condition comprises that the computational complexity of the backbone network and the time consumption of the backbone network on a hardware platform are both less than corresponding preset threshold values; and the multi-scale differential entropy is a weighted result of differential entropies of output feature maps of a plurality of feature extraction layers located at the last end in the backbone network. In the feature extraction layer, the context extraction module is a strip-shaped context extraction module, the strip-shaped context extraction module comprises a batch normalization layer and a mixed convolution block connected in series, and the mixed convolution block comprises an orthogonal strip-shaped convolution and a depth separable square convolution connected in series, wherein the orthogonal strip-shaped convolution is composed of a 1×k horizontal strip-shaped convolution and a k×1 vertical strip-shaped convolution, and k is a hyperparameter to be searched; The strip-shaped context extraction module further comprises a residual connection and an inverse residual structure, and the inverse residual structure is used to first perform channel dimension increasing on input features, then extract spatial features, and finally perform channel dimension reduction to keep the input and output channels unchanged; In the strip-shaped context extraction module: the features input into the strip-shaped context extraction module first pass through the BN layer, then pass through the mixed convolution block for feature extraction, the features output by the mixed convolution block are subjected to first layer residual connection with the features input into the strip-shaped context extraction module, then pass through the BN layer again, then pass through the inverse residual network structure, the output features of the inverse residual network structure are subjected to second layer residual connection with the output features of the first layer residual connection, and output features of the corresponding strip-shaped context extraction module are obtained; In the hyperparameters to be optimized: The candidate channel number given in the channel number search space of the feature extraction layer matches the hardware platform; The number of the context extraction modules included in the feature extraction layers satisfies: the number of the context extraction modules included in each layer of the feature extraction layers is not less than 1, and the total number of the context extraction modules included in all the feature extraction layers does not exceed a maximum value , the maximum value is set according to the limitation of the hardware resources.

2. The search method of claim 1, wherein, In the search process, the process of calculating the multi-scale differential entropy of the current candidate backbone network comprises: The network weights in each context extraction module in the current candidate backbone network are initialized using a standard Gaussian distribution; An image filled with randomly generated standard Gaussian distribution noise is input into the current candidate backbone network; The differential entropies of the output feature maps of the plurality of feature extraction layers located at the last end in the current candidate backbone network are calculated and weighted to obtain the multi-scale differential entropy of the current candidate backbone network.

3. The search method of claim 2, wherein, In the search process, the process of calculating the differential entropy of the output feature map of any feature extraction layer in the current candidate backbone network comprises: calculating a differential entropy for each pixel position in the output feature map : ； wherein, denotes the output feature map, denotes the output feature map denotes the variance of the output feature map, D denotes that the feature extraction layer is the Dth feature extraction layer in the current candidate network, is the Euclidean norm of the feature vector composed of the feature values of all output channels of the feature extraction layer at the pixel position; The differential entropies of all pixel positions in the output feature map are summed to obtain the differential entropy of the output feature map.

4. The search method of claim 1, wherein, The backbone network is arranged as four layers of feature extraction layers, and the multi-scale differential entropy is a weighted result of differential entropies of output feature maps of three feature extraction layers located at the last end in the backbone network.

5. The search method of claim 1, wherein, The genetic algorithm is used to search for the backbone network with optimal hyperparameters, and an execution process of the genetic algorithm comprises: initialization: a plurality of backbone networks meeting constraint conditions are randomly generated in a network search space as an initial population, and each backbone network is taken as a different individual; iteration: taking the multi-scale differential entropy as fitness, calculating the fitness of each individual in a current population, and under the premise of meeting the constraint condition, performing selection, crossover and mutation operations according to the fitness to obtain a next generation population; the iteration step is repeatedly executed until the iteration period reaches an iteration upper limit, and then an individual with optimal fitness in the last generation population is output as the backbone network with optimal hyperparameters.

6. The search method of claim 5, wherein, When the crossover and mutation operations are performed, different operations are performed according to the iteration period: when the iteration period is less than a switching threshold, a coarse mutation is performed, and the coarse mutation is a crossover and mutation operation on all hyperparameters to be optimized; when the iteration period is greater than or equal to the switching threshold, a fine mutation is performed, and the fine mutation is a crossover and mutation operation on only convolution sizes and channel numbers. 7.A hardware constraint oriented target detection backbone network searching device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer program comprises the following steps of, The processor implements the steps of the method of any one of claims 1 to 5 when the processor executes the computer program.

Images