A neural network simplification optimization method and system for UAV target recognition

Abstract

The present invention discloses a neural network streamlining and optimization method and system for drone target recognition, which belongs to the field of drone end-side target detection and recognition. The specific steps are as follows: obtaining video data covering work scenes and target types from the drone end-side platform; transmitting the video data back to the cloud, and using the open vocabulary visual large model for inference and calibration of target categories; transmitting the calibration results back to the drone end-side platform, and combining them with the video data to form a data set; loading a super network on the drone end-side platform, and performing multiple iterative training based on a specific update strategy to optimize the super network; extracting some sub-networks from the optimized super network for forward reasoning and performance evaluation, and selecting the optimal sub-network; using the optimal sub-network as a target detection model for real-time drone detection. This method combines the cloud-based open vocabulary visual large model and the drone end-side platform to achieve efficient target detection under low power consumption, and is applicable to a variety of scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of drone-side target detection and recognition, and in particular relates to a neural network streamlining and optimization method and system for drone target recognition. Background Art

[0002] Target detection and recognition technology plays a crucial role in drone applications. With the rapid development of drone technology, its applications are becoming increasingly widespread in areas such as environmental monitoring, agricultural inspections, disaster assessment, and traffic monitoring. In these scenarios, drones must accurately identify and locate targets in real time, placing higher demands on target detection and recognition technology. However, limited by drone hardware and battery life, only efficient and low-energy algorithms can be deployed on drone platforms, posing significant challenges to the implementation of target detection and recognition technology.

[0003] Traditional object detection methods, such as SIFT (Scale-Invariant Feature Transform) and HOG (Histogram of Oriented Gradients), which rely on handcrafted features, often struggle in the face of complex and ever-changing real-world scenarios. These methods suffer from insufficient feature representation, limited generalization, difficulty handling multiple targets, and complex algorithm design, making it difficult to achieve efficient object detection and recognition at a low cost. This is particularly true in drone applications, where the performance of traditional methods is further limited by factors such as variable shooting angles, complex lighting conditions, and large variations in target scale.

[0004] With the rapid development of deep learning technology, significant progress has been made in the field of object detection and recognition. Researchers have proposed a variety of high-performance neural network architectures, such as Inception-V2, ResNet, and DarkNet. These networks significantly enhance feature representation and simplify algorithm design through convolution operations and layer stacking. However, these deep neural networks typically have high computational complexity and parameter count, making them difficult to deploy directly on resource-constrained drone platforms.

[0005] To achieve efficient target detection and recognition on resource-constrained devices, researchers have proposed a variety of lightweight network architectures. For example, the MobileNet series significantly reduces the number of model parameters and computational complexity by introducing depthwise separable convolution technology; ShuffleNet uses channel reordering operations to reduce computational costs while maintaining feature expression capabilities; and EfficientNet achieves a balance between model accuracy and efficiency through a compound scaling strategy. These lightweight networks significantly reduce computational costs while maintaining high detection accuracy, making them more suitable for deployment on drone platforms. However, the detection performance of these lightweight networks in complex scenarios still has certain limitations, especially when facing small targets, occluded targets, and multi-scale targets, the detection results are often unsatisfactory.

[0006] The performance of target detection and recognition algorithms is highly dependent on the quality and scale of training data. In drone applications, due to the diversity and complexity of the shooting scenes, it is difficult and costly to obtain large-scale, high-quality annotated data. Specifically, the data collected by drones has the following characteristics: (1) variable viewing angles, including looking down, looking sideways, and other angles; (2) complex lighting conditions, which may face challenges such as strong light, backlight, and shadows; (3) large differences in target scales, and large and small targets may coexist in the same scene; (4) complex backgrounds, and targets may be occluded or similar to the background. These factors make data annotation more difficult and increase the difficulty of model training and generalization.

[0007] While techniques like self-supervised learning and semi-supervised learning have alleviated the burden of data labeling to some extent, their effectiveness in object detection and recognition tasks remains significantly limited. Both self-supervised and semi-supervised learning utilize large amounts of unlabeled data for training. However, due to the complex data distribution in drone applications and the inability to obtain samples of target scenes in advance, model generalization performance remains poor.

[0008] In practical applications, drone target detection and recognition face the dual challenges of real-time performance and accuracy. On the one hand, drones must complete target detection and recognition within limited battery life, requiring algorithms with high computational efficiency to minimize consumption of computing resources, including memory usage and computational complexity. On the other hand, the accuracy of detection and recognition results directly impacts the effectiveness of subsequent decision-making, so algorithms must maximize accuracy while maintaining efficiency. This trade-off between efficiency and accuracy is one of the core challenges facing drone target detection and recognition technology. Summary of the Invention

[0009] The purpose of the present invention is to solve the problems existing in the prior art and provide a neural network streamlining optimization method and system for drone target recognition.

[0010] In order to achieve the above-mentioned object of the invention, the present invention specifically adopts the following technical solutions:

[0011] In a first aspect, the present invention provides a neural network streamlining and optimization method for drone target recognition, which comprises the following steps:

[0012] S1. Obtain multiple video data segments captured by the drone-mounted camera from the drone-side platform. The video data should cover the work scenarios and target types required for detection.

[0013] S2. The video data is transmitted back to the cloud, where an open vocabulary vision model deployed there performs inference and calibrates the object categories in the video data, obtaining calibration results including bounding boxes, object category labels, and confidence levels.

[0014] S3. The calibration results obtained in the cloud are transmitted back to the drone-side platform. Multiple video segments are loaded on the drone-side platform, and the calibration results in the cloud are used as the true labels. Thus, a dataset is formed by the video data and its corresponding true labels.

[0015] S4. Load a super network containing multiple sub-networks on the UAV terminal platform, perform multiple iterative training on the super network on the data set based on a pre-built update strategy, sample a local sub-network in the super network in each iteration, and after the iterative training is completed, obtain the optimized super network and extract some sub-networks from it again to perform forward reasoning on the data set. After the reasoning is completed, perform performance evaluation based on pre-set evaluation indicators, take the sub-network with the best performance in the optimized super network as the optimal sub-network, reload the optimal sub-network, and use it as the target detection model in the actual detection scenario, and use it to perform real-time detection of the images or videos to be detected taken by the UAV, and output the detection results.

[0016] Based on the above solution, each step can be implemented in the following preferred specific manner.

[0017] As a preferred embodiment of the above-mentioned first aspect, in step S1, the work scenarios include forest fire prevention, agricultural plant protection, power inspection, security monitoring and river monitoring, and the target types include smoke and fire detection, pest and disease detection, power damage detection, pedestrian detection, vehicle detection and ship detection.

[0018] As a preferred embodiment of the first aspect above, in step S2, the reasoning process of the open vocabulary visual large model is as follows: first, the video data is decomposed into images frame by frame, and a series of candidate regions are generated by the open vocabulary visual large model. Then, the decomposed images are passed through a visual encoder to extract the visual features of each candidate region, and the visual features of each candidate region are matched with the embedded vector of the text category name obtained in advance through the visual-language model, and the similarity score is calculated. Then, the category with the highest similarity score is used as the target category corresponding to each candidate region to realize the recognition of known categories and unknown categories, and finally the calibration result is output.

[0019] As a preferred embodiment of the above-mentioned first aspect, in step S4, in the hypernetwork, the input data is processed in multiple stages in sequence, and the feature map output by the last stage is further processed by global pooling and a fully connected layer to obtain the output result; wherein, the first stage contains multiple candidate child nodes, and each candidate child node is composed of a plurality of convolutional layers with BN normalization operations and ReLU activation operations cascaded in sequence; the remaining stages contain multiple modules of the same type, and each module is composed of a plurality of convolutional layers with BN normalization operations and ReLU activation operations cascaded in sequence.

[0020] As a preferred embodiment of the first aspect above, a convolution module is additionally added between two adjacent stages, and the convolution module is composed of a plurality of convolution layers with BN normalization operations and ReLU activation operations cascaded in sequence.

[0021] As a preferred embodiment of the above-mentioned first aspect, in the hypernetwork, the input data is first processed by the first stage, the first stage includes four candidate child nodes, the first candidate child node is formed by the first convolutional layer and the second convolutional layer being cascaded in sequence, the second candidate child node is formed by the third convolutional layer and the fourth convolutional layer being cascaded in sequence, the third candidate child node is formed by the fifth convolutional layer, the sixth convolutional layer and the seventh convolutional layer being cascaded in sequence, and the fourth candidate child node is formed by the eighth convolutional layer, the ninth convolutional layer and the tenth convolutional layer being cascaded in sequence; after the input data is processed by the first stage, a first feature map is obtained, and then the first feature map is processed by the first convolution module, and the first convolution module is formed by the eleventh convolutional layer and the twelfth convolutional layer being cascaded in sequence. After the first feature map is processed by the first convolution module, the second feature map is obtained, and then the second feature map is processed by the second stage. The second stage contains two modules. The first module is composed of the thirteenth convolution layer, the fourteenth convolution layer and the fifteenth convolution layer in sequence, and the second module is composed of the sixteenth convolution layer and the seventeenth convolution layer in sequence. After the second feature map is processed by the second stage, the third feature map is obtained, and then the third feature map is processed by the second convolution module, the third stage, the third convolution module, the fourth stage, the fourth convolution module and the fifth stage in sequence to obtain the fourth feature map. After the fourth feature map is processed by global pooling and a fully connected layer, the output result is obtained.

[0022] Furthermore, the convolution kernel size of the first convolution layer is 3×3 and the stride is 2, the convolution kernel size of the second convolution layer is 3×3 and the stride is 1, the convolution kernel size of the third convolution layer is 3×3 and the stride is 1, the convolution kernel size of the fourth convolution layer is 3×3 and the stride is 2, the convolution kernel size of the fifth convolution layer is 3×3 and the stride is 2, the convolution kernel size of the sixth convolution layer is 3×3 and the stride is 1, the convolution kernel size of the seventh convolution layer is 3×3 and the stride is 1, the convolution kernel size of the eighth convolution layer is 3×3 and the stride is 1, and the convolution kernel size of the ninth convolution layer is 3×3. The convolution kernel size of the tenth convolution layer is 3×3 and the stride is 2, the convolution kernel size of the eleventh convolution layer is 3×3 and the stride is 2, the convolution kernel size of the twelfth convolution layer is 1×1 and the stride is 1, the convolution kernel size of the thirteenth convolution layer is 1×1 and the stride is 1, the convolution kernel size of the fourteenth convolution layer is 3×3 and the stride is 1, the convolution kernel size of the fifteenth convolution layer is 1×1 and the stride is 1, the convolution kernel size of the sixteenth convolution layer is 3×3 and the stride is 1, and the convolution kernel size of the seventeenth convolution layer is 1×1 and the stride is 1.

[0023] As a preferred embodiment of the first aspect, in step S4, the specific processing flow of the super network in each iteration is as follows:

[0024] S41, first, decomposing the video data in the data set into image data frame by frame, then randomly combining all the image data to extract a batch of image data, and performing a pre-set preprocessing operation;

[0025] S42. Then, N different sub-networks are randomly selected from the super-network updated in the previous iteration round. Each selected sub-network is used as a candidate sub-network. The pre-processed image data of the same batch is respectively transmitted to each candidate sub-network. Each candidate sub-network is used as a sampling path. The shared weights in the super-network updated in the previous iteration round are used to perform forward inference on each candidate sub-network to obtain their respective output results.

[0026] S43. Then, based on the true label and the output result of each candidate sub-network, the loss of each candidate sub-network is calculated according to a preset evaluation index, and the loss calculation result is used as the initial loss of each candidate sub-network;

[0027] S44. Randomly select a candidate sub-network, use the stochastic gradient descent method to reversely calculate the gradient and update the parameters of the candidate sub-network, and form a candidate super-network by combining the updated candidate sub-network and the remaining sub-networks in the super-network updated in the previous iteration round;

[0028] S45. Re-perform forward reasoning in the candidate supernetwork according to the same N candidate subnetworks as S42 to obtain a new output result, and recalculate the loss of each candidate subnetwork according to S43, and use the new loss calculation result as the optimization loss of each candidate subnetwork;

[0029] S46. The low-quality model set in the current iteration round is formed by the candidate sub-networks whose optimization loss is greater than the initial loss. The gradients corresponding to the low-quality model set and the updated candidate sub-networks are calculated according to the initial loss. The weights corresponding to the low-quality model set and the updated candidate sub-networks are simultaneously updated in the super-network updated in the previous iteration round to form the super-network updated in the current iteration round.

[0030] As a preferred embodiment of the first aspect, in step S43, the loss of the candidate sub-network includes binary cross entropy loss, generalized intersection-over-union loss, or a combination of the two losses.

[0031] As a preferred embodiment of the first aspect, in step S4, the evaluation indicators used are branch performance and inference speed, and the optimal subnetwork is selected from the optimized supernetwork according to the Pareto frontier method based on the performance evaluation results.

[0032] In a second aspect, the present invention provides a neural network simplification and optimization system for drone target recognition, comprising:

[0033] The drone end-side platform and the cloud, and the communication connection between the drone end-side platform and the cloud are used to implement a neural network streamlining and optimization method for drone target recognition as described in any of the solutions of the first aspect above; the drone end-side platform includes a computer and a drone; the cloud includes a server, and an open vocabulary visual large model is deployed on the server.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] This paper proposes a streamlined neural network optimization method for drone target recognition by combining a large, open-vocabulary visual model in the cloud with a low-power drone-side platform. This method utilizes the large, open-vocabulary visual model in the cloud for calibration and supervised training. Through unique on-device iterative optimization techniques, it stably selects high-performance subnetworks from the hypernetwork, achieving efficient target detection with low power consumption. This method is suitable for a variety of drone visual detection scenarios, including vehicles and animals. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of the steps of the method of the present invention;

[0037] Figure 2 A schematic diagram of a super network structure of the present invention;

[0038] Figure 3 A flowchart of iterative training of a hypernetwork according to the present invention;

[0039] Figure 4 A flow chart of the present invention in generating an optimal subnetwork;

[0040] Figure 5 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0041] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in the various embodiments of the present invention can be combined accordingly without conflicting with each other.

[0042] In the description of the present invention, it should be understood that the terms "first" and "second" are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly specifying the number of technical features being described. Therefore, features defined as "first" or "second" may explicitly or implicitly include at least one of such features.

[0043] To address these challenges, the present invention proposes a neural network optimization method for drone target recognition. By combining the high-performance computing capabilities of the cloud with lightweight inference on the drone's end-to-end platform, this method achieves efficient and accurate target detection and recognition in complex scenarios. For drone target recognition tasks, existing approaches typically first sample image data from the mission scene for manual calibration. Then, a publicly available neural network is selected for training based on the drone's hardware resources and endurance requirements, and then deployed on the drone for application. In cloud-end collaborative approaches, existing solutions collect field data and transmit it back to the cloud for model training and fine-tuning, before transferring the model parameters to the drone for redeployment. Unlike existing approaches, the present invention leverages the powerful capabilities of the cloud's open-vocabulary large-scale visual model to calibrate field data. Through collaborative cloud-end collaboration, the end-to-end model is updated to adapt to specific scenarios. The most efficient inference branch is then selected based on the resource constraints of the drone device, significantly improving the detection and recognition performance and real-time performance of the drone's end-to-end model. This method significantly improves computational efficiency while maintaining detection accuracy, providing strong technical support for drone applications in various complex scenarios.

[0044] like Figure 1 As shown, in a preferred implementation of the present invention, the above-mentioned neural network streamlining and optimization method for drone target recognition includes the following steps 1 to 6. The specific implementation process is described in detail below.

[0045] Step 1: Video Data Collection

[0046] Multiple video data segments captured by the drone-mounted camera are obtained from the drone-side platform. The video data covers the work scenarios and target types that need to be detected to ensure the comprehensiveness and representativeness of the video data.

[0047] It should be noted that in step 1 of the present invention, the work scenarios include but are not limited to forest fire prevention, agricultural plant protection, power inspection, security monitoring, and river monitoring, and the target types include but are not limited to fireworks detection, pest and disease detection, power damage detection, pedestrian detection, vehicle detection, and ship detection.

[0048] It should be noted that, in the present invention, the drone-side platform is a device platform that works with the drone to implement signal reception and transmission and data processing. It is used for real-time computing and data processing, reducing reliance on the cloud, thereby improving the drone's autonomy, responsiveness, and mission execution efficiency. In the present invention, the core functions of the drone-side platform include: real-time processing of data collected by the drone, such as image recognition, target detection, and path planning; and enabling data transmission between the drone and other drones or the cloud.

[0049] Step 2: Video upload and cloud inference

[0050] The video data is transmitted back to the cloud, where the open vocabulary visual model deployed there performs inference and calibrates the target categories in the video data, obtaining calibration results including bounding boxes, target category labels, and confidence levels.

[0051] It should be noted that in step 2 of the present invention, the open vocabulary visual large model is deployed on the cloud GPU and can efficiently process large-scale image data and calibrate the target categories in the video data.

[0052] The reasoning process of the open vocabulary visual big model is as follows: first, the video data is decomposed into images frame by frame, and the open vocabulary visual big model generates a series of candidate regions (proposals). Then, the decomposed images are passed through a visual encoder (such as CNN or Transformer) to extract the visual features of each candidate region. The visual features of each candidate region are matched with the embedded vector of the pre-obtained text category name through a visual-language model (such as CLIP), and the similarity score is calculated. The category with the highest similarity score is then used as the target category corresponding to each candidate region (the most likely category for each region is determined based on the similarity score), realizing the recognition of known and unknown categories, and finally outputting the calibration result.

[0053] In this example, an open-vocabulary visual model is used to accomplish the open-vocabulary object detection (OVD) task. This is a novel computer vision task that aims to enable the model to recognize and localize object categories that are not explicitly labeled in the training set. Unlike traditional object detection models, which can only recognize a limited number of predefined categories, the OVD model combines visual and linguistic information to recognize "open-vocabulary" categories. This means that during testing, it can recognize and localize categories that have not been seen in the training set. The core challenges of this task lie in enabling the model to generalize from seen categories to unseen categories and effectively align image regions with text descriptions to achieve cross-modal understanding. To achieve this, various technical approaches are required, including leveraging vision-language pre-trained models (such as CLIP) to learn the correspondence between image and text, employing region-text alignment mechanisms to align candidate image regions with text descriptions, generating pseudo-labels for unlabeled image regions using weakly supervised or unsupervised methods, and transferring knowledge from the pre-trained model to the object detection model through knowledge distillation. In recent years, a number of research works have made significant progress in the field of OVD, such as Grounding-DINO and OV-DETR.

[0054] Step 3: Return calibration results

[0055] The calibration results obtained in the cloud are transmitted back to the drone-side platform, where multiple video data segments are loaded and the calibration results in the cloud are used as true labels, thus forming a dataset consisting of video data and its corresponding true labels.

[0056] Step 4: Iterative training of the on-device supernetwork and subnetwork optimization

[0057] A supernet containing multiple subnetworks is loaded on the drone-side platform. The supernet is iteratively trained on the dataset based on a pre-built update strategy. Local subnetworks in the supernet are sampled in each iteration. After the iterative training is completed, an optimized supernet is obtained.

[0058] It should be noted that in the present invention, the number of sub-networks selected in each iteration needs to be determined based on the computing power of the end-side device, and then set by technical personnel in this field according to actual needs. A specific numerical value can also be specified, so it is not limited in the present invention.

[0059] It should be noted that to facilitate a better understanding of the present invention by those skilled in the art, a brief introduction to the background knowledge of supernets is now provided. In Neural Architecture Search (NAS), a supernet is an over-parameterized network structure that includes all subnetworks. It has a large model capacity and diversity, and significantly reduces search costs through a weight sharing mechanism. During the training phase, the supernet samples and trains the paths of different subnetworks, allowing each subnetwork to share some parameters, thereby avoiding the high cost of training each architecture separately. During the search phase, the trained supernet can be used to quickly evaluate and compare the performance of different subnetworks and select the optimal architecture. This method not only improves search efficiency, but also achieves excellent performance in tasks such as image classification and object detection.

[0060] In the hypernetwork of the present invention, the input data is processed in multiple stages in sequence, and the feature map output by the last stage is further processed by global pooling and a fully connected layer to obtain the output result; wherein, the first stage contains multiple candidate child nodes, each of which is composed of a plurality of convolutional layers with BN normalization operations and ReLU activation operations cascaded in sequence; the remaining stages contain multiple modules of the same type, each of which is composed of a plurality of convolutional layers with BN normalization operations and ReLU activation operations cascaded in sequence.

[0061] Furthermore, the present invention also adds an additional convolution module between two adjacent stages, and each convolution module is composed of a plurality of convolution layers with BN normalization operations and ReLU activation operations cascaded in sequence.

[0062] It should be noted that the network structure of the above-mentioned hypernetwork can be selected according to actual needs, and the technical solution of the present invention can be implemented by replacing other hypernetwork structures, and is not limited to the network structure provided by the present invention.

[0063] In this embodiment, the structural diagram of the above-mentioned super network is as follows: Figure 2 As shown. The input data is first processed by the first stage. The first stage contains four candidate sub-nodes. The convolution layer of each candidate sub-node has BN normalization operation and ReLU activation operation. The first candidate sub-node is formed by the first convolution layer and the second convolution layer in sequence. The convolution kernel size of the first convolution layer is 3×3 and the stride is 2, and the convolution kernel size of the second convolution layer is 3×3 and the stride is 1. The second candidate sub-node is formed by the third convolution layer and the fourth convolution layer in sequence. The convolution kernel size of the third convolution layer is 3×3 and the stride is 1, and the convolution kernel size of the fourth convolution layer is 3×3 and the stride is 2. The third candidate sub-node is formed by the fifth convolution layer, the sixth convolution layer and the seventh convolution layer in sequence. The convolution kernel size of the fifth convolution layer is 3×3 and the stride is 2, the convolution kernel size of the sixth convolution layer is 3×3 and the stride is 1, the convolution kernel size of the seventh convolution layer is 3×3 and the stride is 1, and the fourth candidate child node is formed by the eighth convolution layer, the ninth convolution layer and the tenth convolution layer in sequence. The convolution kernel size of the eighth convolution layer is 3×3 and the stride is 1, the convolution kernel size of the ninth convolution layer is 3×3 and the stride is 2, and the convolution kernel size of the tenth convolution layer is 3×3 and the stride is 1. Figure 2 In the figure, "conv3x3 / s1" indicates a convolutional layer with a kernel size of [3,3] and a stride of 1; "conv3x3 / s2" indicates a convolutional layer with a kernel size of [3,3] and a stride of 2; and "conv1x1 / s1" indicates a convolutional layer with a kernel size of [1,1] and a stride of 1. After the input data is processed by the first stage, the first feature map is obtained, and then the first feature map is processed by the first convolution module. The first convolution module is composed of the eleventh and twelfth convolution layers cascaded in sequence. The convolution kernel size of the eleventh convolution layer is 3×3 and the stride is 2, and the convolution kernel size of the twelfth convolution layer is 1×1 and the stride is 1. After the first feature map is processed by the first convolution module, the second feature map is obtained, and then the second feature map is processed by the second stage. The second stage contains two modules (blocks). The first module ( Figure 2Res_Block_131 in ( ) is composed of the 13th, 14th, and 15th convolutional layers cascaded in sequence. The convolution kernel size of the 13th convolutional layer is 1×1 and the stride is 1, the convolution kernel size of the 14th convolutional layer is 3×3 and the stride is 1, and the convolution kernel size of the 15th convolutional layer is 1×1 and the stride is 1; the second module ( Figure 2 Res_Block_13 in the figure is composed of the 16th and 17th convolutional layers cascaded in sequence. The convolution kernel size of the 16th convolutional layer is 3×3 and the stride is 1, and the convolution kernel size of the 17th convolutional layer is 1×1 and the stride is 1. After the second feature map is processed by the second stage, the third feature map is obtained. The third feature map is then processed by the second convolution module, the third stage, the third convolution module, the fourth stage, the fourth convolution module, and the fifth stage in sequence to obtain the fourth feature map. The fourth feature map is then processed by global pooling and a fully connected layer to obtain the output result. Figure 2 In

[15] , Res_Block_131 is conv1x1 / s1-conv3x3 / s1-conv1x1 / s1, and Res_Block_13 is conv3x3 / s1-conv1x1 / s1. "Expansion ratio" is the ratio of the number of output channels to the number of input channels of the first convolutional layer in the block.

[0064] It should be noted that, in step 4 of the present invention, Figure 3 As shown in Figure 2, the specific processing flow of the hypernetwork at each iteration is as follows:

[0065] Step 41: First, decompose the video data in the data set into image data frame by frame, then randomly combine all the image data to extract a batch of image data, and perform a preset preprocessing operation.

[0066] In this embodiment, the preprocessing operations include but are not limited to rotation, flipping, random cropping, random scaling, color jittering, noise addition, etc.

[0067] Step 42: Then, N different sub-networks are randomly selected from the super-network updated in the previous iteration round. Each selected sub-network is used as a candidate sub-network. The pre-processed image data of the same batch is respectively passed into each candidate sub-network. Each candidate sub-network is used as a sampling path. The shared weights in the super-network updated in the previous iteration round are used to perform forward inference on each candidate sub-network to obtain their respective output results.

[0068] Step 43: Next, the loss of each candidate sub-network is calculated based on the true label and the output result of each candidate sub-network according to the preset evaluation index, and the loss calculation result is used as the initial loss of each candidate sub-network.

[0069] In this embodiment, the type of loss includes but is not limited to binary cross-entropy loss (BCE Loss), generalized intersection over union loss (GIOU Loss), or a combination of the two losses.

[0070] Furthermore, the binary cross entropy loss The functional form is:

[0071]

[0072] in, is the true label, is the output result of the candidate sub-network.

[0073] Furthermore, the generalized intersection-over-union loss The functional form is:

[0074]

[0075]

[0076]

[0077] in, is the predicted box; B is the true value box, which comes from the calibration result obtained in the cloud; is the intersection area of ​​the predicted box and the true value box; is the union area of ​​the predicted box and the true value box; It is the smallest enclosing rectangular box that can contain the prediction box and the true value box (Smallest Enclosing Box); is the area of ​​the minimum circumscribed rectangular frame; The area of ​​the minimum bounding rectangle that does not belong to the intersection of the prediction box and the true value box; Represents the generalized intersection over union between the predicted box and the true value box; Represents the Intersection over Union between the predicted box and the true value box.

[0078] Step 44: Randomly select a candidate sub-network, use the stochastic gradient descent (SGD) method to reversely calculate the gradient and update the parameters of the candidate sub-network. The updated candidate sub-network and the remaining sub-networks in the super-network updated in the previous iteration round constitute the candidate super-network A'.

[0079] It should be noted that, in the present invention, the candidate supernetwork A' cannot overwrite the supernetwork A updated in the previous iteration round, but must be stored separately.

[0080] Step 45: Re-perform forward reasoning in the candidate super-network according to the same N candidate sub-networks as in step 42 to obtain a new output result, and recalculate the loss of each candidate sub-network according to step 43, and use the new loss calculation result as the optimization loss loss' of each candidate sub-network.

[0081] Step 46: The candidate sub-networks whose optimization loss loss' is greater than the initial loss loss constitute the low-quality model set worse_models in the current iteration round. The gradients corresponding to the low-quality model set and the updated candidate sub-network P1 are calculated based on the initial loss. The weights corresponding to the low-quality model set and the updated candidate sub-network P1 are simultaneously updated in the super-network updated in the previous iteration round to form the super-network A'' updated in the current iteration round.

[0082] It should be noted that, in the present invention, unlike the candidate supernetwork A', the supernetwork A'' updated in the current iteration round needs to overwrite the supernetwork A updated in the previous iteration round, without the need for separate storage.

[0083] The update strategy adopted in the above process is the core innovation of the method of the present invention. Taking into account that the super network training update of the traditional NAS method mainly extracts and updates the sub-network space, it will ignore the impact of this update on other sub-network spaces that share weights, resulting in a low degree of overlap between the optimal ranking and the theoretical performance ranking, and it is impossible to select a truly good model. In order to solve this problem, the present invention designs a specific update strategy, which has the advantage that the strategy will judge the relationship between the parameter update of the selected sub-network and other sub-networks through a tentative update, and adopt a unified update strategy for associated sub-networks with side effects, avoiding the above problems, further preventing local sub-networks in the super network from damaging the performance of other sub-networks during updates, and improving sorting performance and stability.

[0084] Step 5: Optimal subnetwork selection

[0085] On the drone side platform, some sub-networks are extracted again from the optimized super-network for forward reasoning on the dataset. After the reasoning is completed, performance evaluation is performed based on pre-set evaluation indicators, and the sub-network with the best performance in the optimized super-network is selected as the optimal sub-network.

[0086] It should be noted that in step 5 of the present invention, the evaluation metrics used are branch performance and inference speed. Based on the performance evaluation results, the optimal subnetwork is selected from the optimized supernetwork using the Pareto front method. The evaluation metrics used here are common performance indicators in the field of drone target recognition and can also be selected by those skilled in the art based on actual needs. In addition to the Pareto front method, the optimal subnetwork can also be obtained by weighted summing various evaluation metrics to select the optimal subnetwork, and existing public solutions can also be used.

[0087] The following briefly introduces the two evaluation indicators used in this embodiment and the principle of the Pareto frontier method.

[0088] In this embodiment, branch performance represents the task performance of the hypernetwork. When it performs target detection tasks, branch performance is generally the mean average precision (mAP). When it performs target classification tasks, branch performance is generally the classification accuracy.

[0089] In this embodiment, inference speed refers to the time it takes for a sampled subnetwork to infer a batch of image data on the drone-side platform. This is measured using both hardware measurements and model prediction. Hardware measurements involve deploying the subnetwork on the drone-side platform and then calculating the actual inference time required for a batch of image data. Model prediction involves training a time prediction model on the drone-side platform that maps neural network structure to inference time. When evaluating inference speed, the approximate inference time of the subnetwork on the drone-side platform is estimated directly from the subnetwork structure and the time prediction model.

[0090] In this embodiment, the Pareto Front method is a multi-objective optimization strategy designed to find the optimal trade-off between multiple conflicting evaluation metrics. Model evaluation often involves trade-offs between performance metrics such as accuracy and inference speed. The Pareto Front method constructs a set of optimal solutions, known as the Pareto Front, by identifying a set of solutions that are not dominated by other solutions on all objectives. These solutions represent the optimal balance between improving one metric without significantly compromising other metrics, helping decision makers select the most appropriate model solution for their specific needs without requiring pre-set weights.

[0091] The main process of selecting the optimal subnetwork by the Pareto frontier method is as follows:

[0092] 1) Extract N subnetworks to perform data set inference and record the inference time set X.

[0093] 2) Compare with the label information to obtain the inference performance set Y of the N sub-networks.

[0094] 3) According to the Pareto frontier principle, take the X, Y combination on the frontier and its corresponding sub-network set.

[0095] The Pareto front is defined as follows: assuming T1 and T2 are solutions to a multi-objective optimization problem, if T1's score for all optimization objectives is not inferior to T2 and at least one score is superior to T2, then T1 is said to completely dominate T2, and T2 is the dominated solution. If no solution in the entire set of solutions to the multi-objective optimization problem dominates T3, then T3 is called a non-dominated solution. The set of all non-dominated solutions to a multi-objective optimization problem is called the Pareto front.

[0096] 4) Select the sub-network structure corresponding to the X, Y pair with the highest performance within the time limit as the optimal structure.

[0097] Therefore, if Figure 4 As shown, according to the above steps 1 to 5 of the present invention, the generation of the optimal sub-network is completed.

[0098] Step 6: Load the optimal subnetwork

[0099] The optimal sub-network is reloaded on the drone-side platform and used as the target detection model in the actual detection scenario. It is used to perform real-time detection on the images or videos to be detected taken by the drone and output the detection results.

[0100] In order to better demonstrate the specific implementation and technical effects of the present invention, a neural network streamlining optimization method for drone target recognition shown in steps 1 to 6 in the above preferred implementation method is applied to a specific example.

[0101] Example

[0102] The specific implementation process of the neural network streamlining optimization method for drone target recognition adopted in this embodiment is as described above and will not be repeated here.

[0103] To demonstrate the effectiveness of the proposed method, this example demonstrates its implementation in the MS-COCO public dataset and in a traffic road object detection scenario. The MS-COCO dataset is a widely used benchmark dataset for object detection, containing 80 common object categories in daily life, including images with occlusion, varying scales, varying viewpoints, and complex scenes.

[0104] This example uses the publicly available ResNet50-YoloV3 model as the baseline model. Using the method of the present invention, streamlined optimization is performed to obtain streamlined models A and B, which are benchmarked against the baseline model. The streamlined models also use YoloV3 for detection algorithms. Table 1 shows a comparison of the size and performance of the three models.

[0105] Table 1. Performance of the present invention on the MS-COCO dataset

[0106]

[0107] Table 1 shows that the proposed method significantly improves object detection performance while maintaining similar computational and parameter requirements. Simplified Model A, with computational and parameter requirements similar to the baseline model, achieves a 0.8 percentage point improvement in mAP over the baseline model (ResNet50-YoloV3) of 35.4. Simplified Model B, with approximately a 50% increase in computational and parameter requirements, achieves a further improvement in mAP to 37.0, demonstrating that the proposed method achieves a good balance between model performance and computational efficiency.

[0108] To verify the applicability of the present invention in complex scenarios, this embodiment conducted experiments on pedestrian and vehicle detection tasks in a traffic road monitoring scenario. The results are shown in Table 2. The baseline model used for comparison is the publicly available FasterRCNN-Darknet23 model. Simplified models C and D are both FasterRCNN detection models, but differ in their backbone architecture from the baseline model. The pedestrian recall rate is the performance at an fppi of 0.3, while the vehicle recall rate is the performance at an fppi of 0.1.

[0109] Table 2. Comparison of effects in traffic road scenes

[0110]

[0111] Experimental results show that the streamlined models C and D optimized using our method achieve significantly improved recall for both pedestrian and vehicle detection compared to publicly available models, while maintaining similar inference efficiency. The pedestrian detection recall of streamlined model C reaches 70.01%, an increase of 1.74 percentage points compared to the baseline model's 68.27%. This result demonstrates the effectiveness and practicality of our method in real-world scenarios.

[0112] Through the above embodiments, the following conclusions can be drawn:

[0113] 1. The proposed method outperforms the baseline model in both the MS-COCO dataset and traffic road vehicle detection scenarios, demonstrating its effectiveness in target detection and recognition tasks.

[0114] 2. The method of the present invention significantly improves the detection accuracy with similar computational complexity and parameter amounts, indicating that it has achieved a good balance between model performance and computational efficiency.

[0115] 3. In the traffic road monitoring scenario, the recall rate of the proposed method is significantly higher than that of the baseline model, proving its wide applicability in practical applications.

[0116] In summary, the proposed neural network optimization method for drone target recognition not only performs well on academic datasets but also demonstrates good model optimization results in real-world scenarios. By combining high-performance computing in the cloud with lightweight inference on the drone side, the proposed method provides an efficient and accurate solution for drone target detection, with broad application prospects.

[0117] It should also be noted that the neural network simplification and optimization for drone target recognition in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a neural network simplification and optimization system for drone target recognition corresponding to the neural network simplification and optimization method for drone target recognition provided in the above embodiment, such as Figure 5 As shown, it includes:

[0118] The drone end-side platform and the cloud, and the communication connection between the drone end-side platform and the cloud are used to implement a neural network streamlining and optimization method for drone target recognition as described in any of the solutions of the first aspect above; the drone end-side platform includes a computer and a drone; the cloud includes a server, and an open vocabulary visual large model is deployed on the server.

[0119] Specifically, the drone in the drone-side platform is used to shoot the video data required for target detection. After the video data is transmitted to the cloud, it is calibrated by the open vocabulary visual model deployed on the cloud server, and the calibration results are output and transmitted back to the drone-side platform. The computer on the drone-side platform loads the super network and performs multiple iterative training on it to optimize the super network. Then, some sub-networks are extracted from the optimized super network for forward reasoning. After the reasoning is completed, performance evaluation is performed, and the sub-network with the best performance is used as the optimal sub-network. Finally, only the optimal sub-network is retained on the computer as the target detection model, and the remaining network paths are deleted. In the actual target detection process, the drone can shoot the image or video to be detected, and the target detection model on the computer can perform real-time detection and output the detection results.

[0120] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A neural network optimization method for UAV target recognition, characterized in that: The following steps are involved: S1. Obtain multiple video data segments captured by the drone-mounted camera from the drone-side platform. The video data should cover the work scenarios and target types required for detection. S2. The video data is transmitted back to the cloud, where an open vocabulary vision model deployed there performs inference and calibrates the object categories in the video data, obtaining calibration results including bounding boxes, object category labels, and confidence levels. S3. The calibration results obtained in the cloud are transmitted back to the drone-side platform. Multiple video segments are loaded on the drone-side platform, and the calibration results in the cloud are used as the true labels. Thus, a dataset is formed by the video data and its corresponding true labels. S4. Load a super network containing multiple sub-networks on the UAV terminal platform, perform multiple iterative training on the super network on the data set based on a pre-built update strategy, sample a local sub-network in the super network in each iteration, and after the iterative training is completed, obtain the optimized super network and extract some sub-networks from it again to perform forward reasoning on the data set. After the reasoning is completed, perform performance evaluation based on pre-set evaluation indicators, take the sub-network with the best performance in the optimized super network as the optimal sub-network, reload the optimal sub-network, and use it as the target detection model in the actual detection scenario, and use it to perform real-time detection of the images or videos to be detected taken by the UAV, and output the detection results.

2. A neural network simplification and optimization method for drone target recognition according to claim 1, characterized in that: In step S1, the work scenarios include forest fire prevention, agricultural plant protection, power inspection, security monitoring and river monitoring, and the target types include smoke and fire detection, pest detection, power damage detection, pedestrian detection, vehicle detection and ship detection.

3. The neural network simplification and optimization method for drone target recognition according to claim 1, characterized in that: In step S2, the reasoning process of the open vocabulary visual big model is as follows: first, the video data is decomposed into images frame by frame, and a series of candidate regions are generated by the open vocabulary visual big model. Then, the decomposed images are passed through the visual encoder to extract the visual features of each candidate region, and the visual features of each candidate region are matched with the embedded vector of the pre-obtained text category name through the visual-language model. The similarity score is calculated, and then the category with the highest similarity score is used as the target category corresponding to each candidate region to realize the recognition of known categories and unknown categories, and finally the calibration result is output.

4. The neural network optimization method for drone target recognition according to claim 1, wherein: In step S4, in the hypernetwork, the input data is processed in multiple stages in sequence, and the feature map output by the last stage is processed by global pooling and a fully connected layer to obtain the output result; wherein, the first stage contains multiple candidate child nodes, each candidate child node is composed of a plurality of convolutional layers with BN normalization operations and ReLU activation operations cascaded in sequence; the remaining stages contain multiple modules of the same type, each module is composed of a plurality of convolutional layers with BN normalization operations and ReLU activation operations cascaded in sequence.

5. A neural network simplification and optimization method for drone target recognition according to claim 4, characterized in that: A convolution module is added between two adjacent stages. The convolution module is composed of a plurality of convolution layers with BN normalization operations and ReLU activation operations cascaded in sequence.

6. A neural network simplification and optimization method for drone target recognition according to claim 5, characterized in that: In the hypernetwork, the input data is first processed by the first stage, and the first stage includes four candidate child nodes. The first candidate child node is formed by the first convolutional layer and the second convolutional layer being cascaded in sequence, the second candidate child node is formed by the third convolutional layer and the fourth convolutional layer being cascaded in sequence, the third candidate child node is formed by the fifth convolutional layer, the sixth convolutional layer and the seventh convolutional layer being cascaded in sequence, and the fourth candidate child node is formed by the eighth convolutional layer, the ninth convolutional layer and the tenth convolutional layer being cascaded in sequence; after the input data is processed by the first stage, a first feature map is obtained, and the first feature map is then processed by the first convolution module, and the first convolution module is formed by the eleventh convolutional layer and the twelfth convolutional layer being cascaded in sequence; the first feature map is obtained by the first convolution module. After the feature map is processed by the first convolution module, the second feature map is obtained, and then the second feature map is processed by the second stage. The second stage contains two modules. The first module is composed of the thirteenth convolution layer, the fourteenth convolution layer and the fifteenth convolution layer in sequence, and the second module is composed of the sixteenth convolution layer and the seventeenth convolution layer in sequence. After the second feature map is processed by the second stage, the third feature map is obtained. Then the third feature map is processed by the second convolution module, the third stage, the third convolution module, the fourth stage, the fourth convolution module and the fifth stage in sequence to obtain the fourth feature map. The fourth feature map is processed by global pooling and a fully connected layer to obtain the output result.

7. The neural network simplification and optimization method for drone target recognition according to claim 1, characterized in that: In step S4, the specific processing flow of the super network at each iteration is as follows: S41, first, decomposing the video data in the data set into image data frame by frame, then randomly combining all the image data to extract a batch of image data, and performing a pre-set preprocessing operation; S42. Then, N different sub-networks are randomly selected from the super-network updated in the previous iteration round. Each selected sub-network is used as a candidate sub-network. The pre-processed image data of the same batch is respectively transmitted to each candidate sub-network. Each candidate sub-network is used as a sampling path. The shared weights in the super-network updated in the previous iteration round are used to perform forward inference on each candidate sub-network to obtain their respective output results. S43. Then, based on the true label and the output result of each candidate sub-network, the loss of each candidate sub-network is calculated according to a preset evaluation index, and the loss calculation result is used as the initial loss of each candidate sub-network; S44. Randomly select a candidate sub-network, use the stochastic gradient descent method to reversely calculate the gradient and update the parameters of the candidate sub-network, and form a candidate super-network by combining the updated candidate sub-network and the remaining sub-networks in the super-network updated in the previous iteration round; S45. Re-perform forward reasoning in the candidate supernetwork according to the same N candidate subnetworks as S42 to obtain a new output result, and recalculate the loss of each candidate subnetwork according to S43, and use the new loss calculation result as the optimization loss of each candidate subnetwork; S46. The low-quality model set in the current iteration round is formed by the candidate sub-networks whose optimization loss is greater than the initial loss. The gradients corresponding to the low-quality model set and the updated candidate sub-networks are calculated according to the initial loss. The weights corresponding to the low-quality model set and the updated candidate sub-networks are simultaneously updated in the super-network updated in the previous iteration round to form the super-network updated in the current iteration round.

8. A neural network simplification and optimization method for drone target recognition according to claim 7, characterized in that: In step S43, the loss of the candidate sub-network includes binary cross entropy loss, generalized intersection-over-union loss, or a combination of the two losses.

9. The neural network simplification and optimization method for drone target recognition according to claim 1, characterized in that: In step S4, the evaluation indicators used are branch performance and inference speed, and the optimal subnetwork is selected from the optimized supernetwork based on the performance evaluation results according to the Pareto frontier method.

10. A neural network streamlined optimization system for drone target recognition, characterized in that: include: The drone end-side platform and the cloud, and the communication connection between the drone end-side platform and the cloud, are used to implement a neural network streamlining and optimization method for drone target recognition as described in any one of claims 1 to 9; the drone end-side platform includes a computer and a drone; the cloud includes a server, and an open vocabulary visual large model is deployed on the server.

Images