Cloud edge collaborative target detection method based on dynamic super network

Abstract

The invention provides a cloud edge cooperative target detection method based on a dynamic super network, and the method comprises the steps: constructing a dynamic segmentation super network based on a YOLOv5s single-branch network, the dynamic segmentation super network comprises a plurality of convolution layers and a plurality of segmentation points, adding a fine-grained compensation F-Com block and an Identity block at each segmentation point, and obtaining a fine-grained compensation F-Com block and an Identity block; adding a coarse-grained compensation C-Com block at the tail end of the dynamic segmentation super network; based on the sample image, training the dynamic segmentation super network to obtain a trained super network; based on the trained super network, constructing an inference model, deploying the inference model at edge equipment and a cloud end, performing neural network architecture search in real time, determining an optimal segmentation point, and obtaining a target detection model; and the edge device and the cloud end cooperatively utilize the target detection model to perform target detection on the to-be-detected image to obtain a target detection result. The method is high in real-time performance, and can improve the reasoning efficiency and precision in a dynamic scene.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a cloud-edge collaborative target detection method based on a dynamic hypernetwork. Background Art

[0002] In diversified application scenarios such as drone inspections, disaster relief, and augmented reality, lightweight edge devices can continue to work normally under poor network conditions or limited bandwidth resources, highlighting the importance of edge computing in such situations. However, due to the limited computing power, energy consumption, and bandwidth resources of edge devices, it is difficult to meet the low-latency and high-precision perception requirements when running intelligent models with many parameters and large computational workloads. In recent years, cloud-edge collaborative reasoning based on deep neural networks has attracted widespread attention. The key to collaborative reasoning is to find the optimal segmentation point of the neural network. At present, the existing technology uses a static method to determine the optimal segmentation point, and the segmentation point is fixed. Due to the limited computing power of the device, the reasoning efficiency and accuracy are reduced in scenarios where the bandwidth changes dynamically. Summary of the invention

[0003] The present invention provides a cloud-edge collaborative target detection method based on a dynamic hypernetwork, so as to solve the defects in the prior art of using a static method to determine the optimal segmentation point, the segmentation point is fixed, and it is difficult to adapt to the dynamically changing bandwidth, resulting in reduced reasoning efficiency and decreased accuracy.

[0004] In a first aspect, the present invention provides a cloud-edge collaborative target detection method based on a dynamic hypernetwork, comprising: A dynamic segmentation super network is constructed based on a YOLOv5s single-branch network, wherein the dynamic segmentation super network includes multiple convolutional layers and corresponding multiple segmentation points, a fine-grained compensation F-Com block and an Identity block are added to each segmentation point, and a coarse-grained compensation C-Com block is added to the end of the dynamic segmentation super network; Based on the sample image, the dynamic segmentation hypernetwork is trained to obtain a trained hypernetwork; Based on the trained super network, an inference model is constructed, and the inference model is deployed on edge devices and the cloud. Based on the inference model, a neural network architecture search is performed in real time to determine an optimal segmentation point, and based on the optimal segmentation point, an optimal subnetwork is determined to obtain a target detection model; The edge device and the cloud collaboratively use the target detection model to perform target detection on the image to be detected, and obtain the target detection result of the image to be detected.

[0005] In some embodiments, the performing a neural network architecture search in real time based on the inference model to determine an optimal segmentation point includes: An objective function is constructed with the goal of minimizing the total inference delay of the inference model, where the total inference delay includes the computation delay of the edge device, the data transmission delay, and the computation delay of the cloud; The objective function is solved by using an adaptive segmentation point selection algorithm, and an optimal segmentation point is determined from the multiple segmentation points.

[0006] In some embodiments, the objective function is calculated as follows: ; ; in, represents the total inference delay, Indicates that the edge device adopts the inference model The computational latency of data processing at the layer, represents the data transmission delay from the edge device to the cloud, Indicates that the cloud uses the inference model The computational latency of data processing at the layer, is the current bandwidth of the communication channel between the edge device and the cloud, is the current signal-to-noise ratio of the communication channel, is the first The amount of feature data output by the layer represents the amount of feature data transmitted by the edge device to the cloud. The first layer, The first layer, , Represents the total number of convolutional layers of the inference model.

[0007] In some embodiments, the step of training the dynamic segmentation hypernetwork based on the sample image to obtain the trained hypernetwork includes: Setting the binary vector groups corresponding to the fine-grained compensation F-Com blocks and the Identity blocks of the plurality of segmentation points, so that the target segmentation point selects the corresponding fine-grained compensation F-Com block for feature compression and fine-grained feature compensation, and the common segmentation point selects the corresponding Identity block for direct feature transfer; Dynamically adjust the binary vector group, adopt a uniform sampling strategy, simultaneously train sub-networks corresponding to different target segmentation points based on the sample image, and calculate the cumulative gradient of each sub-network; Based on the accumulated gradient of each sub-network, the parameters of each sub-network are optimized to obtain a trained super-network.

[0008] In some embodiments, the object detection model includes a feature encoding module, a feature decoding module, and an object detection module; The edge device and the cloud collaboratively use the target detection model to perform target detection on the image to be detected, and obtain the target detection result of the image to be detected, including: The edge device acquires the image to be detected, uses the feature encoding module to perform feature extraction on the image to be detected to obtain an original feature map of the image to be detected, and compresses and quantizes the original feature map to obtain a compressed feature map; The edge device sends the compressed feature map to the cloud; The cloud receives the compressed feature map sent by the edge device, decodes the compressed feature map using the feature decoding module to obtain a decoded feature map, and adopts a coarse-grained and fine-grained collaborative compensation strategy to perform feature processing on the decoded feature map to obtain a multi-scale feature map of the image to be detected; The cloud uses the target detection module to perform target detection on the multi-scale feature map of the image to be detected, and obtains the target detection result of the image to be detected.

[0009] In some embodiments, the coarse-grained and fine-grained collaborative compensation strategy is used to perform feature processing on the decoded feature map to obtain a multi-scale feature map of the image to be detected, including: Extracting statistical information of the original feature map, wherein the statistical information includes a channel-level mean and variance of the original feature map; Based on the statistical information of the original feature map, adaptively adjust the channel level mean and variance of the decoded feature map, perform fine-grained compensation on the decoded feature map, and obtain a fine-grained compensated feature map; Based on the feature map after fine-grained compensation, coarse-grained compensation is performed to obtain the multi-scale feature map.

[0010] In a second aspect, the present invention further provides a cloud-edge collaborative target detection device based on a dynamic hypernetwork, comprising: A construction unit is used to construct a dynamic segmentation super network based on a YOLOv5s single-branch network, wherein the dynamic segmentation super network includes multiple convolutional layers and corresponding multiple segmentation points, a fine-grained compensation F-Com block and an Identity block are added to each segmentation point, and a coarse-grained compensation C-Com block is added to the end of the dynamic segmentation super network; A training unit, used for training the dynamic segmentation super network based on sample images to obtain a trained super network; An optimization unit, configured to construct an inference model based on the trained super network, deploy the inference model on edge devices and the cloud, perform a neural network architecture search in real time based on the inference model, determine an optimal segmentation point, determine an optimal subnetwork based on the optimal segmentation point, and obtain a target detection model; The target detection unit is used for the edge device and the cloud to collaboratively use the target detection model to perform target detection on the image to be detected, and obtain the target detection result of the image to be detected.

[0011] The cloud-edge collaborative target detection method based on dynamic hypernetwork provided by the present invention constructs a dynamic segmentation hypernetwork based on a YOLOv5s single-branch network, trains the dynamic segmentation hypernetwork based on sample images to obtain a trained hypernetwork, constructs an inference model based on the trained hypernetwork, deploys the inference model on edge devices and the cloud, performs a neural network architecture search in real time based on the inference model, determines the optimal segmentation point, obtains a target detection model, and the edge device and the cloud collaboratively use the target detection model to perform target detection on the image to be detected to obtain a target detection result, which can realize dynamic estimation of the optimal segmentation point of the hypernetwork and improve the reasoning efficiency and accuracy in resource-constrained scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 It is a statistical chart of the output data volume and computational complexity of each layer of YOLOv5s network provided by existing technology; Figure 2 It is a structural diagram of a cloud-edge collaborative target detection system based on a dynamic hypernetwork provided by an embodiment of the present invention; Figure 3 It is a flowchart of a cloud-edge collaborative target detection method based on a dynamic hypernetwork provided by an embodiment of the present invention; Figure 4 It is a flowchart of the training process of the dynamic segmentation hypernetwork provided by an embodiment of the present invention; Figure 5 is a schematic diagram of the structure of a coarse-grained and fine-grained compensation module provided by an embodiment of the present invention; Figure 6 It is a structural schematic diagram of a cloud-edge collaborative target detection device based on a dynamic hypernetwork provided by an embodiment of the present invention; Figure 7It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] Figure 1 The statistical chart of the output data volume and computational complexity of each layer of YOLOv5s network provided by the existing technology. Figure 1 As shown in the figure, as the deep neural network runs, the amount of output data decreases layer by layer, but the amount of calculation increases layer by layer. Therefore, a common method is to find an optimal segmentation point of the deep neural network (DNN) and then segment the model. The edge device is responsible for the reasoning of the first half, and transmits the features of the intermediate output to the cloud, which then completes the reasoning of the second half. However, the segmentation of the model is easily affected by bandwidth fluctuations and device computing power. Because the multiple branches of the network significantly increase the amount of transmitted features, it is difficult for existing segmentation algorithms to find the optimal segmentation point of the convolutional neural network in a dynamic environment.

[0016] Existing research compresses models and intermediate features to reduce computational and communication delays. Intermediate feature compression methods include the use of JPEG, PNG encoders, Huffman coding, autoencoders, and quantization techniques. Some studies also reduce the computational complexity of edge models through lightweight methods such as pruning, knowledge distillation, and Neural Architecture Search (NAS). However, most compression methods rely on fixed segmentation points to keep the intermediate feature quantities unchanged, which can increase transmission delays when bandwidth fluctuates. In addition, applying feature compression to different segmentation points of the network may lose key information, resulting in reduced model accuracy.

[0017] To this end, the present invention provides a cloud-edge collaborative target detection method based on a dynamic hypernetwork, by building a dynamic segmentation hypernetwork based on a YOLOv5s single-branch network, training the dynamic segmentation hypernetwork based on sample images to obtain a trained hypernetwork, building an inference model based on the trained hypernetwork, deploying the inference model on edge devices and the cloud, and performing a real-time neural network architecture search based on the inference model to determine the optimal segmentation point to obtain a target detection model. The edge device and the cloud collaboratively use the target detection model to perform target detection on the image to be detected to obtain a target detection result. The present invention can realize dynamic estimation of the optimal segmentation point of the hypernetwork, and can improve the efficiency and accuracy of reasoning in resource-constrained scenarios.

[0018] Figure 2 The schematic diagram of the structure of the cloud-edge collaborative target detection system based on the dynamic hypernetwork provided by the embodiment of the present invention. Figure 2 As shown, a cloud-edge collaborative target detection system based on a dynamic hypernetwork is provided, which includes an edge device end and a cloud server end. The edge device end and the cloud server end are used to collaboratively utilize the dynamic segmentation hypernetwork to perform target detection on the image to be detected. The edge device end is also used to realize adaptive segmentation point selection, and select different optimal segmentation points according to different application scenarios.

[0019] Figure 3 The following is a flow chart of a cloud-edge collaborative target detection method based on a dynamic hypernetwork provided by an embodiment of the present invention. Figure 3 As shown, a cloud-edge collaborative target detection method based on a dynamic hypernetwork is provided, comprising the following steps: step 310, step 320, step 330 and step 340. The method flow steps are only a possible implementation of the present invention.

[0020] Step 310: construct a dynamic segmentation super network based on the YOLOv5s single-branch network, the dynamic segmentation super network includes multiple convolutional layers and corresponding multiple segmentation points, add fine-grained compensation F-Com blocks and Identity blocks at each segmentation point, and add coarse-grained compensation C-Com blocks at the end of the dynamic segmentation super network.

[0021] Among them, YOLOv5s is the smallest model in the YOLOv5 series, with fewer parameters and less computation, and is suitable for use in resource-constrained environments; YOLOv5 is an efficient target detection model and the fifth version of the YOLO (You Only Look Once) series.

[0022] It should be noted that the dynamic segmentation network includes multiple sub-networks corresponding to multiple segmentation points, each sub-network includes a segmentation point, and the segmentation point is located between two adjacent convolutional layers.

[0023] Among them, the hypernetwork is a neural network architecture. The hypernetwork provides a new perspective for the design of neural network architecture. By dynamically generating network parameters, it can achieve more flexible and efficient model design, which is suitable for processing a variety of complex tasks.

[0024] Among them, the F-Com (Feature Communication) block enhances the feature expression capability through cross-layer feature communication, so that the shallow detail features and deep semantic features can be effectively combined; the Identity block directly transmits the input to the output through jump connections, which helps to transmit information in the network; the C-Com (Class Communication) block optimizes the classification performance through information exchange between categories.

[0025] It should be noted that the F-Com block operates at the optimal split point to highly compress the output feature map for more efficient transmission between the edge device and the cloud server.

[0026] Step 320: Based on the sample image, the dynamic segmentation hypernetwork is trained to obtain a trained hypernetwork.

[0027] Optionally, multiple sample images are obtained, and a multi-scale feature label of each sample image is determined.

[0028] Optionally, each sample image is input into a dynamic segmentation hypernetwork to obtain a prediction result of a multi-scale feature of each sample image output by the dynamic segmentation hypernetwork.

[0029] Optionally, the loss function value is calculated based on the prediction result of the multi-scale feature of each sample image and the multi-scale feature label of each sample image.

[0030] Based on the loss function value, the parameters of the dynamic segmentation hypernetwork are iteratively optimized.

[0031] Step 330: Based on the trained super network, build an inference model, deploy the inference model on edge devices and the cloud, and based on the inference model, perform a real-time neural network architecture search to determine the optimal segmentation point. Based on the optimal segmentation point, determine the optimal subnetwork to obtain a target detection model.

[0032] Optionally, obtain a current bandwidth and a current signal-to-noise ratio of a communication channel from the edge device to the cloud.

[0033] Optionally, based on the current bandwidth and the current signal-to-noise ratio, a neural network architecture search is performed in real time to determine the optimal segmentation point.

[0034] Optionally, the optimal sub-network is combined with the detection head to obtain an object detection model.

[0035] It should be noted that thanks to the hypernetwork design, the same inference model can be deployed on edge devices and the cloud, saving the storage overhead of split point switching and multiple models; the edge device selects the optimal split point based on its computing power and communication rate, and shares the optimal split point information with the cloud to synchronously switch to the same subnet architecture.

[0036] It can be understood that real-time neural network architecture search and determination of the optimal segmentation point can achieve dynamic estimation of the optimal segmentation point, which has strong real-time and flexibility and can adapt to dynamically changing application scenarios.

[0037] Step 340: The edge device and the cloud collaboratively use the target detection model to perform target detection on the image to be detected, and obtain the target detection result of the image to be detected.

[0038] Optionally, the edge device uses the target detection model to extract, compress and quantize features of the image to be detected, obtains a compressed feature map, and transmits the compressed feature map to the cloud.

[0039] Optionally, the cloud uses a target detection model to decode, perform fine-grained compensation, and coarse-grained compensation on the compressed feature map to obtain multi-scale features, and obtain target detection results based on the multi-scale features.

[0040] In an embodiment of the present invention, a dynamic segmentation supernetwork is constructed based on a YOLOv5s single-branch network, and the dynamic segmentation supernetwork is trained based on a sample image to obtain a trained supernetwork. An inference model is constructed based on the trained supernetwork, and the inference model is deployed on edge devices and the cloud. Based on the inference model, a neural network architecture search is performed in real time to determine the optimal segmentation point and obtain a target detection model. The edge device and the cloud collaboratively use the target detection model to perform target detection on the image to be detected and obtain a target detection result. Dynamic estimation of the optimal position of the cutting point can be achieved, and the reasoning efficiency and accuracy can be improved in resource-constrained scenarios.

[0041] In some embodiments, based on the inference model, a neural network architecture search is performed to determine the optimal segmentation point, including: The objective function is constructed with the goal of minimizing the total inference delay of the inference model. The total inference delay includes the computing delay of the edge device, the data transmission delay, and the computing delay of the cloud. An adaptive segmentation point selection algorithm is used to solve the objective function and determine the optimal segmentation point from multiple segmentation points.

[0042] Optionally, the current bandwidth and the current signal-to-noise ratio are obtained, and based on the current bandwidth and the current signal-to-noise ratio, the Shannon formula is used to calculate the communication transmission rate from the edge device to the cloud.

[0043] Optionally, the data transmission delay is calculated based on the communication transmission rate.

[0044] Optionally, an adaptive segmentation point selection algorithm is used to determine the optimal segmentation point, including: S1, input: edge computing delay set, cloud computing delay set, communication bandwidth B, signal-to-noise ratio SNR, segmentation point set N; S2. Output: Optimal segmentation subnet ; S3. Initialization: segmentation point selection vector ; S4、fori in ; S5. Edge computing latency of computing network 0-i layers , and the cloud computing latency of (i+1)-N layers , get the feature quantity of the i-th layer output ; S6, ; S7, ; S8, ; S9, end for; S10, ; S11, .

[0045] Optionally, set a set of binary vectors Used to indicate the selection of the F-Com block. Each split point can choose to enable the F-Com block or the Identity block.

[0046] F-Com block: When the i-th layer is selected as the optimal split point, set the i-th F-Com block ; At this split point, the output features are channel compressed and quantized on the edge device side to reduce the amount of data transmission; in the cloud, fine-grained compensation is used to restore the compressed feature information.

[0047] Identity block: If the i-th layer is not the optimal split point, set , the features are passed directly to the next layer of the network through the Identity block.

[0048] Optionally, after determining the optimal split point p, set the pth F-Com block , the optimal segmentation sub-network is obtained at the segmentation point p between the edge device and the cloud.

[0049] In some embodiments, the objective function is calculated as follows: ; ; in, represents the total inference delay, Indicates that edge devices use inference models The computational latency of data processing at the layer, represents the data transmission delay from edge devices to the cloud. Indicates the inference model used in the cloud The computational latency of data processing in the layer, is the current bandwidth of the communication channel between the edge device and the cloud, is the current signal-to-noise ratio of the communication channel, The inference model The amount of feature data output by the layer represents the amount of feature data transmitted from the edge device to the cloud. Represents the inference model layer, Represents the inference model layer, , Represents the total number of convolutional layers of the inference model.

[0050] It should be noted that according to Shannon’s theorem, the bandwidth 𝐵 and the signal-to-noise ratio 𝑆𝑁𝑅 determine the maximum transmission rate of the communication channel.

[0051] Figure 4 The following is a flow chart of the training process of the dynamic segmentation hypernetwork provided by the embodiment of the present invention. Figure 4 As shown, in some embodiments, based on the sample image, the dynamic segmentation hypernetwork is trained to obtain the trained hypernetwork, including: Step 410: Set the binary vector groups corresponding to the fine-grained compensation F-Com blocks and Identity blocks of multiple segmentation points, so that the target segmentation point selects the corresponding fine-grained compensation F-Com block for feature compression and fine-grained feature compensation, and the common segmentation point selects the corresponding Identity block for direct feature transfer; Step 420: dynamically adjust the binary vector group, adopt a uniform sampling strategy, train sub-networks corresponding to different target segmentation points based on the sample image at the same time, and calculate the cumulative gradient of each sub-network; Step 430: Based on the accumulated gradient of each sub-network, the parameters of each sub-network are optimized to obtain a trained super-network.

[0052] It should be noted that during the training of the hypernetwork, the vector group , in order to train sub-networks with different segmentation points at the same time. Since each sub-network has only one segmentation point, only one F-Com block of the segmentation point network is selected at the same time. The vector group It is limited to a one-hot vector; the uniform sampling strategy is used to sample the sub-networks, and the accumulated gradients of the sub-networks are used to optimize the super-network, which can effectively ensure that all sub-networks are fully trained.

[0053] In some embodiments, the object detection model includes a feature encoding module, a feature decoding module, and an object detection module; The edge device and the cloud collaboratively use the target detection model to perform target detection on the image to be detected, and obtain the target detection results of the image to be detected, including: The edge device obtains the image to be detected, uses the feature encoding module to extract features of the image to be detected, obtains the original feature map of the image to be detected, compresses and quantizes the original feature map, and obtains a compressed feature map; The edge device sends the compressed feature map to the cloud; The cloud receives the compressed feature map sent by the edge device, decodes the compressed feature map using the feature decoding module to obtain the decoded feature map, and uses the coarse-grained and fine-grained collaborative compensation strategy to perform feature processing on the decoded feature map to obtain a multi-scale feature map of the image to be detected; The cloud uses the target detection module to perform target detection on the multi-scale feature map of the image to be detected, and obtains the target detection result of the image to be detected.

[0054] It can be understood that by collaboratively utilizing the target detection model based on the dynamic segmentation hypernetwork by edge devices and the cloud, the detected image can be processed in real time, which improves the reasoning efficiency, has strong real-time and flexibility, and is suitable for complex application scenarios with dynamic changes and limited resources. By adopting a coarse-grained and fine-grained collaborative compensation strategy to compensate for the restored features, the coarse-grained compensation restores the multi-level feature information in the network, while the fine-grained compensation restores the single-layer compressed feature information at the segmentation point, thereby improving the reasoning accuracy and the accuracy of target detection.

[0055] Figure 5 The schematic diagram of the structure of the coarse-grained and fine-grained compensation module provided by the embodiment of the present invention is shown in FIG. Figure 5 As shown in the figure, the fine-grained compensation F-Com block is inserted at each segmentation point of the dynamic segmentation super network to compress the features output at the segmentation point and perform fine-grained compensation based on the statistical information of the original features; the coarse-grained compensation C-Com block is placed at the end of the dynamic segmentation super network to restore multiple high-level features through a single-layer low-level feature for coarse-grained compensation.

[0056] In some embodiments, a coarse-grained and fine-grained collaborative compensation strategy is adopted to perform feature processing on the decoded feature map to obtain a multi-scale feature map of the image to be detected, including: Extracting statistical information of the original feature map, including the channel-level mean and variance of the original feature map; Based on the statistical information of the original feature map, the channel-level mean and variance of the decoded feature map are adaptively adjusted, and the decoded feature map is fine-grained compensated to obtain a fine-grained compensated feature map; Based on the feature map after fine-grained compensation, coarse-grained compensation is performed to obtain a multi-scale feature map.

[0057] Optionally, the expression for compressing and quantizing the original feature map by the fine-grained compensation F-Com block is as follows: ; ; in, represents the original feature map, Represents compressed feature map, ConvReduce represents feature compression, and Sign is the feature quantization function; are the statistical information of the original feature map, which respectively represent the channel-level mean and variance of the original feature map.

[0058] Optionally, the compressed feature map is decoded as follows: ; in, It represents the decoded feature map, and ConvRecover represents the feature recovery process.

[0059] Optionally, the mean and variance of the decoded feature map are matched with the mean and variance of the original feature map, as follows: ; in, represents the feature map after fine-grained compensation, They represent the channel-level mean and variance of the decoded feature map respectively.

[0060] It should be noted that the C-Com block is used to recover multiple low-level features from a single high-level feature and pass them to the feature pyramid of the neck. Through the compensation mechanism, the edge device no longer needs to transmit multi-branch features, but only needs to transmit the features of the current layer, reducing the amount of multi-branch feature transmission; in addition, the C-Com block can also perform coarse-grained reconstruction of quantitative features through a deconvolution network, thereby improving the feature expression capability.

[0061] It can be understood that by performing feature compensation in the form of fine-grained compensation and coarse-grained compensation, the restored features can be made closer to the original features, which helps to reduce the impact of quantization and improve the accuracy of the restored features.

[0062] Example: 1) Dataset description: The embodiment of the present invention trains and tests the dynamic segmentation hypernetwork on the COCO2017 dataset. The COCO dataset contains more than 200,000 high-resolution images, covering 80 categories of objects in natural scenes. The dataset includes a training set, a validation set, and a test set, wherein the training set contains about 118,000 images and the validation set contains about 5,000 images. The target objects in the dataset vary in size, ranging from small to large objects. During the training process, the input images were scaled to 640 × 640 for all experiments, and the performance was reported on the validation dataset.

[0063] 2) The experimental setup is as follows: The embodiment of the present invention uses NVIDIA Jetson TX2 as an edge device and NVIDIA A100 as a cloud server device. The model is based on the PyTorch framework and is trained in the cloud. During the training process, the Stochastic Gradient Descent (SGD) optimizer is used, and the initial learning rate is 1e-2, and the learning rate decays to 2e-3 with training. The total number of training rounds in the training process is 400 (epochs) and the batch size is 32. Due to the presence of binary non-differentiable operations in model training, the Straight-Through Estimator (STE) strategy is adopted to directly pass the gradient of the last layer of the quantization module back to the previous layer of quantization. When training the super network model, based on the weight sharing strategy, the sub-networks of four branches are trained simultaneously in each epoch. The mean average precision (mAP) is used as the evaluation indicator of model accuracy.

[0064] 3) Ablation analysis: Table 1 is the ablation experiment results provided by an embodiment of the present invention. The original YOLOv5 model is used as the baseline, without modification, the feature size is several thousand KB at each segmentation point, and the accuracy is 0.567. After adding C-Com, the amount of multi-branch transmission features of the network is significantly reduced, while the accuracy is reduced by no more than 0.4%. Through channel compression and 1-bit quantization, the features are reduced by 128 times, but the accuracy is reduced, especially at segmentation points 3 and 10, the accuracy is lost by 2.1% and 3.4% compared to the baseline. Through the compensation strategy, the accuracy of segmentation points 3 and 10 is improved by 1.2% and 1.4% respectively, while the feature inclusion statistical information is only increased by 1KB and 3KB, and the accuracy is lost by 1.7% and 2% compared to the baseline. These results show that DS-Net not only effectively reduces the amount of transmitted features, but also effectively reduces the accuracy loss caused by feature compression.

[0065] Table 1 Ablation experiment results provided by the embodiment of the present invention

[0066] 4) Adaptive analysis of segmentation points: In order to evaluate the adaptive reasoning performance of the cloud-edge collaborative target detection method based on dynamic hypernetwork proposed in the embodiment of the present invention under different bandwidth conditions, WonderShaper is used to control the upload bandwidth, and three cases are simulated: 8 Mbps, 16 Mbps, and 56 Mbps. Comparative experiments are performed using three settings: cloud only, edge only, and the proposed method. The reasoning latency, accuracy, and other parameters of these methods under different bandwidths are compared.

[0067] Table 2 shows the adaptability analysis results under different bandwidths provided by an embodiment of the present invention. When edge-only computing is performed, the inference latency is 70.2ms and the accuracy is 0.567. This shows that edge-only device processing provides low latency while maintaining accuracy, but it lacks the collaborative capabilities of the cloud, which puts a huge burden on edge devices. For methods that only use the cloud, the inference latency is severely affected by bandwidth. At 8 Mbps, the longer transmission time results in a total inference latency of 423.5 ms. Even when the bandwidth is increased to 56 Mbps, the total latency is only reduced to 116.8 ms. This shows that the transmission bottleneck in edge computing under low bandwidth conditions can significantly affect the inference efficiency.

[0068] In contrast, the method proposed in the embodiment of the present invention shows obvious advantages under different bandwidth conditions. At a speed of 8Mbps, the proposed method segments the hypernetwork at the 10th layer and effectively compresses the feature data to 6 KB, achieving a total latency of 68.5 ms. Due to the limited network bandwidth, the proposed method selects a later segmentation point to reduce the transmission latency by transmitting fewer features. At a speed of 16 Mbps, the proposed method selects the 5th layer for segmentation and compresses the feature data to 15 KB, resulting in a total inference latency of 61.0 ms, which is 76% lower than the cloud-only method, with only a 1.1% loss in accuracy. When the bandwidth increases to 56 Mbps, the proposed method segments at the 3rd layer and compresses the feature data to 26 KB, further reducing the total latency to 53.6 ms. Due to the abundant network transmission resources and the low transmission latency, the earlier segmentation point is selected to minimize the computational latency. These results show that the proposed method maintains low inference latency under different bandwidths while ensuring accuracy by flexibly switching segmentation points and adopting efficient coarse-fine granularity compensation technology.

[0069] Table 2 Adaptability analysis results under different bandwidths provided by the embodiments of the present invention

[0070] 5) Comparison with other methods: Table 3 shows the comparative analysis results between the proposed method in the embodiment of the present invention and other existing methods. All methods are based on YOLOv5 as a benchmark, and the bandwidth is set to 16Mbps. Compared with other methods, the method proposed in the embodiment of the present invention has superior performance. The 1-bit quantization method provides moderate latency and feature size, resulting in a 40% decrease in network accuracy and a 46% increase in latency compared to the proposed method. In contrast, int8 quantization shows a higher mAP (0.560), but at the cost of higher latency and larger feature size, with latency almost increased by 10 times compared to the proposed method. The JPEG compressor performs well in terms of latency, but the lossy compression of arbitrary information leads to lower accuracy, which is 49.7% and 26.4% lower than the proposed method, respectively. BottleNet++ adopts a traditional encoder and decoder architecture and takes channel noise into account, which shows that it is not efficient in terms of speed and storage. DeepAdaln-Net achieves a good balance between latency (63.3ms), feature size (26KB), and mAP (0.552), but the segmentation point is fixed and the performance improvement is still limited. Compared with DeepAdaln-Net, the proposed method reduces the feature size by 42% and improves the accuracy by 0.4%, showing the best performance of dynamic segmentation and compensation technology under bandwidth changes.

[0071] Table 3 Comparative analysis results between the method proposed in the embodiment of the present invention and other existing methods

[0072] The following describes a cloud-edge collaborative target detection device based on a dynamic hypernetwork provided in an embodiment of the present invention. The cloud-edge collaborative target detection device based on a dynamic hypernetwork described below and the cloud-edge collaborative target detection method based on a dynamic hypernetwork described above can refer to each other.

[0073] Figure 6 A schematic diagram of the structure of a cloud-edge collaborative target detection device based on a dynamic hypernetwork provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown, the cloud-edge collaborative target detection device 600 based on dynamic hypernetwork includes: A construction unit 610 is used to construct a dynamic segmentation super network based on a YOLOv5s single-branch network, wherein the dynamic segmentation super network includes multiple convolutional layers and corresponding multiple segmentation points, a fine-grained compensation F-Com block and an Identity block are added to each segmentation point, and a coarse-grained compensation C-Com block is added to the end of the dynamic segmentation super network; A training unit 620, configured to train the dynamic segmentation super network based on the sample image to obtain a trained super network; The optimization unit 630 is used to build an inference model based on the trained super network, deploy the inference model on the edge device and the cloud, perform a neural network architecture search in real time based on the inference model, determine the optimal segmentation point, determine the optimal sub-network based on the optimal segmentation point, and obtain the target detection model; The target detection unit 640 is used for the edge device and the cloud to collaboratively use the target detection model to perform target detection on the image to be detected, and obtain the target detection result of the image to be detected.

[0074] Optionally, based on the inference model, a neural network architecture search is performed to determine the optimal split point, including: The objective function is constructed with the goal of minimizing the total inference delay of the inference model. The total inference delay includes the computing delay of the edge device, the data transmission delay, and the computing delay of the cloud. An adaptive segmentation point selection algorithm is used to solve the objective function and determine the optimal segmentation point from multiple segmentation points.

[0075] Optionally, the objective function is calculated as follows: ; ; in, represents the total inference delay, Indicates that edge devices use inference models The computational latency of data processing in the layer, represents the data transmission delay from edge devices to the cloud. Indicates the inference model used in the cloud The computational latency of data processing in the layer, is the current bandwidth of the communication channel between the edge device and the cloud, is the current signal-to-noise ratio of the communication channel, The inference model The amount of feature data output by the layer represents the amount of feature data transmitted from the edge device to the cloud. Represents the inference model layer, Represents the inference model layer, , Represents the total number of convolutional layers of the inference model.

[0076] Optionally, based on the sample image, the dynamic segmentation hypernetwork is trained to obtain a trained hypernetwork, including: Set the binary vector groups corresponding to the fine-grained compensation F-Com blocks and Identity blocks of multiple segmentation points, so that the target segmentation point selects the corresponding fine-grained compensation F-Com block for feature compression and fine-grained feature compensation, and the common segmentation point selects the corresponding Identity block for direct feature transfer; Dynamically adjust the binary vector group, adopt a uniform sampling strategy, train the sub-networks corresponding to different target segmentation points based on the sample image, and calculate the cumulative gradient of each sub-network; Based on the accumulated gradient of each sub-network, the parameters of each sub-network are optimized to obtain the trained super-network.

[0077] Optionally, the target detection model includes a feature encoding module, a feature decoding module and a target detection module; The target detection model is used to perform target detection on the image to be detected, and the target detection result of the image to be detected is obtained, including: The edge device obtains the image to be detected, uses the feature encoding module to extract features of the image to be detected, obtains the original feature map of the image to be detected, compresses and quantizes the original feature map, and obtains a compressed feature map; The edge device sends the compressed feature map to the cloud; The cloud receives the compressed feature map sent by the edge device, decodes the compressed feature map using the feature decoding module to obtain the decoded feature map, and uses the coarse-grained and fine-grained collaborative compensation strategy to perform feature processing on the decoded feature map to obtain a multi-scale feature map of the image to be detected; The cloud uses the target detection module to perform target detection on the multi-scale feature map of the image to be detected, and obtains the target detection result of the image to be detected.

[0078] Optionally, a coarse-grained and fine-grained collaborative compensation strategy is adopted to perform feature processing on the decoded feature map to obtain a multi-scale feature map of the image to be detected, including: Extracting statistical information of the original feature map, including the channel-level mean and variance of the original feature map; Based on the statistical information of the original feature map, the channel-level mean and variance of the decoded feature map are adaptively adjusted, and the decoded feature map is fine-grained compensated to obtain a fine-grained compensated feature map; Based on the feature map after fine-grained compensation, coarse-grained compensation is performed to obtain a multi-scale feature map.

[0079] It should be noted here that the cloud-edge collaborative target detection device based on dynamic hypernetwork provided in an embodiment of the present invention can implement all the method steps implemented in the above-mentioned cloud-edge collaborative target detection method embodiment based on dynamic hypernetwork, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0080] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 7 As shown, the electronic device may include: a processor (processor) 710 , a communication interface (Communications Interface) 720 , a memory (memory) 730 and a communication bus 740 , wherein the processor 710 , the communication interface 720 , and the memory 730 communicate with each other through the communication bus 740 . The processor 710 can call the logic instructions in the memory 730 to execute a cloud-edge collaborative target detection method based on a dynamic supernetwork, the method including: constructing a dynamic segmentation supernetwork based on a YOLOv5s single-branch network, the dynamic segmentation supernetwork including multiple convolutional layers and corresponding multiple segmentation points, adding fine-grained compensation F-Com blocks and Identity blocks at each segmentation point, and adding coarse-grained compensation C-Com blocks at the end of the dynamic segmentation supernetwork; based on the sample image, training the dynamic segmentation supernetwork to obtain a trained supernetwork; based on the trained supernetwork, constructing an inference model, deploying the inference model on edge devices and the cloud, based on the inference model, performing a real-time neural network architecture search to determine the optimal segmentation point, and based on the optimal segmentation point, determining the optimal subnetwork to obtain a target detection model; using the target detection model to perform target detection on the image to be detected to obtain a target detection result of the image to be detected.

[0081] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0082] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the cloud-edge collaborative target detection method based on the dynamic supernetwork provided by the above methods, and the method includes: constructing a dynamic segmentation supernetwork based on a YOLOv5s single-branch network, the dynamic segmentation supernetwork includes multiple convolutional layers and corresponding multiple segmentation points, adding fine-grained compensation F-Com blocks and Identity blocks at each segmentation point, and adding coarse-grained compensation C-Com blocks at the end of the dynamic segmentation supernetwork; based on the sample image, the dynamic segmentation supernetwork is trained to obtain a trained supernetwork; based on the trained supernetwork, an inference model is constructed, and the inference model is deployed on the edge device and the cloud; based on the inference model, a neural network architecture search is performed in real time to determine the optimal segmentation point, and based on the optimal segmentation point, the optimal subnetwork is determined to obtain a target detection model; using the target detection model to perform target detection on the image to be detected, and obtain the target detection result of the image to be detected.

[0083] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the cloud-edge collaborative target detection method based on a dynamic supernetwork provided by the above-mentioned methods, the method comprising: constructing a dynamic segmentation supernetwork based on a YOLOv5s single-branch network, the dynamic segmentation supernetwork comprising multiple convolutional layers and corresponding multiple segmentation points, adding fine-grained compensation F-Com blocks and Identity blocks at each segmentation point, and adding coarse-grained compensation C-Com blocks at the end of the dynamic segmentation supernetwork; based on a sample image, training the dynamic segmentation supernetwork to obtain a trained supernetwork; based on the trained supernetwork, constructing an inference model, deploying the inference model on edge devices and the cloud, and based on the inference model, performing a real-time neural network architecture search to determine the optimal segmentation point, and based on the optimal segmentation point, determining the optimal subnetwork to obtain a target detection model; using the target detection model to perform target detection on the image to be detected to obtain a target detection result of the image to be detected.

[0084] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0085] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cloud-edge collaborative target detection method based on dynamic hypernetwork, characterized in that: include: A dynamic segmentation super network is constructed based on a YOLOv5s single-branch network, wherein the dynamic segmentation super network includes multiple convolutional layers and corresponding multiple segmentation points, a fine-grained compensation F-Com block and an Identity block are added to each segmentation point, and a coarse-grained compensation C-Com block is added to the end of the dynamic segmentation super network; Based on the sample image, training the dynamic segmentation hypernetwork to obtain a trained hypernetwork; Based on the trained super network, an inference model is constructed, and the inference model is deployed on edge devices and the cloud. Based on the inference model, a neural network architecture search is performed in real time to determine an optimal segmentation point, and based on the optimal segmentation point, an optimal subnetwork is determined to obtain a target detection model; The edge device and the cloud collaboratively use the target detection model to perform target detection on the image to be detected, and obtain the target detection result of the image to be detected.

2. The cloud-edge collaborative target detection method based on dynamic hypernetwork according to claim 1 is characterized in that: The method of searching a neural network architecture in real time based on the inference model to determine an optimal segmentation point includes: An objective function is constructed with the goal of minimizing the total inference delay of the inference model, where the total inference delay includes the computation delay of the edge device, the data transmission delay, and the computation delay of the cloud; The objective function is solved by using an adaptive segmentation point selection algorithm, and an optimal segmentation point is determined from the multiple segmentation points.

3. The cloud-edge collaborative target detection method based on dynamic hypernetwork according to claim 2 is characterized in that: The calculation formula of the objective function is as follows: ； ； in, represents the total inference delay, Indicates that the edge device adopts the inference model The computational latency of data processing at the layer, represents the data transmission delay from the edge device to the cloud, Indicates that the cloud uses the inference model The computational latency of data processing at the layer, is the current bandwidth of the communication channel between the edge device and the cloud, is the current signal-to-noise ratio of the communication channel, is the first The amount of feature data output by the layer represents the amount of feature data transmitted by the edge device to the cloud. The first layer, The first layer, , Represents the total number of convolutional layers of the inference model.

4. The cloud-edge collaborative target detection method based on dynamic hypernetwork according to claim 1 is characterized in that: The step of training the dynamic segmentation hypernetwork based on the sample image to obtain the trained hypernetwork includes: Setting the binary vector groups corresponding to the fine-grained compensation F-Com blocks and the Identity blocks of the plurality of segmentation points, so that the target segmentation point selects the corresponding fine-grained compensation F-Com block for feature compression and fine-grained feature compensation, and the common segmentation point selects the corresponding Identity block for direct feature transfer; Dynamically adjust the binary vector group, adopt a uniform sampling strategy, simultaneously train sub-networks corresponding to different target segmentation points based on the sample image, and calculate the cumulative gradient of each sub-network; Based on the accumulated gradient of each sub-network, the parameters of each sub-network are optimized to obtain a trained super-network.

5. The cloud-edge collaborative target detection method based on dynamic hypernetwork according to claim 1 is characterized in that: The target detection model includes a feature encoding module, a feature decoding module and a target detection module; The edge device and the cloud collaboratively use the target detection model to perform target detection on the image to be detected, and obtain the target detection result of the image to be detected, including: The edge device acquires the image to be detected, uses the feature encoding module to perform feature extraction on the image to be detected to obtain an original feature map of the image to be detected, and compresses and quantizes the original feature map to obtain a compressed feature map; The edge device sends the compressed feature map to the cloud; The cloud receives the compressed feature map sent by the edge device, decodes the compressed feature map using the feature decoding module to obtain a decoded feature map, and adopts a coarse-grained and fine-grained collaborative compensation strategy to perform feature processing on the decoded feature map to obtain a multi-scale feature map of the image to be detected; The cloud uses the target detection module to perform target detection on the multi-scale feature map of the image to be detected, and obtains the target detection result of the image to be detected.

6. The cloud-edge collaborative target detection method based on dynamic hypernetwork according to claim 5 is characterized in that: The method adopts a coarse-grained and fine-grained collaborative compensation strategy to perform feature processing on the decoded feature map to obtain a multi-scale feature map of the image to be detected, including: Extracting statistical information of the original feature map, wherein the statistical information includes a channel-level mean and variance of the original feature map; Based on the statistical information of the original feature map, adaptively adjust the channel level mean and variance of the decoded feature map, perform fine-grained compensation on the decoded feature map, and obtain a fine-grained compensated feature map; Based on the feature map after fine-grained compensation, coarse-grained compensation is performed to obtain the multi-scale feature map.

7. A cloud-edge collaborative target detection device based on dynamic hypernetwork, characterized in that: include: A construction unit is used to construct a dynamic segmentation super network based on a YOLOv5s single-branch network, wherein the dynamic segmentation super network includes multiple convolutional layers and corresponding multiple segmentation points, a fine-grained compensation F-Com block and an Identity block are added to each segmentation point, and a coarse-grained compensation C-Com block is added to the end of the dynamic segmentation super network; A training unit, used for training the dynamic segmentation super network based on sample images to obtain a trained super network; An optimization unit, configured to construct an inference model based on the trained super network, deploy the inference model on edge devices and the cloud, perform a neural network architecture search in real time based on the inference model, determine an optimal segmentation point, determine an optimal subnetwork based on the optimal segmentation point, and obtain a target detection model; The target detection unit is used for the edge device and the cloud to collaboratively use the target detection model to perform target detection on the image to be detected, and obtain the target detection result of the image to be detected.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the cloud-edge collaborative target detection method based on dynamic hypernetwork as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the cloud-edge collaborative target detection method based on a dynamic hypernetwork as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the cloud-edge collaborative target detection method based on a dynamic hypernetwork as described in any one of claims 1 to 6.

Images