Edge cloud cooperative detection method, system and device based on cross-layer feature consistency

Abstract

This invention discloses an edge-cloud collaborative detection method, system, and device based on cross-layer feature consistency. It achieves low-latency detection of common categories through a lightweight target detection model at the edge. By comparing the degree of cross-layer feature consistency between shallow and deep features through an offloading decision model, it determines whether the image to be detected exceeds the representational capacity of the edge detection model, achieving semantic-aware offloading decision for rare targets. The image to be detected containing rare targets is offloaded to a large cloud model for enhanced detection. This significantly improves the overall detection accuracy in long-tail scenarios while ensuring system real-time performance, realizing edge-cloud collaborative detection for long-tail target detection scenarios.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, computer vision, edge computing and edge-cloud collaborative computing technology, and relates to an edge-cloud collaborative long-tail target detection method, system and device based on cross-layer feature consistency. Background Technology

[0002] With the continuous growth of visual data scale in edge computing environments, object detection has become a key foundational capability in intelligent perception systems and is widely used in scenarios such as autonomous driving, industrial inspection, intelligent monitoring, augmented reality, and power equipment inspection. Object detection tasks require systems to accurately identify and locate various target objects from input images, thereby providing support for subsequent semantic understanding, state analysis, and intelligent decision-making.

[0003] In this process, deep learning-based object detection models play a crucial role. Compared to traditional rule-based image processing methods, deep learning models have stronger feature extraction and pattern recognition capabilities, significantly improving object detection accuracy in complex scenes. However, to achieve higher detection performance, existing object detection models often have complex network structures, resulting in large parameter sizes, high computational overhead, and high inference latency, limiting their practical deployment and operational efficiency on edge devices.

[0004] In practical edge intelligent detection tasks, inference is typically performed collaboratively by the edge and the cloud. Edge devices handle real-time processing of regular images, while the cloud utilizes its superior computing resources to process complex images or difficult-to-identify targets, thereby improving overall detection performance. However, due to limitations in computing resources, storage capacity, and power consumption budgets for edge devices, target detection models deployed at the edge usually require lightweight architectures. This makes it difficult for these models to maintain low-latency inference capabilities while simultaneously achieving high-precision recognition of all target categories.

[0005] Especially in real-world scenarios, target categories typically exhibit a pronounced long-tail distribution, meaning a few high-frequency categories dominate the majority of samples, while a large number of low-frequency categories are scarce, exhibit significant intra-class variations, and possess complex features. For such long-tailed data, lightweight edge models are better able to learn stable features of common categories, but their ability to represent low-frequency rare categories is clearly insufficient, resulting in poor detection performance for rare categories. Although large cloud-based models can compensate for the shortcomings of edge models in rare category detection to some extent, uploading all images to the cloud for processing incurs significant communication overhead and additional latency, making it difficult to meet real-time detection requirements.

[0006] Existing edge-cloud collaborative detection methods typically employ a confidence-based offloading strategy, where the image is sent to the cloud for further processing when the edge model outputs a low confidence score. However, low confidence scores can arise from the presence of rare targets in the image that exceed the representation capabilities of the edge model, or from data perturbations such as occlusion, background interference, and noise. Therefore, a simple confidence-based offloading method lacks an effective characterization of the image's semantic features and the boundaries of the model's representation capabilities, making it difficult to accurately determine whether the current image truly needs to be offloaded to the cloud.

[0007] Therefore, a new technical solution is urgently needed to effectively solve the above problems, and to improve the detection accuracy of rare targets while taking into account the low latency detection capability at the edge, which has significant application value and engineering significance. Summary of the Invention

[0008] This invention aims to provide an edge-cloud collaborative detection method, system, and device based on cross-layer feature consistency. While ensuring low-latency detection capabilities at the edge, it improves the detection accuracy of rare targets. It is suitable for target detection tasks in resource-constrained edge devices, and is especially suitable for general vision scenarios and target detection scenarios with long-tailed category distributions, such as power line inspection, industrial inspection, and intelligent monitoring.

[0009] To achieve the above objectives, the technical solution adopted by this invention is: an edge-cloud collaborative detection method based on cross-layer feature consistency, comprising:

[0010] S1. Input the image to be detected into the edge detection model, obtain shallow and deep features, and obtain the edge detection results;

[0011] S2. Input the shallow features and the deep features into the constructed unloading decision model, and obtain the unloading decision result; the unloading decision result is obtained based on whether the cross-layer feature consistency score of the shallow features and the deep features is less than a preset decision threshold. If it is less than the threshold, the unloading decision result is a first identifier; if it is not less than the threshold, the unloading decision result is a second identifier; wherein, the first identifier indicates that the image to be detected may include rare category targets, and the second identifier indicates that the image to be detected only contains common category targets;

[0012] S3. When the unloading decision result is the first identifier, the image to be detected is sent to the cloud detection model to obtain the cloud detection result. The cloud detection result is then fused with the edge detection result to obtain the final detection result. When the unloading decision result is the second identifier, the final detection result is the edge detection result.

[0013] The solution provided by this invention, based on the long-tail distribution characteristics of target categories in real-world scenes, assigns images containing only common categories to a lightweight target detection model at the edge, while images containing rare categories are processed by a large cloud-based model. The lightweight edge-based target detection model achieves low-latency detection of common categories. An offloading decision model compares the cross-layer feature consistency between shallow and deep features to determine whether the image exceeds the representational capacity of the edge-based detection model, enabling semantic-aware offloading decisions for rare targets. Images containing rare targets are then offloaded to a large cloud-based model for enhanced detection, significantly improving overall detection accuracy in long-tail scenarios while ensuring system real-time performance.

[0014] According to embodiments of the present invention, the present invention can be further optimized, and the optimized technical solution is as follows:

[0015] In one preferred embodiment, prior to step S1, the process of classifying the common category and the rare category specifically includes:

[0016] The total number of instances in the object detection dataset is calculated using the following formula:

[0017] ;

[0018] in, The total number of instances in the target detection dataset. The total number of categories in the target detection dataset. For the first Number of class target instances;

[0019] The proportion of each category instance in the object detection dataset is obtained by the following formula:

[0020] ;

[0021] in, For the first Percentage of class instances;

[0022] Sort the target instances of each category from highest to lowest, and obtain the category frequency distribution sequence;

[0023] The cumulative percentage of instances by category is calculated using the following formula:

[0024] ;

[0025] in, The first one in the frequency distribution sequence of the category The cumulative percentage of instances in each category ;

[0026] When the cumulative instance percentage When the preset segmentation threshold is reached, Corresponding front One category is classified as the common category, and the remaining categories are classified as the rare category.

[0027] In one preferred embodiment, in step S1, the edge detection model includes a backbone network, a feature aggregation network, and a detection head; the process of obtaining the edge detection result specifically includes: the image to be detected is input into the backbone network to obtain backbone feature maps at multiple scales; at least one layer is selected from the backbone feature maps at multiple scales as a shallow feature map, and the backbone feature maps at multiple scales are input into the feature aggregation network to obtain a deep semantic feature map; the deep semantic feature map is input into the detection head to output the edge detection result.

[0028] The above scheme extracts shallow and deep features from the edge detection model. It utilizes the structural characteristics of convolutional neural networks, where shallow features can reflect all the underlying information of the target in the image, and deep features mainly encode the semantic information learned by the model.

[0029] In one preferred embodiment, step S2 includes an unloading decision model comprising a shared encoding module and a similarity discrimination module; the process of obtaining the cross-layer feature consistency score specifically includes:

[0030] Several layers of shallow feature maps are selected from the backbone network, and several layers of deep semantic feature maps are selected from the feature aggregation network; the selected shallow feature maps are aligned in size and then spliced ​​and aggregated to obtain shallow aggregated features; the selected deep semantic feature maps are aligned in size and then spliced ​​and aggregated to obtain deep aggregated features.

[0031] The shared encoding module maps the shallow aggregated features and deep aggregated features into shallow feature embedding vectors and deep feature embedding vectors of the same dimension.

[0032] The shallow feature embedding vector is concatenated with the deep feature embedding vector to obtain a concatenated vector; the concatenated vector is input to the similarity discrimination module, and the cross-layer feature consistency score is obtained through a multilayer perceptron.

[0033] The above scheme performs similarity analysis on the shallow and deep features of the edge detection model, compares the degree of cross-layer feature consistency between the shallow and deep features, and thus determines whether the current image contains rare targets.

[0034] In one preferred embodiment, in step S1, the formulas for calculating the shallow feature embedding vector and the deep feature embedding vector are as follows:

[0035] ;

[0036] ;

[0037] ;

[0038] ;

[0039] in, This is the shallow feature embedding vector. The deep feature embedding vector, Let be the first parameter matrix. The second parameter matrix, To represent the ReLU activation function, Indicates global average pooling. Indicates the flattening operation. Representation layer normalization, This is a shallow aggregation feature. This is a deep aggregation feature. This indicates a splicing and aggregation operation. This is a shallow, small-sized feature map. This is a size feature map in the shallow layer. This is a shallow, large-size feature map. For deep, small-sized feature maps, This is a size feature map in the deep layer. This is a deep, large-size feature map;

[0040] In step S2, the formula for calculating the cross-layer feature consistency score is as follows:

[0041] ;

[0042] ;

[0043] in, The cross-layer feature consistency score is given. , The third parameter matrix, The fourth parameter matrix, Represents the ReLU activation function. This represents the Sigmoid function. The splicing vector is the vector.

[0044] In one preferred embodiment, the process of obtaining the first parameter matrix, the second parameter matrix, the third parameter matrix, and the fourth parameter matrix is ​​as follows:

[0045] A1. Label the training images according to the target detection classification. If the training images only include common category targets, the sample label is the first category sample; if the training images include rare category targets, the sample label is the second category sample; construct training samples, each training sample including the training image and the corresponding sample label; wherein, the target detection classification includes common categories and rare categories;

[0046] A2. Input the training image into the edge detection model to obtain the shallow aggregation features and deep aggregation features of the training image;

[0047] A3. Input the shallow aggregation features and deep aggregation features of the training image into the shared encoding module, and use the first parameter matrix... and the second parameter matrix After completing linear projection and embedding representation learning, the shallow feature embedding vector and deep feature embedding vector of the training image are obtained;

[0048] A4. After concatenating the shallow feature embedding vector and the deep feature embedding vector of the training image, the third parameter matrix is ​​used... and the fourth parameter matrix Output consistent prediction results;

[0049] A5. Based on the consistency prediction results and the sample labels, the first parameter matrix, the second parameter matrix, the third parameter matrix, and the fourth parameter matrix are iteratively updated using the gradient backpropagation algorithm and the optimizer until convergence, resulting in the optimized first parameter matrix, the second parameter matrix, the third parameter matrix, and the fourth parameter matrix.

[0050] In one preferred embodiment, in step S3, when the unloading decision result is the first identifier, the formula for calculating the final detection result is:

[0051] ;

[0052] When the unloading decision result is the second identifier, the formula for calculating the final detection result is: ;

[0053] in, The final detection result is... The cloud-based detection results are as follows. The image to be detected. This is a cloud-based detection function. The results are for edge detection. For edge prediction boxes, Predict the category for the edge. For the corresponding confidence level, This represents the number of detected targets output by the edge detection model on the image to be detected.

[0054] Based on the same concept, the present invention also provides an edge-cloud collaborative detection system based on cross-layer feature consistency, comprising:

[0055] The edge detection module is used to input the image to be detected into the edge detection model, obtain shallow and deep features, and obtain the edge detection results.

[0056] A cross-layer feature consistency unloading decision module is used to input the shallow features and the deep features into the constructed unloading decision model and obtain the unloading decision result. The unloading decision result is obtained based on whether the cross-layer feature consistency score between the shallow features and the deep features is less than a preset decision threshold. If it is less than the threshold, the unloading decision result is a first identifier; if it is not less than the threshold, the unloading decision result is a second identifier. The first identifier indicates that the image to be detected may contain rare category targets, and the second identifier indicates that the image to be detected only contains common category targets.

[0057] The detection result fusion module is used to send the image to be detected to the cloud detection model to obtain the cloud detection result when the unloading decision result is the first identifier; to fuse the cloud detection result with the edge detection result to obtain the final detection result; and when the unloading decision result is the second identifier, the final detection result is the edge detection result.

[0058] Based on the same concept, the present invention also provides an electronic device, including a memory, a processor, and a computer program / instructions stored in the memory, wherein the processor executes the computer program / instructions to implement the edge-cloud collaborative detection method based on cross-layer feature consistency as described above.

[0059] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention provides an edge-cloud collaborative detection method, system, and device based on cross-layer feature consistency. According to the long-tail distribution characteristics of the target category in real-world scenes, images containing only common categories are processed by a lightweight target detection model at the edge, while images containing rare categories are processed by a large cloud model. The lightweight edge target detection model achieves low-latency detection of common categories. An offloading decision model compares the degree of cross-layer feature consistency between shallow and deep features to determine whether the image exceeds the representational capacity of the edge detection model, achieving semantic-aware offloading decision-making for rare targets. Images containing rare targets are offloaded to a large cloud model for enhanced detection, significantly improving the overall detection accuracy in long-tail scenarios while ensuring system real-time performance, thus achieving edge-cloud collaborative detection for long-tail target detection scenarios. Attached Figure Description

[0060] Figure 1 This is a flowchart of an edge-cloud collaborative detection method based on cross-layer feature consistency according to an embodiment of the present invention;

[0061] Figure 2 This is a cumulative distribution chart of the proportion of category instances in the target detection dataset according to an embodiment of the present invention;

[0062] Figure 3 This is a comparison chart of the detection performance of a representative target detection model of an embodiment of the present invention on common and rare categories;

[0063] Figure 4 This is a framework diagram of an edge-cloud collaborative long-tail target detection system based on cross-layer feature consistency according to an embodiment of the present invention;

[0064] Figure 5 This is a graph showing the performance comparison of common and rare category detection on the VOC dataset for an embodiment of the present invention;

[0065] Figure 6 This is a graph showing the performance comparison of common and rare category detection on the Inspection dataset for an embodiment of the present invention;

[0066] Figure 7 This is a comparison chart of inference latency results for an embodiment of the present invention on the VOC dataset;

[0067] Figure 8 This is a graph showing the comparison of inference latency on the Inspection dataset for an embodiment of the present invention. Detailed Implementation

[0068] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0069] Example 1

[0070] This invention proposes an edge-cloud collaborative long-tail target detection method based on cross-layer feature consistency, which improves the recognition accuracy of rare targets in long-tail target detection scenarios while ensuring low-latency inference capabilities of edge devices. The flowchart of the method is as follows: Figure 1 As shown, the specific steps are as follows:

[0071] S1. Input the image to be detected into the edge detection model, obtain shallow and deep features, and obtain the edge detection results;

[0072] A lightweight object detection model is deployed on an edge device to perform common category detection on input images. A lightweight object detection network is selected as the base model for the edge device, and its backbone network, feature aggregation network, and detection head are used to extract multi-scale features and output the edge detection results. That is, the edge object detection model includes a backbone network, a feature aggregation network, and a detection head. Given an input image... Multi-scale feature maps are extracted through the backbone network, and are represented as follows:

[0073] ;

[0074] in, This represents the feature map of the l-th layer. Indicates the first Layer feature maps. Shallow features mainly reflect low-level visual information such as edges, textures, and local structures, while deep features mainly encode high-level semantic information related to the target category.

[0075] The feature maps are then fused using a feature aggregation network, resulting in the following representation:

[0076] ;

[0077] Finally, the detection head outputs the edge detection results:

[0078] ;

[0079] in, The results are for edge detection. For edge prediction boxes, Predict the category for the edge. For the corresponding confidence level, This represents the number of detected targets output by the edge detection model on the image to be detected. The lightweight edge detection model is mainly responsible for fast detection of high-frequency common categories to meet low latency requirements.

[0080] Before step S1, category distribution information is obtained, specifically including: obtaining the dataset corresponding to the object detection task; statistically analyzing the number of instances of each category in the object detection dataset and their proportion of the total number of instances, analyzing the category frequency distribution, and dividing the target categories into common categories and rare categories based on the category instance proportion distribution. Common categories are target categories with a large number of samples and relatively stable appearance patterns, while rare categories are target categories with a small number of samples and large appearance variations, serving as the basis for task allocation between the edge object detection model and the cloud detection model.

[0081] Specifically, the total number of instances in the object detection dataset is calculated using the following formula: ;in, The total number of instances in the target detection dataset. The total number of categories in the target detection dataset. For the first Number of class target instances;

[0082] The proportion of each category instance in the object detection dataset is obtained by the following formula: ;in, For the first Percentage of class instances;

[0083] Sort the target instances of each category from highest to lowest quantity to obtain the category frequency distribution sequence; obtain the cumulative instance percentage of each category using the following formula: ;in, The first one in the frequency distribution sequence of the category The cumulative percentage of instances in each category ;

[0084] When the cumulative instance percentage When the preset segmentation threshold is reached, Corresponding front One category is classified as the common category, and the remaining categories are classified as the rare category. Common categories usually have the following characteristics: (1) a large number of samples; (2) small differences in appearance within the category; (3) stable feature distribution; (4) lightweight models at the edge are easy to learn and represent. Rare categories usually have the following characteristics: (1) a small number of samples; (2) large variations within the category; (3) significant changes in target scale; (4) easily affected by occlusion, background interference, and changes in lighting; (5) it is difficult for edge models to establish stable semantic representations.

[0085] like Figure 2 The figure shows the cumulative distribution of class instance proportions in the object detection dataset. Statistical analysis of both general and domain-specific object detection datasets reveals a clear long-tail characteristic in the class distribution, meaning a few classes account for the majority of instances, while a large number of classes represent only a small percentage of samples. This phenomenon indicates a natural difference between common and rare classes in object detection tasks, providing a basis for the division of labor in edge-cloud collaborative models.

[0086] In this embodiment of the invention, the general object detection dataset uses the VOC dataset (Pascal Visual Object Classes); the domain-specific object detection dataset uses the Inspection dataset (power line inspection dataset).

[0087] The VOC dataset is one of the classic and widely used benchmark datasets in the field of object detection, primarily used to evaluate the performance of object detection and image understanding algorithms. This dataset contains over 17,000 images, covering 20 common object categories, such as people, vehicles, animals, and everyday objects. The images come from diverse sources, depict complex scenes with varying scales, occlusions, and background interference, effectively reflecting the visual environment of the real world. In terms of data annotation, VOC provides accurate bounding box annotations and category labels for each object instance, making it suitable for various tasks such as object detection, classification, and segmentation. Although the category distribution in VOC is relatively balanced, it still exhibits a certain degree of long-tail characteristics in actual statistics, meaning that a few categories have a large number of samples, while some categories have fewer samples. Due to its high degree of standardization and mature evaluation system (such as the mAP metric), the VOC dataset is widely used to validate the detection capabilities and generalization performance of models in general scenarios.

[0088] The Inspection dataset is a domain-specific target detection dataset built for power line inspection and industrial monitoring scenarios, primarily used to detect safety hazards in power transmission line environments. This dataset contains approximately 2,896 training images and 361 test images, covering 11 target categories related to power line inspection, such as foreign objects, defects, and potentially hazardous objects. Compared to general datasets, this dataset is more scenario-specific and task-oriented. The Inspection dataset has the following significant characteristics: Significant long-tail distribution: a few categories constitute the majority of samples, while many categories have few samples, highlighting class imbalance; High visual complexity: targets are typically small in size, diverse in shape, and easily affected by background interference, occlusion, and changes in lighting; Obvious domain characteristics: the data distribution differs significantly from general datasets, placing higher demands on the model's generalization ability. Therefore, the Inspection dataset is often used to evaluate the detection performance and robustness of models under long-tail distribution, complex scenarios, and domain transfer conditions. Experiments show that on this dataset, lightweight edge models are significantly insufficient in detecting rare categories, relying more on cloud-based detection models for compensation.

[0089] like Figure 3The figure shows a comparison of the detection accuracy of representative object detection models on common and rare categories, using mAP50:95 as the standard evaluation metric. The representative object detection models selected in this embodiment include: YOLO (a single-stage object detection algorithm), Cascade R-CNN (a cascaded region convolutional neural network), and RT-DETR (a real-time detection Transformer model). Evaluation of various detection models on different datasets reveals that lightweight edge detection models perform well on common categories, but their performance drops significantly on rare categories. This indicates that small edge models struggle to effectively represent object categories with scarce samples and large intra-class differences. Therefore, it is necessary to introduce large cloud-based models to compensate for the insufficient representational ability of edge detection models on rare categories.

[0090] S2. Input the shallow features and the deep features into the constructed unloading decision model, and obtain the unloading decision result; the unloading decision result is obtained based on whether the cross-layer feature consistency score of the shallow features and the deep features is less than a preset decision threshold. If it is less than the threshold, the unloading decision result is a first identifier; if it is not less than the threshold, the unloading decision result is a second identifier; wherein, the first identifier indicates that the image to be detected may include rare category targets, and the second identifier indicates that the image to be detected only contains common category targets;

[0091] The unloading decision model includes a shared encoding module and a similarity discrimination module. The process of obtaining the cross-layer feature consistency score specifically includes: selecting several layers of shallow feature maps from the backbone network and several layers of deep semantic feature maps from the feature aggregation network, respectively serving as shallow information representation and deep semantic representation; aligning the selected shallow feature maps by size and then concatenating them to obtain shallow aggregated features; aligning the selected deep semantic feature maps by size and then concatenating them to obtain deep aggregated features; mapping the shallow aggregated features and deep aggregated features to shallow feature embedding vectors and deep feature embedding vectors of the same dimension through the shared encoding module; concatenating the shallow feature embedding vectors and deep feature embedding vectors to obtain a concatenated vector; inputting the concatenated vector to the similarity discrimination module, and obtaining the cross-layer feature consistency score through a multilayer perceptron.

[0092] The formulas for calculating the shallow feature embedding vector and the deep feature embedding vector are as follows:

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] in, This is the shallow feature embedding vector. The deep feature embedding vector, Let be the first parameter matrix. The second parameter matrix, To represent the ReLU activation function, Indicates global average pooling. Indicates the flattening operation. Representation layer normalization, This is a shallow aggregation feature. This is a deep aggregation feature. This indicates a splicing and aggregation operation. This is a shallow, small-sized feature map. This is a size feature map in the shallow layer. This is a shallow, large-size feature map. For deep, small-sized feature maps, This is a size feature map in the deep layer. This is a deep, large-size feature map;

[0098] The formula for calculating the cross-layer feature consistency score is as follows:

[0099] ;

[0100] ;

[0101] in, The cross-layer feature consistency score is given. , The third parameter matrix, The fourth parameter matrix, Represents the ReLU activation function. This represents the Sigmoid function. The splicing vector is the vector.

[0102] The cross-layer feature consistency score measures the degree to which low-level visual information in shallow features is preserved in deep features. A higher score indicates that the image content better matches the common category features learned by the edge model; a lower score indicates that the current image is more likely to contain rare targets that the edge model struggles to fully represent. Shallow features describe the low-level information of all targets in the input image, while deep features describe the semantic representation information learned by the model.

[0103] In this embodiment of the invention, the first parameter matrix The first linear projection parameter matrix is ​​used to map shallow or deep fused features to the embedding space, and the second parameter matrix is ​​used to map these features to the embedding space. The third parameter matrix is ​​the second linear projection parameter matrix within the embedded space. For the splicing vector The linear mapping parameter matrix to the similarity discrimination module, the fourth parameter matrix This is the linear mapping parameter matrix for the output layer of the similarity discrimination module. The specific processes for obtaining the first parameter matrix, the second parameter matrix, the third parameter matrix, and the fourth parameter matrix are as follows:

[0104] A1. Label the training images according to the target detection classification. If the training images only include common category targets, the sample label is the first category sample; if the training images include rare category targets, the sample label is the second category sample; construct training samples, each training sample including the training image and the corresponding sample label; wherein, the target detection classification includes common categories and rare categories;

[0105] A2. Input the training image into the edge detection model to obtain the shallow aggregation features and deep aggregation features of the training image;

[0106] A3. Input the shallow aggregation features and deep aggregation features of the training image into the shared encoding module, and use the first parameter matrix... and the second parameter matrix After completing linear projection and embedding representation learning, the shallow feature embedding vector and deep feature embedding vector of the training image are obtained;

[0107] A4. After concatenating the shallow feature embedding vector and the deep feature embedding vector of the training image, the third parameter matrix is ​​used... and the fourth parameter matrix Output consistent prediction results;

[0108] A5. Construct a loss function based on the consistency prediction result and the sample label, and iteratively update the first parameter matrix, the second parameter matrix, the third parameter matrix and the fourth parameter matrix using the gradient backpropagation algorithm and optimizer until convergence, to obtain the updated first parameter matrix, the second parameter matrix, the third parameter matrix and the fourth parameter matrix.

[0109] After training is complete, the updated parameter matrix is ​​fixed as the model parameters used in the inference phase, which are used to perform feature consistency judgment and unloading decisions on new input images.

[0110] In this embodiment of the invention, the unloading decision is executed as follows: based on the cross-layer feature consistency score With preset decision threshold The comparison result will trigger an uninstallation check. Less than hour, ;when Greater than or equal to hour, ;in, This indicates that the current image may contain rare targets and needs to be unloaded to the cloud for further processing; This indicates that the current image was detected independently by the target detection model at the edge.

[0111] S3. When the unloading decision result is the first identifier, the image to be detected is sent to the cloud detection model to obtain the cloud detection result. The cloud detection result is then fused with the edge detection result to obtain the final detection result. When the unloading decision result is the second identifier, the final detection result is the edge detection result.

[0112] In this embodiment of the invention, the cloud-enhanced detection is specifically as follows: when the edge unloading decision result is... At this time, the edge device uploads the current input image to the cloud; after receiving the image, the cloud uses a high-capacity object detection model deployed on the cloud server to perform object detection across all categories and outputs the cloud detection results:

[0113] ;

[0114] in, Indicates the input image. This indicates the cloud-based detection function. This represents the set of detection results output by the cloud-based detection model, which includes the target bounding box, category label, and corresponding confidence level.

[0115] Compared to the lightweight model at the edge, the cloud-based detection model has a larger parameter scale and stronger feature representation capabilities, enabling it to more fully model target features in low-frequency, rare categories and complex visual scenes. Therefore, The function is primarily used to compensate for the insufficient representational power of edge-based object detection models in long-tail category detection, and is particularly suitable for rare object detection tasks with sparse sample sizes, large intra-class differences, and complex visual features. After cloud-based detection is completed, The function will detect the results. Return to the edge. When the unloading decision result is... ,but The function is not called; the system directly outputs the edge detection results to avoid unnecessary communication overhead and cloud computing latency. Through this method, The function is triggered only on images that are suspected of containing rare targets, i.e., those that are beyond the representation capabilities of the edge detection model. For images that trigger the unloading, the detection is performed by a cloud-based high-capacity target detection model to enhance the recognition capabilities of rare target categories and complex scene targets. This improves the overall detection accuracy in long-tail target detection scenarios while ensuring low system latency.

[0116] like Figure 5 As shown, this figure compares the detection performance of the edge-cloud collaborative detection method provided in this embodiment of the invention on the VOC dataset for common and rare categories. In the figure, CoLTD represents the edge-cloud collaborative long-tail detection model based on cross-layer consistency learning provided in this embodiment of the invention. Figure 6 The figure shows the performance comparison of the edge-cloud collaborative detection method provided in this embodiment of the invention on the Inspection dataset for common categories and rare categories. Experimental results show that the edge-cloud collaborative detection method can significantly improve the detection performance of rare categories while maintaining a relatively stable performance for common categories, thus verifying the effectiveness of the edge-cloud collaborative detection method provided in this embodiment of the invention in long-tail detection scenarios.

[0117] like Figure 7 The figure shows the comparison results of inference latency of the edge-cloud collaborative detection method provided in this embodiment of the invention on the VOC dataset. Figure 8 The figure shows the comparison results of inference latency of the edge-cloud collaborative detection method provided in this embodiment of the invention on the Inspection dataset. Experimental results show that the overall inference latency of the edge-cloud collaborative detection method provided in this embodiment of the invention is close to that of the lightweight model at the edge, and cloud processing is triggered only on a small number of images that are suspected to contain rare targets. Therefore, it can achieve enhanced detection of complex samples while ensuring low latency, demonstrating good edge-cloud collaborative efficiency.

[0118] Example 2

[0119] This embodiment 2 also provides an edge-cloud collaborative detection system based on cross-layer feature consistency, including:

[0120] The edge detection module is used to input the image to be detected into the edge detection model, obtain shallow and deep features, and obtain the edge detection results.

[0121] A cross-layer feature consistency unloading decision module is used to input the shallow features and the deep features into the constructed unloading decision model and obtain the unloading decision result. The unloading decision result is obtained based on whether the cross-layer feature consistency score between the shallow features and the deep features is less than a preset decision threshold. If it is less than the threshold, the unloading decision result is a first identifier; if it is not less than the threshold, the unloading decision result is a second identifier. The first identifier indicates that the image to be detected may contain rare category targets, and the second identifier indicates that the image to be detected only contains common category targets.

[0122] The detection result fusion module is used to send the image to be detected to the cloud detection model to obtain the cloud detection result when the unloading decision result is the first identifier; to fuse the cloud detection result with the edge detection result to obtain the final detection result; and when the unloading decision result is the second identifier, the final detection result is the edge detection result.

[0123] like Figure 4 The diagram shows the framework of the edge-cloud collaborative detection system proposed in this embodiment of the invention, comprising three parts: an edge-side common target detection module, a cross-layer feature consistency offloading decision module, and a detection result fusion module. Given an input image, the lightweight detection model at the edge first performs detection and extracts shallow and deep features; then, the cross-layer feature consistency offloading decision module calculates feature similarity scores to determine whether the current image contains rare targets; if offloading is required, the current image is transmitted to a large cloud model for enhanced detection; finally, the edge-side results are fused with the cloud detection results to generate the final detection result.

[0124] This embodiment 2 also provides an electronic device, which includes: a memory, a processor, and a computer program or instructions stored in the memory. The processor executes the computer program or instructions to implement the edge-cloud collaborative detection method based on cross-layer feature consistency in embodiment 1.

[0125] Although not shown, the electronic device includes a processor that can perform various appropriate operations and processes based on programs and / or data stored in read-only memory (ROM) or loaded from a storage portion into random access memory (RAM). The processor can be a multi-core processor or may contain multiple processors. In some embodiments, the processor may include a general-purpose main processor and one or more specialized coprocessors, such as a central processing unit, graphics processing unit (GPU), neural network processor (NPU), digital signal processor (DSP), etc. Various programs and data required for device operation are also stored in RAM. The processor, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0126] The above embodiments should be understood as being used only to illustrate the present invention more clearly, and not to limit the scope of the present invention. After reading the present invention, any modifications of the present embodiments by those skilled in the art will fall within the scope defined by the appended claims.

Claims

1. A collaborative edge-cloud detection method based on cross-layer feature consistency, characterized in that, include: S1. Input the image to be detected into the edge detection model, obtain shallow and deep features, and obtain the edge detection results; S2. Input the shallow features and the deep features into the constructed unloading decision model, and obtain the unloading decision result; the unloading decision result is obtained based on whether the cross-layer feature consistency score of the shallow features and the deep features is less than a preset decision threshold. If it is less than the threshold, the unloading decision result is a first identifier; if it is not less than the threshold, the unloading decision result is a second identifier; wherein, the first identifier indicates that the image to be detected may include rare category targets, and the second identifier indicates that the image to be detected only contains common category targets; S3. When the unloading decision result is the first identifier, the image to be detected is sent to the cloud detection model to obtain the cloud detection result. The cloud detection result is then fused with the edge detection result to obtain the final detection result. When the unloading decision result is the second identifier, the final detection result is the edge detection result. The edge detection model includes a backbone network, a feature aggregation network, and a detection head; The process of obtaining the edge detection results specifically includes: inputting the image to be detected into the backbone network to obtain backbone feature maps at multiple scales; selecting at least one layer from the backbone feature maps at multiple scales as shallow feature maps, and inputting the backbone feature maps at multiple scales into a feature aggregation network to obtain deep semantic feature maps; inputting the deep semantic feature maps into the detection head to output the edge detection results; The unloading decision model includes a shared encoding module and a similarity discrimination module; The process of obtaining the cross-layer feature consistency score specifically includes: Several layers of shallow feature maps are selected from the backbone network, and several layers of deep semantic feature maps are selected from the feature aggregation network; the selected shallow feature maps are aligned in size and then spliced ​​and aggregated to obtain shallow aggregated features; the selected deep semantic feature maps are aligned in size and then spliced ​​and aggregated to obtain deep aggregated features. The shared encoding module maps the shallow aggregated features and deep aggregated features into shallow feature embedding vectors and deep feature embedding vectors of the same dimension. The shallow feature embedding vector and the deep feature embedding vector are concatenated to obtain a concatenated vector; the concatenated vector is input to the similarity discrimination module, and the cross-layer feature consistency score is obtained through a multilayer perceptron; The formulas for calculating the shallow feature embedding vector and the deep feature embedding vector are as follows: ； ； ； ； in, This is the shallow feature embedding vector. The deep feature embedding vector, Let be the first parameter matrix. The second parameter matrix, To represent the ReLU activation function, Indicates global average pooling. Indicates the flattening operation. Representation layer normalization, This is a shallow aggregation feature. This is a deep aggregation feature. This indicates a splicing and aggregation operation. This is a shallow, small-sized feature map. This is a size feature map in the shallow layer. This is a shallow, large-size feature map. For deep, small-sized feature maps, This is a size feature map of the deep layers. This is a deep, large-size feature map; The formula for calculating the cross-layer feature consistency score is as follows: ； ； in, The cross-layer feature consistency score is given. , The third parameter matrix, The fourth parameter matrix, Represents the ReLU activation function. This represents the Sigmoid function. The splicing vector is the vector.

2. The edge-cloud collaborative detection method based on cross-layer feature consistency according to claim 1, characterized in that, Before step S1, the process of classifying the common categories and the rare categories also includes: The total number of instances in the object detection dataset is calculated using the following formula: ； in, The total number of instances in the target detection dataset. The total number of categories in the target detection dataset. For the first Number of class target instances; The proportion of each category instance in the object detection dataset is obtained by the following formula: ； in, For the first Percentage of class instances; Sort the target instances of each category from highest to lowest quantity to obtain the category frequency distribution sequence; obtain the cumulative instance percentage of each category using the following formula: ； in, The first one in the frequency distribution sequence of the category The cumulative percentage of instances in each category ; When the cumulative instance percentage When the preset segmentation threshold is reached, Corresponding front One category is classified as the common category, and the remaining categories are classified as the rare category.

3. The edge-cloud collaborative detection method based on cross-layer feature consistency according to claim 1, characterized in that, The process of obtaining the first parameter matrix, the second parameter matrix, the third parameter matrix, and the fourth parameter matrix is ​​as follows: A1. Label the training images according to the target detection classification. If the training images only include common category targets, the sample label is the first category sample; if the training images include rare category targets, the sample label is the second category sample; construct training samples, each training sample including the training image and the corresponding sample label; wherein, the target detection classification includes common categories and rare categories; A2. Input the training image into the edge detection model to obtain the shallow aggregation features and deep aggregation features of the training image; A3. Input the shallow and deep aggregation features of the training image into the shared encoding module, and use the first parameter matrix... and the second parameter matrix After completing linear projection and embedding representation learning, the shallow feature embedding vector and deep feature embedding vector of the training image are obtained; A4. After concatenating the shallow feature embedding vector and the deep feature embedding vector of the training image, the third parameter matrix is ​​used... and the fourth parameter matrix Output consistent prediction results; A5. Based on the consistency prediction results and the sample labels, the first parameter matrix, the second parameter matrix, the third parameter matrix, and the fourth parameter matrix are iteratively updated using the gradient backpropagation algorithm and optimizer until convergence, resulting in the updated first parameter matrix, second parameter matrix, third parameter matrix, and fourth parameter matrix.

4. The edge-cloud collaborative detection method based on cross-layer feature consistency according to claim 1, characterized in that, In step S3, when the unloading decision result is the first identifier, the formula for calculating the final detection result is: ； When the unloading decision result is the second identifier, the formula for calculating the final detection result is: ; in, The final detection result is... The cloud-based detection results are as follows. The image to be detected. This is a cloud-based detection function. The results are for edge detection. For edge prediction boxes, Predict the category for the edge. For the corresponding confidence level, This represents the number of detected targets output by the edge detection model on the image to be detected.

5. An edge-cloud collaborative detection system based on cross-layer feature consistency, used to execute the edge-cloud collaborative detection method based on cross-layer feature consistency as described in claim 1, characterized in that, include: The edge detection module is used to input the image to be detected into the edge detection model, obtain shallow and deep features, and obtain the edge detection results. A cross-layer feature consistency unloading decision module is used to input the shallow features and the deep features into the constructed unloading decision model and obtain the unloading decision result. The unloading decision result is obtained based on whether the cross-layer feature consistency score between the shallow features and the deep features is less than a preset decision threshold. If it is less than the threshold, the unloading decision result is a first identifier; if it is not less than the threshold, the unloading decision result is a second identifier. The first identifier indicates that the image to be detected may contain rare category targets, and the second identifier indicates that the image to be detected only contains common category targets. The detection result fusion module is used to send the image to be detected to the cloud detection model to obtain the cloud detection result when the unloading decision result is the first identifier, and then fuse the cloud detection result with the edge detection result to obtain the final detection result; when the unloading decision result is the second identifier, the final detection result is the edge detection result.

6. An electronic device comprising a memory, a processor, and a computer program / instructions stored in the memory, characterized in that, The processor executes the computer program / instructions to implement the edge-cloud collaborative detection method based on cross-layer feature consistency as described in any one of claims 1 to 4.

Images