A method and system for detecting generic objects based on dynamic inference networks
By constructing a multi-exit dynamic inference object detector and a multi-scale adaptive gating network, and combining hyperparameter-free training and variable latency strategies, the deployment flexibility problem of a general object detector in various hardware environments is solved, detection accuracy is improved and detection latency is reduced, and model deployment is simplified.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PEKING UNIV
- Filing Date
- 2023-03-30
- Publication Date
- 2026-05-01
AI Technical Summary
Existing general-purpose object detectors suffer from limitations in deployment flexibility, making it difficult to quickly deploy and apply the same model effectively under various hardware requirements. In particular, it is difficult to achieve a good balance between accuracy and speed in different hardware environments.
We construct a dynamic inference object detector with multiple exits and supplement it with a multi-scale adaptive gating network. By combining a hyperparameter-free training strategy and a variable-latency inference strategy, we can enable an object detector to adapt to different network complexity requirements in various deployment environments.
This invention enables an object detector to meet the requirements of different network complexities in various hardware environments, improves detection accuracy and reduces detection latency, simplifies the model deployment process, and reduces the consumption of manpower and material resources.
Smart Images
Figure CN116402139B_ABST
Abstract
Description
A method and system for detecting general objects based on dynamic inference networks Technical Field
[0001] This invention belongs to the field of object detection technology, and relates to computer vision and deep learning technology, and in particular to a method and system for detecting general objects based on dynamic inference networks. Background Technology
[0002] General object detection is one of the most important tasks in the field of computer vision, and it has been widely applied in areas such as intelligent security, autonomous driving, and drone surveying. In recent years, based on the rapid development of deep learning networks, general object detection has made great progress, and more and more accurate and faster detectors have been proposed, such as several detectors based on Network Architecture Search (NAS) algorithms (DetNAS, MAE-DET, etc.), YOLO series detectors (YOLOv4, YOLOv5, YOLOv6, YOLOv7, etc.), and so on.
[0003] However, because these detectors often have definite complexity and inference latency, when faced with various hardware requirements, it is necessary to repeatedly design and train many general-purpose object detectors with different complexities and latency to achieve a good trade-off between accuracy and speed. This is not flexible enough for practical deployment and application. Therefore, existing general-purpose object detectors still have certain bottlenecks in deployment flexibility and cannot quickly achieve effective deployment and application of the same model on various hardware requirements. Summary of the Invention
[0004] To overcome the shortcomings of the prior art, this invention provides a method and system for detecting general objects based on a dynamic inference network. Specifically, this invention first constructs a multi-exit dynamic inference object detector based on a given general object detector, and then supplements it with a multi-scale adaptive gating network to control the inference path selection of this dynamic inference object detector. Simultaneously, a hyperparameter-free training strategy and a variable inference latency strategy are proposed for this dynamic inference network. In summary, the method for detecting general objects based on a dynamic inference network provided by this invention, by utilizing a dynamic inference network, enables the use of only one object detector to meet the requirements of different network complexities (such as inference latency) in various deployment environments.
[0005] The technical solution of this invention is:
[0006] A method for detecting general objects based on dynamic inference networks includes the following steps:
[0007] Based on a given general object detector, construct a dynamic reasoning object detector with multiple exits;
[0008] A multi-scale adaptive gating network is inserted into the dynamic inference object detector with multiple exits;
[0009] The multi-exit dynamic reasoning object detector and the multi-scale adaptive gating network are trained using a training strategy without hyperparameters.
[0010] Using the trained multi-exit dynamic inference object detector and the multi-scale adaptive gating network, a variable-delay inference strategy is employed for general object detection.
[0011] Furthermore, the construction of a multi-exit dynamic reasoning object detector based on a given general object detector includes:
[0012] For a given general object detector, first extract its backbone network (which can be any network such as ResNet, ResNeXt, CSPNet, etc.), and then stack two of these backbone networks in a cascaded manner.
[0013] Then, the corresponding neck network and detector head are connected to the two backbone networks respectively. These two backbone networks naturally generate two exits. The image inference route of the first exit is "backbone network 1 → neck network 1 → detector head 1", and the image inference route of the second exit is "backbone network 1 → backbone network 2 → neck network 2 → detector head 2".
[0014] This cascaded network, together with the subsequent neck network and detector head, is called a multi-exit dynamic reasoning object detector.
[0015] Furthermore, in the cascading stacking of the two backbone networks, considering that each backbone network contains multiple (typically four) convolutional blocks (stages), this invention introduces a feature reuse connection to fully utilize and enhance its feature information. In the feature reuse connection, the output of each stage of the first backbone network is used as input to the same and lower-level stages of the next backbone network. The structure of this connection includes a 1×1 convolutional layer and an upsampling operation; the 1×1 convolutional layer aligns the number of channels of the output feature of a certain stage of the first backbone network with the number of channels of the input feature of the corresponding stage of the next backbone network, and the upsampling operation aligns the spatial size of the features of the two stages of the cascaded backbone networks.
[0016] Furthermore, the input to the multi-scale adaptive gating network is the multi-scale feature of the first backbone network in the cascaded backbone network, referred to as F1:
[0017]
[0018] Where L represents the number of convolutional blocks (stages) contained in the backbone network, which is 4 in most detector models.
[0019] To reduce the latency of subsequent operations for F1, global pooling and channel-dimension concatenation are first used to reduce the feature dimensionality, resulting in compressed multi-scale features. as follows:
[0020]
[0021] Where C represents the channel dimension concatenation operation and C represents the global pooling operation.
[0022] Then, two linear layers are used to process the above multi-scale features. Mapped to 1 dimension (i.e., the score φ) to reflect the difficulty level of the input image. As shown in the following formula:
[0023]
[0024] Where δ represents the ReLU activation function, σ represents the Sigmoid activation function, and W1, W2, b1, b2 are the learnable parameters of the linear layer.
[0025] Generally speaking, it is expected that simpler images (with smaller scores φ) go through the first exit (i.e., are processed by only one backbone network), while more difficult images (with larger scores φ) go through the second exit (i.e., are processed by two backbone networks).
[0026] Furthermore, the above method utilizes a hyperparameter-free training strategy to train the aforementioned multi-exit dynamic inference object detector and multi-scale adaptive gating network. "Hyperparameter-free" means that the training objective of this method does not involve adjusting hyperparameters for different models. Adopting a hyperparameter-free training strategy avoids the process of human experts manually adjusting hyperparameters using a "trial and error" approach for different models. The training process includes the following steps:
[0027] First, train a dynamic reasoning object detector with multiple exits. The overall training objective is:
[0028]
[0029] Where x and y represent the input data and the corresponding label, respectively, and Θ1 and Θ2 represent the parameters of the first backbone network and its corresponding neck network and detector head, and the parameters of the second backbone network and its corresponding neck network and detector head, respectively. These represent the loss functions of the detector heads corresponding to the first and second backbone networks, respectively, and typically include bounding box loss and classification loss.
[0030] After training the dynamic reasoning object detector with multiple exits, two sets of trained detectors were obtained (one is the backbone network 1, neck network 1, and detector head 1, and the other is the backbone network 2, neck network 2, and detector head 2). Therefore, the trained parameters Θ1 and Θ2 were directly frozen, that is, their values were not updated in the subsequent training process.
[0031] Next, a multi-scale adaptive gating network is trained, with the overall training objective being:
[0032]
[0033] Where Θ R denoted by , where Δ represents the parameters of the multi-scale adaptive gating network, and Δ represents the median difference in loss between the two trained detector heads on the training set.
[0034] Furthermore, in the process of general object detection, the above method utilizes a variable-delay inference strategy to test the dynamic inference object detector and perform object detection, specifically including the following steps:
[0035] First, score all images on the validation set, let's say the score is S. val .
[0036] Next, the inference delay of the dynamic inference detector at the first and second exits were tested, assuming they are lat1 and lat2 respectively.
[0037] Based on the given target delay lat t A scoring threshold τ can be obtained. val This is used to ensure that the dynamic inference detector can provide the inference speed of the target latency on the corresponding dataset. The specific formula is as follows:
[0038]
[0039] Where percentile(·,k) represents the kth quantile of the data.
[0040] When inferring on the test set (or detecting general objects in a real-world scenario within a specific domain), because the validation set and test set are independent and identically distributed, the threshold obtained from the validation set can be directly applied to the test set (or the real-world scenario) for direct testing. Specifically, if the score φ is less than the threshold, it is considered a simple sample and is allowed to pass through the first exit (i.e., "backbone network 1 → neck network 1 → detector head 1"); conversely, if the score φ is greater than the threshold, it is considered a difficult sample and is allowed to pass through the second exit (i.e., "backbone network 1 → backbone network 2 → neck network 2 → detector head 2").
[0041] A system for detecting general objects based on a dynamic inference network, comprising:
[0042] The detector building module is used to build a multi-exit dynamic reasoning object detector based on a given general object detector;
[0043] A gating network construction module is used to insert a multi-scale adaptive gating network into the multi-exit dynamic inference object detector;
[0044] The training module is used to train the multi-exit dynamic reasoning object detector and the multi-scale adaptive gating network using a training strategy without hyperparameters.
[0045] The detection module is used to perform general object detection using the trained multi-exit dynamic inference object detector and the multi-scale adaptive gating network, employing a variable-delay inference strategy.
[0046] The aforementioned method for detecting general objects based on dynamic inference networks can be widely applied to the deployment of various general object detectors, and further applied to various application scenarios such as intelligent security, autonomous driving, and drone surveying, so as to quickly realize the effective deployment and application of the same model under various hardware requirements and reduce the consumption of manpower, material resources, and financial resources.
[0047] Compared with the prior art, the beneficial effects of the present invention are:
[0048] This invention first constructs a multi-exit dynamic inference object detector based on a given general object detector, and then supplements it with a multi-scale adaptive gating network to control the inference route selection of the dynamic inference object detector. At the same time, it proposes a hyperparameter-free training strategy and a variable inference latency strategy for the dynamic inference network, thus efficiently realizing an object detector to meet the needs of different network complexities (such as inference latency) in various deployment environments.
[0049] This invention breaks away from the conventional approach where a single object detector model supports only one deployment requirement (such as inference latency). By introducing a dynamic inference network, it enables a single object detector to meet varying network complexity requirements across different deployment environments. This method is simple and effective, applicable to various object detectors (such as Faster R-CNN and YOLOv7), requires minimal changes to the overall detector architecture, and is convenient and efficient. Furthermore, while supporting a range of inference latencies, the detection accuracy achieved surpasses that of methods that directly scale (e.g., widen or deepen) and train from scratch.
[0050] The method of this invention has achieved leading detection results on the MS COCO object detection dataset by combining multiple detector models (including YOLOv7 series models, Faster R-CNN ResNet series models, Mask R-CNN Swin Transformer series models, etc.). Specifically, by combining the YOLOv7-W6 model, the dynamic inference detector can achieve an accuracy of 56.8% mAP at 46 FPS, which is 39% faster than the baseline YOLOv7-E6E at the same accuracy, and 12% faster than YOLOv7-D6 with an accuracy improvement of 0.2% mAP. By combining Faster R-CNN ResNet50, the dynamic inference detector can achieve an accuracy of 40.4% mAP at 17 FPS, which is 1.0% mAP faster than the baseline Faster R-CNN ResNet101. By combining Mask R-CNN Swin Transformer-Tiny, the dynamic inference detector can achieve an accuracy of 49.9% mAP at 11 FPS, which is 1.7% mAP faster than the baseline Mask R-CNN Swin Transformer-Small. Attached Figure Description
[0051] Figure 1 is a flowchart of the present invention.
[0052] Figure 2 is a diagram of the multi-exit dynamic reasoning object detector and multi-scale adaptive gating network architecture of the present invention. Detailed Implementation
[0053] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the scope of the invention is not limited in any way.
[0054] This invention proposes a method for detecting general objects based on dynamic inference networks, the overall process of which is shown in Figure 1. The following section uses the construction of a dynamic inference detector based on YOLOv7 as an example for description.
[0055] 1) Construct a dynamic reasoning object detector with multiple exits (as shown in Figure 2).
[0056] For the YOLOv7 object detector, its backbone network is first extracted, which is generally considered to be a CSPDarkNet architecture. Then, two of these backbone networks are stacked in a cascaded manner.
[0057] In cascading and stacking, considering that each backbone network CSPDarkNet contains 5 convolutional blocks (stages), a feature reuse connection is introduced to fully utilize and enhance its feature information. In the feature reuse connection, the output of each stage of the first backbone network is used as input to the same and lower-level stages of the next backbone network. The structure of this connection includes a 1×1 convolutional layer and an upsampling operation; the 1×1 convolutional layer aligns the number of channels of the output feature of a certain stage of the first backbone network with the number of channels of the input feature of the corresponding stage of the next backbone network, and the upsampling operation aligns the spatial size of the features of these two stages of the cascaded backbone network.
[0058] Then, corresponding neck networks and detector heads are connected to the two backbone networks, corresponding to the neck network and detector head of the original YOLOv7 model, respectively. These two backbone networks naturally generate two exits. The image inference route of the first exit is "backbone network 1 → neck network 1 → detector head 1", and the image inference route of the second exit is "backbone network 1 → backbone network 2 → neck network 2 → detector head 2".
[0059] This cascaded network, together with the subsequent neck network and detector head, is called a multi-exit dynamic reasoning object detector.
[0060] 2) Insert a multi-scale adaptive gating network into the dynamic reasoning object detector with multiple exits (as shown in Figure 2).
[0061] The input to this multi-scale adaptive gating network is the multi-scale features of the first backbone network, CSPDarkNet, in the cascaded backbone network, which is called F1:
[0062]
[0063] Where L represents the number of convolutional blocks (stages) contained in the backbone network, which is 5 for CSPDarkNet.
[0064] For F1, to reduce the latency of subsequent operations, global pooling and channel-dimensional concatenation are first used to reduce the feature dimensionality, resulting in compressed multi-scale features. as follows:
[0065]
[0066] Where C represents the channel-level concatenation operation, and C represents the global pooling operation. The final result is... It is a vector of length 2752.
[0067] Then, two linear layers are used to process the above multi-scale features. Mapped to 1 dimension (i.e., score φ) to reflect the difficulty rating of the input image. As shown in the following formula:
[0068]
[0069] Where δ represents the ReLU activation function, σ represents the Sigmoid activation function, and W1, W2, b1, b2 are the learnable parameters of the linear layer.
[0070] 3) The above-mentioned multi-exit dynamic reasoning object detector and multi-scale adaptive gating network are trained using a hyperparameter-free training strategy.
[0071] First, train a dynamic reasoning object detector with multiple exits. The overall training objective is:
[0072]
[0073] Where x and y represent the input data and the corresponding label, respectively, and Θ1 and Θ2 represent the parameters of the first backbone network and its corresponding neck network and detector head, and the parameters of the second backbone network and its corresponding neck network and detector head, respectively. These represent the loss functions of the detector heads corresponding to the first and second backbone networks, respectively, which are the original training loss functions of the YOLOv7 model.
[0074] After training the dynamic reasoning object detector with multiple exits, two trained detectors were obtained (one is the backbone network 1, neck network 1, and detector head 1, and the other is the backbone network 2, neck network 2, and detector head 2). Therefore, their trained parameters Θ1 and Θ2 were directly frozen, meaning that their values were not updated during subsequent training.
[0075] Next, a multi-scale adaptive gating network is trained, with the overall training objective being:
[0076]
[0077] Where Θ R denoted by , where Δ represents the parameters of the multi-scale adaptive gating network, and Δ represents the median difference in loss between the two trained detector heads on the training set.
[0078] 4) Test the above dynamic reasoning object detector using a variable time delay reasoning strategy.
[0079] First, score all images on the validation set, let's say the score is S. val .
[0080] Next, the inference delay of the dynamic inference detector at the first and second exits were tested, assuming they are lat1 and lat2 respectively.
[0081] Based on the given target delay lat t A scoring threshold τ can be obtained. val This is used to ensure that the dynamic inference detector can provide the inference speed of the target latency on the corresponding dataset. The specific formula is as follows:
[0082]
[0083] Where percentile(·,k) represents the kth quantile of the data.
[0084] During inference on the test set, since the validation set and test set are independent and identically distributed, the threshold obtained from the validation set can be directly applied to the test set for testing. Specifically, if the score φ is less than the threshold, it is considered a simple sample and is allowed to pass through the first exit (i.e., "backbone network 1 → neck network 1 → detector head 1"); conversely, if the score φ is greater than the threshold, it is considered a difficult sample and is allowed to pass through the second exit (i.e., "backbone network 1 → backbone network 2 → neck network 2 → detector head 2").
[0085] Another embodiment of the present invention provides a system for detecting general objects based on a dynamic inference network, comprising:
[0086] The detector building module is used to build a multi-exit dynamic reasoning object detector based on a given general object detector;
[0087] A gating network construction module is used to insert a multi-scale adaptive gating network into the multi-exit dynamic inference object detector;
[0088] The training module is used to train the multi-exit dynamic reasoning object detector and the multi-scale adaptive gating network using a training strategy without hyperparameters.
[0089] The detection module is used to perform general object detection using the trained multi-exit dynamic inference object detector and the multi-scale adaptive gating network, employing a variable-delay inference strategy.
[0090] For the specific implementation process of each module, please refer to the description of the method of the present invention above.
[0091] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.
[0092] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.
[0093] The method for detecting general objects based on dynamic inference networks described above in this invention can be widely applied to the deployment of various general object detectors. It automatically classifies detection samples into "simple" and "difficult" categories for specific application scenarios and automatically selects exit points with different inference complexities for different samples, thereby significantly reducing the detection latency of "simple" samples while ensuring detection accuracy. The downstream applications of this invention are extensive, including personnel detection in intelligent security, vehicle detection in autonomous driving, and object detection in UAV surveying. For intelligent security scenarios, simple samples refer to indoor scenarios with low pedestrian density and sufficient lighting, while complex samples refer to outdoor scenarios with high pedestrian density and significant impact from rain and fog. For autonomous driving scenarios, simple samples refer to scenarios with simple environments, few vehicles and people, and good lighting, while complex samples refer to scenarios with complex environments, many vehicles and people, severe obstruction, and rain or fog. For UAV surveying scenarios, simple samples refer to scenarios with large, clearly defined objects and good lighting, while complex samples refer to scenarios with small objects and rain or fog.
[0094] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.
Claims
1. A method for detecting general objects based on dynamic inference networks, characterized in that, Includes the following steps: Based on a given general object detector, construct a dynamic reasoning object detector with multiple exits; A multi-scale adaptive gating network is inserted into the dynamic inference object detector with multiple exits; The multi-exit dynamic inference object detector and the multi-scale adaptive gating network are trained using a hyperparameter-free training strategy. A variable-delay inference strategy is then employed to perform general object detection using the trained multi-exit dynamic inference object detector and the multi-scale adaptive gating network. The construction of the multi-exit dynamic inference object detector based on a given general object detector includes: firstly, extracting the backbone network of the given general object detector and stacking two backbone networks in a cascaded manner; connecting corresponding neck networks and detector heads to the two backbone networks respectively, with the two backbone networks generating two exits. The image inference path for the first exit is "backbone network 1 → neck network 1 → detector head 1", and the image inference path for the second exit is "backbone network 1 → backbone network 2 → neck network 2 → detector head 2". The cascaded backbone network and the subsequent neck network and detector head are referred to as the multi-exit dynamic inference object detector. The variable-delay inference strategy includes: calculating the score of all images on the validation set, assuming it is... Test the inference delay of the dynamic inference detector at the first and second exits respectively, assuming they are respectively Based on the given target delay ,according to 、 、 Find a scoring threshold When detecting general objects, if the score Less than the threshold If it is considered a simple sample, it is allowed to pass through the first exit; conversely, if the score is... Greater than the threshold They considered it a difficult sample and allowed it to pass through the second exit.
2. The method according to claim 1, characterized in that, The two backbone networks are stacked in a cascaded manner, and feature reuse connections are introduced during the cascaded stacking. In the feature reuse connections, the output of each stage of the first backbone network is used as input to the same and lower stages of the second backbone network. The structure of the feature reuse connection includes a 1×1 convolutional layer and an upsampling operation. The 1×1 convolutional layer aligns the number of channels of the output feature of a certain stage of the first backbone network with the number of channels of the input feature of the corresponding stage of the second backbone network. The upsampling operation aligns the spatial size of the features of the two stages of the cascaded backbone network.
3. The method according to claim 1, characterized in that, The input to the multi-scale adaptive gating network is the multi-scale feature of the first backbone network in the cascaded backbone network, which is referred to as... ;against The feature dimensionality is reduced by using global pooling and channel-dimensional concatenation. Obtain compressed multi-scale features Then, two linear layers are used to combine multi-scale features. Mapping to 1 dimension yields a score. The score, reflecting the difficulty of the input image, is calculated as follows: in, Represents the ReLU activation function. This represents the Sigmoid activation function. These are the learnable parameters for the linear layer.
4. The method according to claim 3, characterized in that, The hyperparameter-free training strategy includes: First, training a multi-exit dynamic reasoning object detector, with the overall training objective being: in These represent the input data and the corresponding labels, respectively. These represent the parameters of the first backbone network and its corresponding neck network in relation to the detector head, and the parameters of the second backbone network and its corresponding neck network in relation to the detector head, respectively. These represent the loss functions of the detector heads corresponding to the first and second backbone networks, respectively; after the multi-exit dynamic inference object detector is trained, the trained parameters are directly passed to the network. Freezing means not updating its values during subsequent training; then, the multi-scale adaptive gating network is trained, with the overall training objective being: in The parameters represent the multi-scale adaptive gating network. This represents the median difference in loss between the two trained detector heads on the training set.
5. The method according to claim 1, characterized in that, The scoring threshold The calculation formula is as follows: in, The first element representing the calculated data Quantiles.
6. A system for detecting general objects based on a dynamic inference network using the method described in any one of claims 1 to 5, characterized in that, include: The detector building module is used to build a multi-exit dynamic reasoning object detector based on a given general object detector; A gating network construction module is used to insert a multi-scale adaptive gating network into the multi-exit dynamic inference object detector; The training module is used to train the multi-exit dynamic reasoning object detector and the multi-scale adaptive gating network using a training strategy without hyperparameters. The detection module is used to perform general object detection using the trained multi-exit dynamic inference object detector and the multi-scale adaptive gating network, employing a variable-delay inference strategy.
7. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 5.