Adaptive object detection

Abstract

Embodiments of this disclosure provide a solution for object detection. In this solution, object distribution information and a performance metric are obtained. The object distribution information indicates the size distribution of detected objects in a set of historical images captured by a camera. The performance metric indicates the corresponding performance level of a set of predetermined object detection models. At least one detection plan is generated based on the object distribution information and the performance metric. This at least one detection plan indicates which predetermined object detection model from the set should be applied to each of at least one sub-image in a target image captured by a camera. Additionally, at least one detection plan is provided for object detection in the target image. In this way, the balance between detection latency and detection accuracy can be improved.

Description

Background Technology

[0001] Object detection is a fundamental building block of video processing and analysis applications. With the development of artificial intelligence technology, neural network (NN)-based object detection models have shown good accuracy in object detection. Furthermore, high-resolution cameras are now widely used to capture images with high quality. To utilize smoother and more detailed images, object detection networks must be designed with much larger capacities (e.g., more convolutional layers, higher dimensions, etc.) to work with high-resolution input. However, when object detection models with more complex structures are deployed on resource-constrained devices such as edge devices, this results in high latency. Therefore, it is desirable to improve the efficiency of object detection while maximizing accuracy, especially on high-resolution images. Summary of the Invention

[0002] According to the embodiments of the subject matter described herein, a solution for adaptive object detection is provided. In this solution, object distribution information and a performance metric are obtained. The object distribution information indicates the size distribution of detected objects in a set of historical images captured by a camera. The performance metric indicates the corresponding performance level of a set of predetermined object detection models. At least one detection plan is generated based on the object distribution information and the performance metric. This at least one detection plan indicates which predetermined object detection model from the set should be applied to each of at least one sub-image in a target image captured by a camera. Additionally, at least one detection plan is provided for object detection in the target image.

[0003] In this way, a detection plan for object detection can be adaptively generated based on the historical distribution of detected objects and the characteristics of a predetermined set of object detection models (e.g., neural network-based models). Thus, corresponding detection models can be adaptively assigned to different regions of the image. For example, a less complex object detection model can be assigned to regions with a lower probability of containing objects. In this way, the balance between detection latency and detection accuracy can be improved.

[0004] This summary is provided to introduce a selection of concepts in a simplified form, which will be further described in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0005] Figure 1 The illustrations show example environments in which various implementations of the themes described herein can be carried out;

[0006] Figure 2An example structure of a generation module in a generation device according to an embodiment of the subject matter described herein is illustrated;

[0007] Figures 3A-3C The illustration shows an example process for updating the detection plan described in this article;

[0008] Figure 4 An example structure of a detection device according to an embodiment of the subject matter described herein is illustrated;

[0009] Figure 5 A flowchart illustrating a process for generating a detection plan according to an embodiment of the subject matter described herein is shown.

[0010] Figure 6 A flowchart illustrating a process for object detection according to an embodiment of the subject matter described herein; and

[0011] Figure 7 A block diagram of a computing device in which various embodiments of the subject matter described herein may be implemented is illustrated.

[0012] Throughout the accompanying drawings, the same or similar reference numerals refer to the same or similar elements. Detailed Implementation

[0013] The subject matter described herein will now be discussed with reference to several exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and thus implement the subject matter described herein, and not to imply any limitation on the scope of the subject matter.

[0014] As used herein, the term "comprising" and its variations shall be interpreted as open-ended terms meaning "including but not limited to". The term "based on" shall be understood as "at least partially based on". The terms "one embodiment" and "an embodiment" shall be interpreted as "at least one embodiment". The term "another embodiment" shall be interpreted as "at least one other embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit or implicit definitions may be included below.

[0015] Object detection now plays a crucial role in numerous video analytics applications such as pedestrian tracking, autonomous driving, traffic monitoring, and / or similar applications. In many video analytics scenarios, video feeds from cameras are analyzed on pre-deployed edge devices to accommodate limited network bandwidth and privacy requirements. However, edge computing is equipped with limited computational resources, posing a significant challenge to running object detection neural networks (NNs) for real-time video analytics.

[0016] On the other hand, an increasing number of high-resolution cameras are being used to capture images or videos with higher quality. To take advantage of smoother and more detailed images, object detection networks must be designed with much larger capacities (e.g., more convolutional layers, higher dimensions, etc.) to work with high-resolution input. However, this results in very high latency, especially when running on resource-constrained edge devices. The implementation of the subject matter described herein provides a solution for adaptive object detection.

[0017] According to the embodiments of the subject matter described herein, a solution for adaptive object detection is provided. In this solution, object distribution information and a performance metric are obtained. The object distribution information indicates the size distribution of detected objects in a set of historical images captured by a camera. The performance metric indicates the corresponding performance level of a set of predetermined object detection models. At least one detection plan is generated based on the object distribution information and the performance metric. This at least one detection plan indicates which predetermined object detection model from the set should be applied to each of at least one sub-image in a target image captured by a camera. Additionally, at least one detection plan is provided for object detection in the target image.

[0018] In this way, a detection plan for object detection can be adaptively generated based on the historical distribution of detected objects and the characteristics of a predetermined set of object detection models (e.g., neural network-based models). This allows different regions of the image to be assigned to corresponding detection models. For example, a less complex object detection model can be assigned to a region with a lower probability of containing an object. Therefore, the balance between detection accuracy and detection latency can be improved.

[0019] The basic principles and several exemplary implementations of the subject matter described herein are explained below with reference to the accompanying drawings.

[0020] Example Environment

[0021] Figure 1 The illustration shows a block diagram of an environment 100 in which various implementations of the subject matter described herein may be carried out. It should be understood that... Figure 1 The environment 100 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein.

[0022] like Figure 1 As shown, environment 100 includes generation device 140. Generation device 140 can be configured to obtain object distribution information 125 associated with a set of historical images 120 captured by camera 110.

[0023] In some implementations, the object distribution information 125 may indicate the size distribution of objects detected in the set of historical images 120. For example, the object distribution information 125 may indicate the location and size of each detected object in the set of historical images 120.

[0024] In some implementations, object distribution information 125 can be generated by any suitable computing device based on object detection results from the set of historical images 120. For example, generating device 140 can receive object detection results and generate object distribution information 125 accordingly.

[0025] Alternatively, another computing device, different from the generating device 140, may generate object distribution information 125 and then send the information to the generating device 140 via wired or wireless communication.

[0026] Reference Figure 2 Describe in detail the process of generating object distribution information 125.

[0027] In addition, such as Figure 1 As shown, the generation device 140 may additionally obtain a set of performance metrics 130 for a predetermined set of object detection models 180. In some embodiments, performance metrics 130 may indicate the performance level of each predetermined object detection model in the set of predetermined object detection models 180. For example, performance metrics 130 may include latency performance metrics and / or accuracy performance metrics, as will be described in detail below.

[0028] In some implementations, the object detection model 180 may include any suitable type of machine learning-based model 180, such as a neural network-based object detection model (e.g., EfficientDet, RetinaNet, Faster-RCNN, YOLO, SSD-MobileNet, etc.). As used herein, the term "neural network" can process inputs and provide corresponding outputs, and it typically includes an input layer, an output layer, and one or more hidden layers between the input and output layers. The layers of a neural network model are sequentially connected such that the output of the previous layer is provided as the input of the next layer, where the input layer receives the inputs of the neural network, and the output of the output layer serves as the final output of the neural network. Each layer of the neural network includes one or more nodes (also called processing nodes or neurons), and each node processes the inputs from the previous layer. In this document, the terms "neural network," "model," "network," and "neural network model" are used interchangeably.

[0029] For reference Figure 2As described in detail, the generating device 140 can generate at least one detection plan 150 based on the obtained object distribution information 125 and performance metrics 130. The at least one detection plan 150 can indicate which predetermined object detection model from the set of predetermined object detection models 180 should be applied to each of at least one sub-image in the target image captured by the camera 110.

[0030] In some implementations, at least one sub-image can be determined according to a predetermined partitioning pattern. The partitioning pattern can indicate how the image view of camera 110 is divided into multiple regions. In one example, the partitioning plan can indicate that the image view is uniformly divided into multiple regions based on a predetermined region size.

[0031] In another example, segmentation patterns can be automatically generated based on characteristics of the environment associated with camera 160. For instance, camera 110 can be fixedly deployed in a parking lot, and a pedestrian detection model can be used to detect pedestrians in images or videos captured by the camera. Since pedestrians can only appear in certain specific areas of the image view, segmentation patterns can be generated based on semantic analysis of elements included in the environment. For example, the segmentation pattern could indicate dividing the image view into different regions, each with a different probability of detecting pedestrians.

[0032] In some implementations, as will be described in detail below, the segmentation pattern can also be adaptively generated based on object distribution information 125 and / or performance metric 130. Additionally, the generated segmentation pattern can also be indicated by at least one generated detection plan 150. For example, at least one detection plan 150 may include the coordinates of the vertices of each region segmented from the image view according to the segmentation pattern.

[0033] like Figure 1 As shown, at least one detection plan 150 generated can be additionally provided to detection device 170 for object detection. Detection device 170 can be coupled to camera 110 to receive target image 160 captured by camera 110 for object detection.

[0034] In some implementations, if the received at least one detection plan 150 includes multiple detection plans, the detection device 170 may first select a target detection plan from the received multiple detection plans 150. For example, the detection device 170 may select a target detection plan based on the expected delay time for object detection on the target image 160.

[0035] Given a target detection plan, the detection device 170 can divide the target image 160 into at least one sub-image according to a segmentation pattern. As discussed, in some embodiments, the segmentation pattern can be predetermined. Alternatively, the segmentation pattern can be indicated by at least one received detection plan 150.

[0036] Additionally, the detection device 170 can select one or more object detection models as indicated by the selected target detection plan. In some embodiments, the set of object detection models 180 may already be deployed on the detection device 170, and the detection device 170 can identify one or more object detection models to be utilized from the set of pre-deployed object detection models 180.

[0037] In some alternative implementations, the detection device 170 may also deploy only one or more object detection models as indicated by the object detection plan. For example, the entire set of predetermined object detection models may include ten models, and the object detection plan may indicate that only two of the ten models will be used for object detection on the target image 160. In this case, the detection device 170 may deploy only the two object detection models indicated by the object detection plan.

[0038] like Figure 1 As shown, the detection device 170 can use an object detection model to detect objects in (multiple) corresponding sub-images generated according to the segmentation pattern, and further determine the objects in the target image 160 based on the objects detected in (multiple) sub-images to obtain the final object detection result 190.

[0039] In some implementations, camera 110 may include a high-resolution camera for capturing high-resolution images / videos. Furthermore, to achieve a stable object distribution, camera 110 may remain stationary at a fixed angle and position for a predetermined time period.

[0040] In some implementations, the generation device 140 and the detection device 170 may be implemented in separate computing devices. The generation device 140 may, for example, include a computing device with high computing power, while the detection device 170 may, for example, include a computing device with low computing power. For instance, the generation device 140 may include a cloud-based server, and the detection device 170 may include an edge device.

[0041] It should be understood that although the generating device 140 and the testing device 170 are in Figure 1 While they are shown as separate entities, they can also be implemented in a single computing device. For example, both generation device 140 and detection device 170 can be implemented in an edge computing device.

[0042] Test plan generation

[0043] First refer to Figure 2 The illustration shows an example structure of a generation module 200 in a generation apparatus according to an embodiment of the subject matter described herein. For illustrative purposes, Figure 1 The generation module in generation device 140 is referred to as an example for implementing the detection plan generation described herein. Generation module 200 includes multiple modules for implementing multiple stages when generating at least one detection plan 150.

[0044] As shown in the figure, the generation module 200 may include a distribution determination module 210. In common object detection applications, the camera 110 is typically kept stationary at a fixed angle and position. Therefore, for a class of objects, their visible size often becomes similar over time when they are in close proximity to the captured view. Additionally, common objects (e.g., pedestrians and vehicles) tend to appear in certain areas of the captured view.

[0045] The distribution determination module 210 can, for example, obtain object detection results from the set of historical images 120 and learn the distribution of object sizes. Given a view V captured from camera 110, the distribution determination module 210 can generate a distribution vector F. V Object distribution information 220 (i.e., Figure 1 Distribution information in 125):

[0046] F V =<φ S0 , φ S1 , …φ M0 , φ M1 , …φ L2 , φ L3 > (1)

[0047] Where φ is the probability distribution of the detected objects at each size level described in Table 1, which illustrates the 12 fine-grained levels from smallest to largest.

[0048] Table 1. Different object size levels and their area in pixels.

[0049]

[0050] In some implementations, the FV can be obtained from the ground reality of the set of historical images 120. Alternatively, the distribution determination module 210 may also apply a labeling model (e.g., an oracle model) for labeling.

[0051] In addition, such as Figure 2 As shown, the generation module 200 may include a model profiler 240. The model profiler 240 may be configured to generate performance metrics 130 for the set of predetermined object detection models 180.

[0052] In some implementations, model profiler 240 can profile a time-to-accuracy tradeoff across objects of different sizes on the set of predetermined object detection models 180. Specifically, model profiler 240 can determine a time-to-latency metric for an object-specific detection model by executing the object-specific model multiple times on the detection device 170 using different batch sizes. This time-to-latency metric can indicate the estimated time required to process a batch of images by the object-specific detection model.

[0053] In some implementations, for each object detection model n∈N, the model profiler 240 can obtain its average latency time. Where b is the batch size, and N refers to the group of predefined object detection models 180.

[0054] Additionally, the model profiler 240 can determine a precision metric for the object-specific detection model based on applying it to detect objects at different size levels. The precision metric can indicate the estimated precision used by the object-specific detection model to detect objects at a specific size level.

[0055] In some implementations, the profiling process of model profiler 240 can evaluate the set of predetermined object detection models 180 for objects with different size levels as defined in Table 1. For each detection model n∈N, model profiler 240 determines a capability vector AP. n :

[0056] AP n =<τ S0 , τ S1 ,…τ M0 , τ M1 ,…τ L2 , τ L3 > (2)

[0057] Where τ is the detection accuracy at a specific object size level.

[0058] like Figure 2 As shown, the generation module 200 may additionally include a performance estimation module 230. The performance estimation module 230 may be configured to generate multiple candidate detection plans based on the distribution information 125 and the performance metric 130.

[0059] In some implementations, the performance estimation module 230 may first generate multiple potential partitioning patterns that indicate how to divide the camera's image view into at least one region. Furthermore, the performance estimation module 230 may generate multiple candidate detection plans based on the multiple partitioning patterns by assigning an object detection model to each of the group of regions.

[0060] In some implementations, the dynamic programming-based algorithm shown below can be used to determine multiple potential partitioning patterns. Intuitively, this algorithm can enumerate each possible partitioning pattern and estimate their detection accuracy and latency. Pseudocode for Algorithm 1 below is shown.

[0061]

[0062] As shown in Algorithm 1, for the camera's captured view, the algorithm combines its historical frames H (i.e., the set of historical images 180) with the capability vector AP of the candidate network (i.e., the object detection model) n∈N. n As input, the algorithm calculates all possible ways to process a given frame. First, the frame can be directly scaled and processed using an object detection model (e.g., a network). Therefore, for each network n∈N, the scaling ratio between the size of the captured view and the desired resolution n is calculated (line 4).

[0063] Next, scale the objects in H accordingly and update the object distribution vector (lines 5-6). Then, the estimated latency eLat and the estimated detection accuracy eAP can be updated using equations 3 and 4 below, respectively.

[0064]

[0065]

[0066] Here, k represents the partitioning plan (also referred to as the "detection plan" in this paper), which indicates not only how the image view is partitioned, but also which object detection model to apply to each region, and p represents the segmented region (or block) of the image view V. The distribution vector F can then be determined based on this set of historical images 120 by counting the corresponding objects whose boundary block centroids lie in region p. p .

[0067] Alternatively, the frames can be divided. For each network n∈N, the frames can be divided uniformly based on the network's input size (lines 10-11). Each divided block can then be recursively processed by the algorithm (lines 13-16). The partitioning scheme obtained from the divided blocks is labeled as the sub-partitioning scheme SK. p (Line 12). SK of all partitioned blocks p The permutations and combinations (line 17) are performed, and their estimated delay time eLat and estimated detection precision eAP are estimated accordingly.

[0068] Specifically, for a partitioning scheme k, its total detection precision eAP can be estimated by combining all estimates of each p contained in k. k .

[0069]

[0070] Where λ P p is the object density relative to the entire view V.

[0071] In some implementations, to reduce the search space, a pruning and search-based method can be employed. Specifically, during planning, if SK... p If multiple partitioning plans have similar eLat values ​​within 1ms, only the one with the highest eAP is retained (lines 9 and 17-18). In some alternative implementations, a cutoff delay threshold can be set, and any partitioning plan whose eLat value exceeds this threshold is discarded. In this way, the search space can be significantly reduced.

[0072] Using Algorithm 1, the performance estimation module 230 can generate multiple candidate detection plans (partition plans) and provide them to the detection plan generation module 260.

[0073] In addition, such as Figure 2 As shown, the detection plan generation module 260 can select at least one detection plan 150 based on the estimated performance of multiple candidate detection plans, wherein the estimated performance is determined based on object distribution information associated with the group of regions and performance metrics associated with the assigned object detection model.

[0074] Specifically, the detection plan generation module 260 can select at least one detection plan 150, for example, based on the estimated latency eLat and the estimated detection accuracy eAP. For instance, the detection plan generation module 260 can select at least one detection plan 150 based on a weighted sum of the estimated latency eLat and the estimated detection accuracy eAP. If the value of the weighted sum is less than a predetermined threshold, a candidate detection plan can be added to at least one detection plan 150.

[0075] In some other implementations, such as Figure 2 As shown, the detection plan generation module 260 can obtain an expected delay time 250 for object detection on the target image, which can indicate an upper limit for the processing time for object detection. For example, the expected delay time 250 can indicate that the processing time for object detection on the target image 160 should not exceed 10 seconds.

[0076] Additionally, the detection plan generation module 260 can select at least one detection plan 150 from multiple candidate detection plans based on a comparison of the estimated latency time of the candidate detection plans with the expected latency time 250. For example, the detection plan generation module 260 can select a detection plan with an eLat range from T to 1.5T, where T is the expected latency time. In some embodiments, the difference between the estimated latency time of at least one detection plan 150 and the expected latency time 250 can be less than a threshold difference.

[0077] Furthermore, segmenting higher-resolution inputs may separate objects around the segmentation boundaries, leading to detection failures. In some implementations, the detection plan generation module 260 may additionally update the at least one detection plan 150. For example, for a first detection plan indicating that an image view is divided into multiple regions, the detection plan generation module 260 may additionally update the first detection plan by adjusting the size of the multiple regions such that each pair of adjacent regions in the multiple regions partially overlaps.

[0078] In some alternative implementations, to minimize overhead, the minimum tolerance can be adaptive to each segmented region. The minimum vertical and horizontal edge sizes can be determined by the height and width of the potential object, which lie precisely at the boundaries of the region. Based on perspective, for a class of objects, their visible height and width within pixels are linearly related to their position on the vertical axis. Therefore, linear regression can be used to predict the height and width of objects using their positions. In some implementations, the detection plan generation module 260 can utilize historical detection results to obtain this linear relationship.

[0079] Figures 3A-3C The following diagram illustrates an example process for updating the detection plan described in this article. Figure 3A The diagram illustrates the original testing plan, 300A. Figure 3A As shown, there is no overlap between any adjacent regions, which may result in the object being divided into multiple regions.

[0080] like Figure 3B As shown, the detection plan generation module 260 can further update the size of the segmented region. For example, for region 310-1, the plan generation module 260 can extend the boundary of the first region by a certain distance. As discussed, the vertical and horizontal edge sizes can be determined by the height and width of the potential objects located at the boundaries 320-1 and 330-1 of region 310-1.

[0081] like Figure 3B As shown, Figure 3A The horizontal boundary 320-1 extends to the horizontal boundary 320-2 by a first distance. This first distance can be determined based on the width of the potential object located at boundary 330-1. Furthermore, Figure 3A The vertical boundary 330-1 extends to the second vertical boundary 330-2. The second distance can be determined based on the height of the potential object located at boundary 320-1.

[0082] By adding a minimum necessary margin to each segmented region, the detection plan generation module 260 can generate a plan such as Figure 3C The final detection plan is shown below. In this way, it can be ensured that objects are always completely covered in a single area.

[0083] In some implementations, the object distribution can be updated, and the detection plan can be regenerated after a period of time. Specifically, the generation module can obtain updated object distribution information associated with a new set of historical images captured by the camera, and generate at least one updated detection plan based on the updated object distribution information. Additionally, the generation module can provide at least one updated detection plan for object detection in the image to be captured by the camera.

[0084] For example, in the case of object detection in a shopping mall, historical images captured during the day can be used to schedule nightly updates and regeneration processes. The updated object detection schedule can then be used again for object detection the following day.

[0085] According to the reference Figure 2 The described process can adaptively generate a detection plan for object detection based on the historical distribution of objects and the characteristics of a predetermined set of object detection models (e.g., neural network-based models). In this way, different regions of the image can be assigned to corresponding detection models. Therefore, the balance between detection accuracy and detection latency can be improved.

[0086] Adaptive object detection

[0087] For further reference Figure 4 The illustration shows an example structure of the detection module 400 in a detection apparatus 170 according to an embodiment of the subject matter described herein. For illustrative purposes, Figure 1 The detection module in the detection device 170 is referred to as an instance for implementing the object detection described herein. Detection module 400 includes multiple modules for implementing multiple stages in object detection.

[0088] like Figure 4 As shown, the detection module 400 may include a sub-image selection module 410. The sub-image selection module 410 can divide the target image 160 captured by the camera 110 into at least one sub-image according to a division pattern. (See above reference...) Figure 2As discussed, in some embodiments, the partitioning pattern may include a predetermined partitioning pattern. For example, the partitioning pattern may indicate that the image view is uniformly divided into multiple regions based on a predetermined region size. In some other embodiments, the partitioning pattern may also be indicated by at least one received detection plan 150.

[0089] In some implementations, if the received at least one detection plan 150 includes multiple detection plans, the sub-image selection module 410 may first determine the target detection plan from the multiple detection plans. In some implementations, the sub-image selection module 410 may select the target detection plan from the multiple detection plans based on the expected delay time for object detection of the target image 160. For example, selecting the plan with the highest eAP. k Meanwhile its eLat k The detection plan within the expected delay time is used as the target detection plan. In some implementations, the estimated delay time of the selected target detection plan may not be greater than the expected delay time, and the difference between the estimated delay time and the expected delay time may be less than a predetermined threshold.

[0090] Once the target detection plan is determined, the sub-image selection module 410 can divide the target image 160 into at least one sub-image according to the division pattern indicated by the target detection plan.

[0091] Furthermore, the object detection module 430 can utilize a corresponding object detection model indicated by the object detection plan to perform object detection in each of at least one sub-image. In some implementations, if multiple sub-images are assigned to the same object detection model, these sub-images can be analyzed in batches by the object detection model, thereby accelerating inference.

[0092] Additionally, if the target image 160 is divided into multiple sub-images, the object detection module 430 can further merge the object detection results of the multiple sub-images to obtain the final object detection result 190 of the target image 160. It should be understood that any suitable merging method (e.g., non-maximum suppression (NMS) algorithm) can be applied, and this disclosure is not intended to be limiting in this respect.

[0093] Additionally, during runtime, some sub-images may not contain any target objects. For example, they may be covered by irrelevant backgrounds, or the objects may simply be temporarily absent from them. In some implementations, object detection in sub-images can also be performed adaptively. In particular, some sub-images can be skipped to save computational resources.

[0094] In some implementations, the sub-image selection module 410 may first determine a first set of sub-images from a plurality of sub-images to be skipped based on object detection results of a plurality of historical images processed according to an object detection plan. In some implementations, the first set of sub-images may include a first sub-image corresponding to a first region, wherein no object was detected from the sub-images corresponding to the first region of the plurality of historical images. In this way, regions determined to be free of objects based on historical images are skipped.

[0095] In some other implementations, the first set of sub-images may include a second sub-image corresponding to a second region, wherein object detection of the sub-image corresponding to the second region of the previous historical image is skipped.

[0096] In some other implementations, if object detection of the sub-image corresponding to the second region is skipped for multiple consecutive historical images, and the number of multiple consecutive historical images is less than a threshold number, then the second sub-image corresponding to the second region will be added to the first group of sub-images.

[0097] In some implementations, the sub-image selection module 410 may utilize previous detection results as feedback and deploy an exploration and exhaustion strategy. To strike a balance between exhaustion and exploration, the sub-image selection module 410 may determine the following rules:

[0098] 1) If no object can be detected in the region of the first number of inferences (i.e., the process for object detection), the sub-image corresponding to that region should be skipped in the subsequent second number of frames to save latency.

[0099] 2) If an object is detected in a sub-image corresponding to a region in the first image, the sub-image corresponding to that region should be processed in a subsequent second image (e.g., the next frame) to ensure detection accuracy.

[0100] 3) Considering that objects may still appear in skipped areas, if a block has already been skipped a third number of past inferences, the sub-image corresponding to that area will not be skipped.

[0101] In some alternative implementations, a concise and efficient Additive-Incremental-Multiplicative-Decrease (AIMD) solution can also be introduced. Specifically, for each partitioned block (or region) p, a penalty window w is assigned. p w p The value of w indicates that block p will be skipped for some of the following inferences. Initially, w... p The value is set to 0, therefore each block will be executed. Once the inference is complete, w is updated based on the detection results according to the following rules. p :

[0102] 1) If no object is detected in p, then according to w p =l p -1 linearly increases w p , where l p It is the continuous inference time when no object is detected in p.

[0103] 2) Once an object can be detected in p, replace multiplicative decrease of w. p w p It can be conservatively reset to 0 and 1 to ensure that p is processed in the next inference.

[0104] 3) If block p is skipped, then for each skipped inference, its w p Decrease by 1. Once its w p Returning to 0, the block will be processed in the next inference, thus giving it another chance to infer whether the object has appeared.

[0105] like Figure 4 As shown, the sub-image selection module 410 can generate a sub-image selection result 420. In such a way... Figure 4 In the sub-image selection result 420 shown, blocks with dark masks will be skipped. By applying the above conservative strategy, the implementation can skip blocks that do not contain objects with a high probability, resulting in a significant speedup of latency with minimal loss of accuracy.

[0106] Since several segmented sub-images may be skipped, the actual inference delay time L can be lower than the expected delay time T. This indicates that the system still has computational capacity. In some embodiments, to fully utilize the computational capacity, the generation module 400 may additionally include a planning controller 440, which can be configured to obtain historical delay times for object detection on historical images captured by the camera, wherein object detection on the historical images is performed according to a first detection plan among a plurality of detection plans. Furthermore, the planning controller 440 may additionally select a target detection plan from the plurality of detection plans based on the difference between the historical delay time and the expected delay time 450.

[0107] In some implementations, if the historical delay time is less than the expected delay time 450 and the difference is greater than a threshold, the planning controller 440 can select a target detection plan from a plurality of detection plans, wherein the estimated delay time of the target detection plan is greater than the estimated delay time of the first detection plan.

[0108] Specifically, the planning controller 440 can use the actual delay time L as feedback and continuously try different plans until L approaches T. More specifically, a closed-loop controller can be employed. The controller's desired setpoint (SP) is set to the desired delay time T. The measured process variable (PV) is L as feedback. Using SP and PV, the controller outputs a control value u. In some embodiments, u is an updated budget used to search for a plan from at least one received detection plan 150. The most accurate plan within u is selected. It should be understood that any suitable closed-loop controller (such as a proportional-integral-derivative (PID) controller) can be applied, and this disclosure is not intended to be limiting in this regard.

[0109] Example process

[0110] Figure 5 A flowchart illustrating a process 500 for generating a detection plan according to some embodiments of the subject matter described herein is shown. Process 500 can be implemented by generation device 140. Process 500 can also be implemented by any other device or cluster of devices similar to generation device 140. For purposes of description, reference is made to... Figure 1 Describe the process 500.

[0111] In block 510, generating device 140 obtains object distribution information associated with a set of historical images captured by a camera, the object distribution information indicating the size distribution of objects detected in the set of historical images.

[0112] In block 520, generating device 140 obtains performance metrics associated with a set of predetermined object detection models.

[0113] In block 530, generating device 140 generates at least one detection plan based on object distribution information and performance metrics, the at least one detection plan indicating which predetermined object detection model from the set of predetermined object detection models should be applied to each of at least one sub-image of the target image to be captured by the camera.

[0114] In block 540, generating device 140 provides at least one detection plan for object detection of target image.

[0115] In some implementations, object distribution information may include a set of distribution probabilities of the detected objects at multiple size levels.

[0116] In some implementations, the performance metrics associated with the object detection model may include at least one of the following: a latency metric indicating the estimated latency time required for the object detection model to process a batch of images; or an accuracy metric indicating the estimated accuracy required for the object detection model to detect objects at a specific size level.

[0117] In some implementations, generating at least one detection plan includes: generating multiple partitioning patterns based on the input size of a set of predetermined object detection models, the partitioning patterns indicating the division of an image view from a camera into a set of regions; generating multiple candidate detection plans based on the multiple partitioning plans by assigning object detection models to each of the regions in the set; and determining at least one detection plan based on the estimated performance of the multiple candidate detection plans, the estimated performance being determined based on object distribution information associated with each region in the set and a performance metric associated with the assigned object detection model.

[0118] In some implementations, determining the at least one detection plan includes: obtaining an expected delay time for object detection on a target image; and selecting at least one detection plan from a plurality of candidate detection plans based on a comparison of the estimated delay time of the candidate detection plans with the expected delay time.

[0119] In some implementations, the difference between the estimated delay time and the expected delay time of at least one detection plan is less than a threshold difference.

[0120] In some implementations, process 500 further includes updating the at least one detection plan, including updating the first detection plan by adjusting the size of the plurality of regions such that each pair of adjacent regions in the plurality of regions partially overlap, for a first detection plan in at least one detection plan indicating that the image view is divided into a plurality of regions.

[0121] In some implementations, updating the first candidate detection plan includes: for a first region in a plurality of regions, extending the boundary of the first region by a distance determined based on an estimated size of a potential object located at the boundary of the first region.

[0122] In some implementations, process 500 further includes: obtaining updated object distribution information associated with a new set of historical images captured by the camera; generating at least one updated detection plan based on the updated object distribution information; and providing at least one updated detection plan for object detection in the images to be captured by the camera.

[0123] Figure 6 A flowchart of an object detection process 600 according to some embodiments of the subject matter described herein is illustrated. Process 600 can be implemented by a detection device 170. Process 600 can also be implemented by any other device or cluster of devices similar to detection device 170. For purposes of description, reference is made to... Figure 1 Describe the process 600.

[0124] In block 610, the detection device 170 divides the target image captured by the camera into at least one sub-image.

[0125] In block 620, detection device 170 detects objects in at least one sub-image according to a target detection plan, the target detection plan indicating which predetermined object detection model from a set of predetermined object detection models should be applied to each of the at least one sub-image.

[0126] In block 630, the detection device 170 determines objects in the target image based on objects detected in at least one sub-image.

[0127] In some implementations, an object detection plan is generated based on object distribution information associated with a set of historical images captured by a camera and performance metrics associated with a set of predetermined object detection models.

[0128] In some implementations, dividing a target image captured by a camera into at least one sub-image includes dividing the target image into at least one sub-image according to a division plan indicated by a target detection plan.

[0129] In some implementations, process 600 further includes: obtaining a plurality of detection plans; and selecting a target plan from the plurality of detection plans based on the expected delay time for object detection of the target image.

[0130] In some implementations, selecting a target plan from multiple detection plans based on the expected delay time for object detection of a target image includes: obtaining a historical delay time for object detection of historical images captured by a camera, wherein object detection of the historical images is performed according to a first detection plan among multiple detection plans; and selecting a target detection plan from multiple detection plans based on the difference between the historical delay time and the expected delay time.

[0131] In some implementations, selecting a target detection plan from multiple detection plans based on the difference between historical latency and expected latency includes: selecting a target detection plan from multiple detection plans based on determining that the historical latency is less than the expected latency and the difference is greater than a threshold, wherein the estimated latency of the target detection plan is greater than the estimated latency of a first detection plan.

[0132] In some implementations, the at least one image includes a plurality of sub-images, and detecting objects in the at least one sub-image according to an object detection plan includes: determining a first set of sub-images from the plurality of sub-images based on object detection results of a plurality of historical images obtained according to the object detection plan; and detecting objects in the plurality of sub-images by skipping object detection of the first set of sub-images.

[0133] In some implementations, the first set of sub-images includes at least one of the following: a first sub-image corresponding to a first region, wherein no object was detected from the sub-images corresponding to the first region of a plurality of historical images, or a second sub-image corresponding to a second region, wherein object detection for the sub-images corresponding to the second region of a previous historical image was skipped.

[0134] In some implementations, for multiple consecutive historical images, object detection of the sub-image corresponding to the second region is skipped, and the number of multiple consecutive historical images is less than a threshold number.

[0135] Exemplary device

[0136] Figure 7 A block diagram of a computing device 700 in which various embodiments of the subject matter described herein may be implemented is illustrated. It should be understood that... Figure 7 The computing device 700 shown is for illustrative purposes only and is not intended to limit the functionality and scope of the embodiments of the subject matter described herein in any way. Figure 7 As shown, computing device 700 includes general-purpose computing device 700. Components of computing device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760.

[0137] In some implementations, computing device 700 can be implemented as any user terminal or server terminal with computing capabilities. The server terminal can be a server provided by a service provider, a large computing device, etc. The user terminal can be, for example, any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, stations, units, devices, multimedia computers, multimedia input boards, Internet nodes, communicators, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is conceivable that computing device 700 can support any type of interface to the user (such as "wearable" circuitry devices, etc.).

[0138] The processing unit 710 can be a physical or virtual processor and can implement various processes based on programs stored in the memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capabilities of the computing device 700. The processing unit 710 may also be referred to as a central processing unit (CPU), microprocessor, controller, or microcontroller.

[0139] Computing device 700 typically includes various computer storage media. These media can be any media accessible to computing device 700, including but not limited to volatile and non-volatile media, or removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache memory, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or flash memory) or any combination thereof. Storage device 730 can be any removable or non-removable media and may include machine-readable media, such as memory, flash drives, disks, or other media that can be used to store information and / or data and can be accessed within computing device 700.

[0140] The computing device 700 may additionally include removable / non-removable, volatile / non-volatile storage media. Although in Figure 7 Not shown, but may include disk drives for reading from and / or writing to removable and non-volatile disks, and optical disc drives for reading from and / or writing to removable and non-volatile optical discs. In such cases, each drive may be connected to a bus (not shown) via one or more data media interfaces.

[0141] The communication unit 740 communicates with another computing device via a communication medium. Furthermore, the functionality of the components in the computing device 700 can be implemented by a single computing cluster or multiple computers that can communicate via communication connections. Therefore, the computing device 700 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general-purpose network nodes.

[0142] Input device 750 can be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 760 can be one or more of various output devices, such as a monitor, speaker, printer, etc. With the aid of communication unit 740, computing device 700 can additionally communicate with one or more external devices (not shown), such as storage devices and display devices, which enable users to interact with computing device 700, or any device (such as a network card, modem, etc.) that enables computing device 700 to communicate with one or more other computing devices, if needed. Such communication can be performed via input / output (I / O) interface (not shown).

[0143] In some implementations, as an alternative to integration into a single device, some or all components of computing device 700 may also be arranged within a cloud computing architecture. In a cloud computing architecture, components can be remotely provided and work together to implement the functionality described in the subject matter herein. In some implementations, cloud computing provides computing, software, data access, and storage services without requiring end users to know the physical location or configuration of the systems or hardware providing these services. In various implementations, cloud computing provides services via a wide area network (WAN), such as the Internet, using appropriate protocols. For example, a cloud computing provider offers applications over a WAN that can be accessed via a web browser or any other computing component. The software or components of the cloud computing architecture, along with the corresponding data, may be stored on servers at remote locations. Computing resources in a cloud computing environment may be consolidated or distributed across locations in remote data centers. Cloud computing infrastructure may be provided through shared data centers, although they represent a single point of access for the user. Thus, cloud computing architectures can be used to provide the components and functionality described herein from service providers at remote locations. Alternatively, they may be provided from conventional servers or directly installed or otherwise installed on client devices.

[0144] The computing device 700 can be used to implement detection plan generation and / or object detection in embodiments of the subject matter described herein. The memory 720 may include one or more generation or detection modules 725 having one or more program instructions. These modules can be accessed and executed by the processing unit 710 to perform the functionality of the various embodiments described herein.

[0145] Example Implementation

[0146] The following are some example implementations of the topics described in this article.

[0147] In a first aspect, the subject matter described herein provides a computer-implemented method. The method includes: obtaining object distribution information associated with a set of historical images captured by a camera, the object distribution information indicating the size distribution of detected objects in the set of historical images; obtaining performance metrics associated with a set of predetermined object detection models; generating at least one detection plan based on the object distribution information and the performance metrics, the at least one detection plan indicating which predetermined object detection model from the set should be applied to each of at least one sub-image in a target image to be captured by a camera; and providing at least one detection plan for object detection in the target image.

[0148] In some implementations, object distribution information includes a set of probability distributions of detected objects across multiple size levels.

[0149] In some implementations, the performance metrics associated with the object detection model include at least one of the following: a latency metric indicating the estimated latency time for processing a batch of images by the object detection model; or an accuracy metric indicating the estimated accuracy for detecting objects at a specific size level by the object detection model.

[0150] In some implementations, generating at least one detection plan includes: generating multiple partitioning patterns based on the input size of a predetermined set of object detection models, the partitioning patterns indicating the division of an image view of a camera into a set of regions; generating multiple candidate detection plans based on the multiple partitioning patterns by assigning an object detection model to each of the regions in the set of regions; and determining at least one detection plan based on the estimated performance of the multiple candidate detection plans, the estimated performance being determined based on object distribution information associated with each region in the set of regions and a performance metric associated with the assigned object detection model.

[0151] In some implementations, obtaining the at least one detection plan includes: obtaining an expected delay time for object detection of a target image; and selecting at least one detection plan from a plurality of candidate detection plans based on a comparison of the estimated delay time of the candidate detection plans with the expected delay time.

[0152] In some implementations, the difference between the estimated delay time and the expected delay time of at least one detection plan is less than a threshold difference.

[0153] In some embodiments, the method further includes updating the at least one detection plan, including updating the first detection plan by adjusting the size of the plurality of regions such that each pair of adjacent regions in the plurality of regions partially overlap, for a first detection plan in at least one detection plan indicating that the image view will be divided into a plurality of regions.

[0154] In some implementations, updating the first candidate detection plan includes: for a first region in a plurality of regions, extending the boundary of the first region by a distance determined based on an estimated size of a potential object located at the boundary of the first region.

[0155] In some embodiments, the method further includes: obtaining updated object distribution information associated with a new set of historical images captured by the camera; generating at least one updated detection plan based on the updated object distribution information; and providing at least one updated detection plan for object detection in the images to be captured by the camera.

[0156] In a second aspect, the subject matter described herein provides a computer-implemented method. The method includes: dividing a target image captured by a camera into at least one sub-image; detecting objects in the at least one sub-image according to an object detection plan, the object detection plan indicating which predetermined object detection model from a set of predetermined object detection models should be applied to each of the at least one sub-image; and determining objects in the target image based on the detected objects in the at least one sub-image.

[0157] In some implementations, an object detection plan is generated based on object distribution information associated with a set of historical images captured by a camera and performance metrics associated with a set of predetermined object detection models.

[0158] In some implementations, dividing a target image captured by a camera into at least one sub-image includes dividing the target image into at least one sub-image according to a division plan indicated by a target detection plan.

[0159] In some implementations, the method further includes: obtaining a plurality of detection plans; and selecting a target plan from the plurality of detection plans based on the expected delay time for object detection of the target image.

[0160] In some implementations, selecting a target plan from multiple detection plans based on the expected delay time for object detection of a target image includes: obtaining a historical delay time for object detection of historical images captured by a camera, wherein object detection of the historical images is performed according to a first detection plan among multiple detection plans; and selecting a target detection plan from multiple detection plans based on the difference between the historical delay time and the expected delay time.

[0161] In some implementations, selecting a target detection plan from multiple detection plans based on the difference between historical latency and expected latency includes: selecting a target detection plan from multiple detection plans based on determining that the historical latency is less than the expected latency and the difference is greater than a threshold, wherein the estimated latency of the target detection plan is greater than the estimated latency of a first detection plan.

[0162] In some implementations, the at least one image includes a plurality of sub-images, and the detection of objects in the at least one sub-image according to a target detection plan includes: obtaining a historical delay time for object detection of historical images captured by the camera, wherein object detection of the historical images is performed according to a first detection plan among a plurality of detection plans; and selecting a target detection plan from the plurality of detection plans based on the difference between the historical delay time and the desired delay time.

[0163] In some implementations, the first set of sub-images includes at least one of the following: a first sub-image corresponding to a first region, wherein no object was detected from the sub-images corresponding to the first region of a plurality of historical images, or a second sub-image corresponding to a second region, wherein object detection for the sub-images corresponding to the second region of a previous historical image was skipped.

[0164] In some implementations, for multiple consecutive historical images, object detection of the sub-image corresponding to the second region is skipped, and the number of multiple consecutive historical images is less than a threshold number.

[0165] In a third aspect, the subject matter described herein provides an electronic device. The electronic device includes a processing unit; and a memory coupled to the processing unit and having instructions stored thereon, which, when executed by the processing unit, cause the electronic device to perform the following actions: obtaining object distribution information associated with a set of historical images captured by a camera, the object distribution information indicating the size distribution of detected objects in the set of historical images; obtaining performance metrics associated with a set of predetermined object detection models; generating at least one detection plan based on the object distribution information and the performance metrics, the at least one detection plan indicating which predetermined object detection model from the set should be applied to each of at least one sub-image in a target image to be captured by a camera; and providing at least one detection plan for object detection in the target image.

[0166] In some implementations, object distribution information includes a set of probability distributions of detected objects across multiple size levels.

[0167] In some implementations, the performance metrics associated with the object detection model include at least one of the following: a latency metric indicating the estimated latency time for processing a batch of images by the object detection model; or an accuracy metric indicating the estimated accuracy for detecting objects at a specific size level by the object detection model.

[0168] In some implementations, generating at least one detection plan includes: generating multiple partitioning patterns based on the input size of a set of predetermined object detection models, the partitioning patterns indicating the division of an image view from a camera into a set of regions; generating multiple candidate detection plans based on the multiple partitioning patterns by assigning an object detection model to each of the regions in the set of regions; and determining at least one detection plan based on the estimated performance of the multiple candidate detection plans, the estimated performance being determined based on object distribution information associated with each region in the set of regions and a performance metric associated with the assigned object detection model.

[0169] In some implementations, obtaining the at least one detection plan includes: obtaining an expected delay time for object detection of a target image; and selecting at least one detection plan from a plurality of candidate detection plans based on a comparison of the estimated delay time of the candidate detection plans with the expected delay time.

[0170] In some implementations, the difference between the estimated delay time and the expected delay time of at least one detection plan is less than a threshold difference.

[0171] In some implementations, these actions further include updating the at least one detection plan, including updating the first detection plan by adjusting the size of the multiple regions such that each pair of adjacent regions in the multiple regions partially overlap, for a first detection plan in at least one detection plan indicating that the image view will be divided into multiple regions.

[0172] In some implementations, updating the first candidate detection plan includes: for a first region in a plurality of regions, extending the boundary of the first region by a distance determined based on an estimated size of a potential object located at the boundary of the first region.

[0173] In some implementations, these actions further include: obtaining updated object distribution information associated with a new set of historical images captured by the camera; generating at least one updated detection plan based on the updated object distribution information; and providing at least one updated detection plan for object detection in the images to be captured by the camera.

[0174] In a fourth aspect, the subject matter described herein provides an electronic device. The electronic device includes a processing unit; and a memory coupled to the processing unit and having instructions stored thereon, which, when executed by the processing unit, cause the electronic device to perform the following actions: dividing a target image captured by a camera into at least one sub-image; detecting objects in the at least one sub-image according to a target detection plan, the target detection plan indicating which predetermined object detection model from a set of predetermined object detection models should be applied to each of the at least one sub-image; and determining objects in the target image based on the detected objects in the at least one sub-image.

[0175] In some implementations, an object detection plan is generated based on object distribution information associated with a set of historical images captured by a camera and performance metrics associated with a set of predetermined object detection models.

[0176] In some implementations, dividing a target image captured by a camera into at least one sub-image includes dividing the target image into at least one sub-image according to a division plan indicated by a target detection plan.

[0177] In some implementations, these actions further include: obtaining multiple detection plans; and selecting a target plan from the multiple detection plans based on the expected delay time for object detection of the target image.

[0178] In some implementations, selecting a target plan from multiple detection plans based on the expected delay time for object detection of a target image includes: obtaining a historical delay time for object detection of historical images captured by a camera, wherein object detection of the historical images is performed according to a first detection plan among multiple detection plans; and selecting a target detection plan from multiple detection plans based on the difference between the historical delay time and the expected delay time.

[0179] In some implementations, selecting a target detection plan from multiple detection plans based on the difference between historical latency and expected latency includes: selecting a target detection plan from multiple detection plans based on determining that the historical latency is less than the expected latency and the difference is greater than a threshold, wherein the estimated latency of the target detection plan is greater than the estimated latency of a first detection plan.

[0180] In some implementations, the at least one image includes a plurality of sub-images, and the detection of objects in the at least one sub-image according to a target detection plan includes: obtaining a historical delay time for object detection of historical images captured by the camera, wherein object detection of the historical images is performed according to a first detection plan among a plurality of detection plans; and selecting a target detection plan from the plurality of detection plans based on the difference between the historical delay time and the desired delay time.

[0181] In some implementations, the first set of sub-images includes at least one of the following: a first sub-image corresponding to a first region, wherein no object was detected from the sub-images corresponding to the first region of a plurality of historical images, or a second sub-image corresponding to a second region, wherein object detection for the sub-images corresponding to the second region of a previous historical image was skipped.

[0182] In some implementations, for multiple consecutive historical images, object detection of the sub-image corresponding to the second region is skipped, and the number of multiple consecutive historical images is less than a threshold number.

[0183] In a fifth aspect, the subject matter described herein provides a computer program product tangibly stored on a computer storage medium and comprising machine-executable instructions that, when executed by a device, cause the device to perform the method according to the first and / or second aspect. The computer storage medium may be a non-transitory computer storage medium.

[0184] In a sixth aspect, the subject matter described herein provides a non-transitory computer storage medium having machine-executable instructions stored thereon, which, when executed by a device, cause the device to perform the method according to the first and / or second aspect.

[0185] The functionality described herein can be performed, at least in part, by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0186] Program code used to perform the methods described herein can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code enables the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed wholly or partially on the machine, partially on the machine as a standalone software package, partially on a remote machine, or entirely on a remote machine or server.

[0187] In the context of this disclosure, a machine-readable medium can be any tangible medium that can contain or store a program used by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples of machine-readable storage media will include electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0188] Furthermore, although operations are described in a specific order, this should not be construed as requiring that such operations be performed in the specific order shown or sequentially, or that all shown operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are contained in the above discussion, these details should not be interpreted as limiting the scope of the subject matter described herein, but rather as descriptions of features that may be specific to particular implementations. Certain features described in the context of discrete implementations may also be implemented in combination in a single implementation. Conversely, various features described in a single implementation may also be implemented discretely in multiple implementations or in any suitable sub-combination.

[0189] Although the subject matter has been described using language specific to structural features and / or methodological actions, it should be understood that the subject matter specified in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as exemplary forms of implementing the claims.

Claims

1. A computer-implemented method, comprising: Obtain object distribution information associated with a set of historical images captured by a camera, the object distribution information indicating the size distribution of objects detected in the set of historical images; Obtain performance metrics associated with a predefined set of object detection models; At least one detection plan is generated based on the object distribution information and the performance metric, the at least one detection plan indicating which of the predetermined object detection models in the set of predetermined object detection models should be applied to at least one sub-image of the target image to be captured by the camera; as well as Provide at least one detection plan for object detection of the target image.

2. The method of claim 1, wherein the performance metric associated with the object detection model includes at least one of the following: A latency metric indicating the estimated latency time required for the object detection model to process a batch of images; or Accuracy metric, indicating the estimated accuracy of the object detection model for detecting objects at a specific size level.

3. The method of claim 1, wherein generating at least one detection plan comprises: Multiple partitioning patterns are generated based on the input size of the set of predetermined object detection models, and the partitioning patterns indicate that the image view of the camera is divided into a set of regions; Multiple candidate detection plans are generated based on the multiple partitioning patterns by assigning object detection models to each of the set of regions. as well as The at least one detection plan is determined based on the estimated performance of the plurality of candidate detection plans, the estimated performance being determined based on object distribution information associated with each of the set of regions and a performance metric associated with the assigned object detection model.

4. The method of claim 3, wherein determining the at least one detection plan comprises: Obtain the expected delay time for object detection in the target image; as well as Based on the comparison between the estimated delay time of the candidate detection plan and the expected delay time, at least one detection plan is selected from the plurality of candidate detection plans.

5. The method of claim 3, further comprising: Updating the at least one detection plan includes: For a first detection plan in at least one detection plan that indicates the image view will be divided into multiple regions, the first detection plan is updated by adjusting the size of the multiple regions such that each pair of adjacent regions in the multiple regions partially overlaps.

6. The method of claim 5, wherein updating the first detection plan comprises: For a first region among the plurality of regions, the boundary of the first region is extended by a certain distance, the distance being determined based on an estimated size of a potential object located at the boundary of the first region.

7. The method according to claim 1, further comprising: Obtain updated object distribution information associated with a new set of historical images captured by the camera; At least one updated detection plan is generated based on the updated object distribution information; as well as Provide at least one updated detection plan for object detection in an image to be captured by the camera.

8. A computer-implemented method, comprising: Divide the target image captured by the camera into at least one sub-image; Objects in the at least one sub-image are detected according to an object detection plan, the object detection plan indicating which of a set of predetermined object detection models should be applied to the at least one sub-image, wherein the object detection plan is generated based on object distribution information associated with a set of historical images captured by the camera and performance metrics associated with the set of predetermined object detection models; as well as The objects in the target image are determined based on the detected objects in the at least one sub-image.

9. The method of claim 8, further comprising: Multiple testing plans were obtained; as well as The target detection plan is selected from the plurality of detection plans based on the expected delay time for object detection of the target image.

10. The method of claim 9, wherein selecting the target detection plan from the plurality of detection plans based on the expected delay time for object detection of the target image comprises: A historical delay time is obtained for object detection on historical images captured by the camera, wherein the object detection on the historical images is performed according to a first detection plan among the plurality of detection plans; as well as The target detection plan is selected from the plurality of detection plans based on the difference between the historical delay time and the expected delay time.

11. The method of claim 8, wherein the at least one sub-image comprises a plurality of sub-images, and wherein detecting objects in the at least one sub-image according to an object detection plan comprises: Based on the object detection results of multiple historical images obtained according to the target detection plan, a first group of sub-images is determined from the multiple sub-images; as well as Objects in the plurality of sub-images are detected by skipping object detection in the first set of sub-images.

12. The method of claim 11, wherein the first set of sub-images comprises at least one of the following: The first sub-image corresponding to the first region, wherein no object was detected from the sub-image corresponding to the first region of the plurality of historical images, or The second sub-image corresponding to the second region, wherein object detection for the sub-image corresponding to the second region of the previous historical image is skipped.

13. The method of claim 12, wherein for a plurality of consecutive historical images, object detection of the sub-image corresponding to the second region is skipped, and wherein the number of the plurality of consecutive historical images is less than a threshold number.

14. An electronic device, comprising: Processing unit; as well as A memory coupled to the processing unit and having instructions stored thereon, which, when executed by the processing unit, cause the electronic device to perform the method according to any one of claims 1-13.

Images