A multi-camera collaborative sensing method and system for heterogeneous computing power
By generating cross-view candidate sensing bands and pre-compiling a heterogeneous execution scheme library, the problem of the disconnect between cross-view collaboration and heterogeneous computing power scheduling in multi-camera collaborative sensing is solved, enabling stable operation and efficient computing of edge devices under high load.
Patent Information
- Application Number
- CN202611311234.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-27
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, multi-camera collaborative sensing in edge devices suffers from problems such as fragmented cross-view collaboration and heterogeneous computing power scheduling, disconnect between computing power scheduling signals and image semantic content, lack of unified arbitration for multi-camera service quality configuration, easy frequency reduction of edge devices under high load, and overhead and jitter introduced by frequent switching.
By generating cross-perspective candidate sensing bands, quantizing them into sensing semantic loads, pre-compiling a heterogeneous execution scheme library, and combining sensing semantic loads, dynamic power budgets, and latency budgets, the heterogeneous execution schemes and service quality configurations are dynamically adjusted to avoid frequent switching and reduce runtime overhead.
It enables heterogeneous computing power scheduling to respond to dynamic changes in screen content and targets, reduces the overhead of switching heterogeneous solutions, maintains the continuous throughput of edge devices under limited heat dissipation conditions, suppresses frequent switching, and ensures power consumption and latency constraints.
Smart Images

Figure CN123123740A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of edge intelligence, video image processing, and heterogeneous computing scheduling, and particularly to a multi-camera collaborative sensing method and system for heterogeneous computing power, as well as related equipment and storage media. Background Technology
[0002] With the application of edge AI devices in scenarios such as perimeter monitoring, mobile robots, industrial online inspection, and traffic intersection sensing, multi-camera collaborative sensing has become a common deployment model. Multiple cameras can expand the sensing range and provide redundant information across different viewpoints, but they also bring problems such as concurrent processing of multiple video streams, repeated inference in repetitive areas, increased power consumption of edge devices, and thermal throttling during continuous operation.
[0003] In existing technologies, one type of approach focuses on regional collaboration among multiple cameras, such as using overlapping views between adjacent cameras to transfer target boxes or regions of interest, thereby reducing some redundant computations. This type of approach typically uses cross-view results to reduce data transfer, pixel processing, or improve detection robustness, but rarely transforms cross-view perception results into execution scheme switching signals between heterogeneous computing units within a single edge device. Even if some approaches perform load balancing based on computing power differences between cameras or devices, their heterogeneity is reflected between different cameras or devices, rather than in the execution scheme switching between heterogeneous computing units such as NPU, GPU, DSP, and CPU within a single edge device.
[0004] Another type of solution focuses on the computational scheduling of video analytics tasks, such as determining the allocation of video stream access and analysis tasks based on the number of cameras, task complexity, target presence, or server load. This type of solution is typically geared towards server clusters, camera clusters, or homogeneous computing power pools, and is difficult to directly adapt to runtime solution switching between heterogeneous computing units such as NPUs, GPUs, DSPs, and CPUs within a single edge device.
[0005] Another type of solution focuses on thermal sensing scheduling for heterogeneous processors or mobile neural network inference, such as migrating tasks between different processors based on temperature, utilization, power consumption, or task load. This type of solution typically uses general task load or processor state as the basis for scheduling, without utilizing the content of multiple camera images, cross-view area relationships, and target dynamic information to form a perceptual semantic load, nor does it jointly adjust the resolution, frame rate, and accuracy of multiple cameras.
[0006] Even when the above-mentioned schemes are combined, such as using heterogeneous processor thermal sensing scheduling to undertake cross-camera region of interest collaboration, they still do not use the perceptual semantic load obtained from cross-view propagation as a unified signal to jointly drive the switching of pre-compiled heterogeneous execution schemes within a single edge device and the arbitration of the quality of service configuration of multiple image acquisition channels under the constraints of junction temperature and power consumption budget. They also do not model and suppress the cost of execution scheme switching itself and the frequent switching jitter caused by load fluctuations.
[0007] Therefore, the existing technology has at least the following shortcomings: multi-camera cross-view collaboration and heterogeneous computing power scheduling are disconnected; computing power scheduling signals are decoupled from the semantic content of the image; there is a lack of unified arbitration for the service quality configuration of multi-camera and heterogeneous execution schemes; edge devices are prone to frequency reduction due to increased junction temperature under continuous high load; and frequent switching of heterogeneous execution schemes during runtime will also introduce additional overhead and jitter. Summary of the Invention
[0008] Technical problems to be solved
[0009] This invention aims to solve at least the following technical problems:
[0010] (1) How to convert cross-view candidate regions among multiple cameras into perceptual semantic loads that can be used for heterogeneous computing power scheduling;
[0011] (2) How to switch between low-overhead and feasible execution schemes based on heterogeneous computing units such as NPU, GPU, DSP, and CPU within a single edge device;
[0012] (3) How to determine the heterogeneous execution scheme and the quality of service configuration of multiple image acquisition channels under the constraints of junction temperature and power consumption budget;
[0013] (4) How to avoid frequent switching of heterogeneous execution schemes due to load fluctuations, thereby reducing switching overhead and latency jitter.
[0014] Technical solution
[0015] To address the aforementioned technical problems, this invention provides a multi-camera collaborative sensing method for heterogeneous computing power. This method is applicable to applications including... Edge computing devices consisting of image acquisition channels and at least two types of heterogeneous computing units, This method first acquires the current frame group of multiple image acquisition channels, and performs low-resolution or low-frequency trigger detection on at least one image acquisition channel to obtain the region of interest.
[0016] For image acquisition channels with overlapping fields of view, this method maps the region of interest (ROI) of the source image acquisition channel to the target image acquisition channel based on a pre-calibrated ground plane homography relationship, thereby generating candidate sensing bands for the target image acquisition channel. Confirmation detection is performed on the candidate sensing bands, and independent detection is performed on non-overlapping regions, thus reducing redundant calculations in overlapping regions while ensuring coverage of non-overlapping regions.
[0017] Furthermore, this method calculates the perceptual semantic load of each image acquisition channel based on the proportion of candidate perceptual zone area, the number of targets, the influx rate of new targets, and the intensity of motion. The perceptual semantic load is used to characterize the actual current demand for computing resources and quality of service for each video stream.
[0018] This method reads the current junction temperature and current power consumption of the edge computing device, determines the dynamic power budget based on the current junction temperature, and judges whether the current execution scheme violates the power hard constraint based on the current power consumption and the dynamic power budget.
[0019] The pre-compiled heterogeneous execution scheme library is established offline by testing the executability, latency, and power consumption of operators or computation graph partitions of the perceptron on different heterogeneous computing units. Based on the test results, the system constructs a finite number of computation graph mappings and discrete precision file combinations, generates a complete executable compilation product for each combination, and verifies the consistency of input / output tensor interfaces, buffer ownership, storage layout, and precision conversion between partitions.
[0020] When an edge computing device starts up, it loads all compiled artifacts and their metadata into memory, and establishes execution handles and corresponding buffer configurations. Runtime scheme switching only changes the execution handles, input / output routes, and buffer configurations; it does not recompile the computation graph online.
[0021] This method, based on perceptual semantic load, dynamic power budget, latency budget, and a pre-compiled heterogeneous execution scheme library, jointly determines candidate execution schemes and candidate quality of service (QoS) configurations from a limited set of execution schemes and finite QoS levels for each image acquisition channel. The perceptual semantic load serves as a unified scheduling signal, simultaneously driving execution scheme selection and QoS arbitration.
[0022] After obtaining candidate execution schemes, the method performs switching gating based on the scheme switching cost, hysteresis threshold, and minimum number of residency frames. The scheme switching cost is determined by a normalized weighted sum of warm-up delay, state transition amount, and cache or context reconstruction overhead. If the net utility gain is insufficient, or the residency time of the current execution scheme is insufficient, the current execution scheme is used; when the current execution scheme violates the hard power constraint or hard latency constraint, the normal switching gating is bypassed to perform a protective switching.
[0023] Finally, the edge computing device performs inference on the heterogeneous computing unit according to the final determined execution scheme and quality of service configuration, outputs collaborative perception results, and updates the cost model based on the measured frame latency and measured power consumption.
[0024] Beneficial effects
[0025] Compared with the prior art, the present invention has at least the following beneficial effects:
[0026] (1) Further quantify the cross-view candidate perception band into perception semantic load, so that the heterogeneous computing power scheduling can directly respond to the dynamic changes of screen content and target;
[0027] (2) By pre-compiling the heterogeneous execution scheme library, arbitrary operator migration and graph recompilation at runtime are avoided, thereby reducing the runtime overhead of switching heterogeneous schemes;
[0028] (3) Under the constraints of junction temperature and power consumption budget, jointly determine the heterogeneous execution scheme and multi-path service quality configuration so that edge devices can maintain continuous throughput under limited heat dissipation conditions;
[0029] (4) Frequent handover is suppressed by handover cost, hysteresis threshold and minimum number of dwell frames, and power consumption and latency hard constraints are given priority by protection handover;
[0030] (5) In overlapping areas, cross-view propagation is used to reduce redundant calculations, and in non-overlapping areas, independent detection of the local path is used to ensure coverage. Attached Figure Description
[0031] Figure 1 This is a schematic diagram of a multi-camera collaborative sensing system for heterogeneous computing power provided in an embodiment of the present invention.
[0032] Figure 2 This is a schematic diagram of the multi-camera collaborative perception method for heterogeneous computing power provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the switching relationship between the pre-compiled heterogeneous execution scheme library and the runtime provided in this embodiment of the invention; Figure 4 This is a schematic diagram of the joint arbitration process for service quality, thermal power consumption, and switching costs provided in an embodiment of the present invention. Detailed Implementation
[0033] The present invention will be further described below with reference to embodiments. It should be understood that the embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.
[0034] Example 1: System Structure
[0035] In one embodiment, the edge computing device includes a multi-channel image acquisition layer, a trigger detection module, a cross-view propagation and candidate sensing band module, a confirmation and supplementary detection module, a sensing semantic load estimation module, a junction temperature and power consumption budget module, a pre-compiled heterogeneous execution scheme library, a scheme selection and quality of service budget arbitration module, a switching gating module, a heterogeneous execution layer, and a runtime monitoring and feedback module.
[0036] The multi-channel image acquisition layer is used for access. Image acquisition channels. At least some of the image acquisition channels have overlapping fields of view. For image acquisition channels with overlapping fields of view, the ground plane homography matrix is obtained in advance during the system deployment or calibration phase.
[0037] The trigger detection module performs low-resolution or low-frequency detection on one or more image acquisition channels to generate regions of interest (ROIs). The cross-view propagation and candidate receptive band module maps ROIs in the source image acquisition channels to target image acquisition channels with overlapping viewpoints and generates candidate receptive bands. The perceptual semantic load estimation module calculates the perceptual semantic load based on the area ratio of candidate receptive bands, the number of targets, the influx rate of new targets, and the intensity of motion.
[0038] The junction temperature and power consumption budget module reads the junction temperature and power consumption information of the edge computing device and generates a dynamic power consumption budget. The scheme selection and quality of service budget arbitration module jointly selects from a pre-compiled heterogeneous execution scheme library and multiple quality of service configurations based on the perceived semantic load, dynamic power consumption budget, and latency budget. The switching gating module controls whether to switch from the current execution scheme to a candidate execution scheme based on switching cost, hysteresis threshold, and minimum number of resident frames. The heterogeneous execution layer includes at least two types of heterogeneous computing units: NPU, GPU, DSP, and CPU.
[0039] Example 2: Candidate Sensing Band Generation
[0040] In one embodiment, for each image acquisition channel The system maintains an overlapping adjacency set. Only when the source image acquisition channel Only then will the system acquire the source image channel. The region of interest in the image is mapped to the image acquisition channel. .
[0041] Specifically, source image acquisition channel The homogeneous coordinates of the vertices of the region of interest are Source image acquisition channel To the target image acquisition channel The homography matrix of the ground plane is Then the mapped pixel coordinates are:
[0042] Equation (1)
[0043] Candidate sensing band From the target image acquisition channel The union of the region of interest in this path's self-checking and the mapped regions in the overlapping adjacency set, followed by tolerance expansion, yields:
[0044] Equation (2)
[0045] in, This is used to absorb mapping deviations caused by camera calibration errors, slight parallax, and target deviations from the common ground plane. For non-overlapping regions, the system does not perform cross-view mapping but instead supplements it with independent detection within the local path.
[0046] Example 3: Perceptual Semantic Load Computation
[0047] In one embodiment, the system will... The proportion of candidate sensing band area in the image acquisition channel is denoted as . The target quantity is denoted as The influx rate of the new target is denoted as The intensity of exercise is recorded as . No. The perceptual semantic load of the image acquisition channel is:
[0048] Equation (3)
[0049] in, The upper bound of the target quantity is normalized. For the load component weights, and , .
[0050] New target inflow rate By statistics window The first new target number in the country Total number of different targets within this window Calculation of the ratio:
[0051] Equation (4)
[0052] Intensity of exercise From the statistics window Calculation of effective tracking targets within the window. For each effective tracking target, calculate the displacement magnitude of the center pixel coordinates of adjacent frames, divide by the frame interval, average all effective tracking values within the window, and then divide by the reference velocity. And cut off to If there is no valid target to track within the window, retrieve... This definition characterizes the influx of new targets and the rapid movement of existing targets, with both quantities taking values of [value missing]. .
[0053] In a further embodiment, the system calculates the global perceptual semantic load based on the priority weights of the image acquisition channels:
[0054] Equation (5)
[0055] in, ,and Priority weights can be set based on the importance of the image acquisition channel, installation location, or business requirements. The globally aware semantic load is used to determine a subset of candidate execution schemes, while the channel-aware semantic load is used for service quality arbitration within that subset.
[0056] Example 4: Dynamic Power Budget
[0057] In one embodiment, the system is based on the current junction temperature. The power consumption budget is dynamically adjusted. The dynamic power consumption budget is:
[0058] Equation (6)
[0059] in, , For rated power consumption budget, This is the minimum power consumption limit. The starting temperature for frequency limiting protection, This is the upper limit of the junction temperature. The system will measure the current power consumption. and The comparison is used to determine whether the current execution plan violates the hard power constraint, and the estimated power consumption of the candidate execution plan is compared with... The feasibility of candidate execution plans is determined by comparison.
[0060] Example 5: Joint Arbitration of Heterogeneous Execution Schemes and Service Quality Configuration
[0061] In one embodiment, the pre-compiled heterogeneous execution scheme library includes multiple offline-generated execution schemes. Each execution scheme may specify a mapping relationship between at least one of trigger detection, confirmation detection, feature extraction, fusion processing, or post-processing and at least two types of heterogeneous computing units among NPU, GPU, DSP, and CPU.
[0062] During offline generation, firstly, executability, latency, and power consumption benchmark tests are performed on each heterogeneous computing unit according to the perceptron operator type, tensor size, and precision level; then, the computation graph is divided into a finite number of partitions based on the tensor interface and the transmission cost between devices, forming a finite number of computation graph mappings and discrete precision level combinations.
[0063] For each combination, the compiler or graph optimization tool for the corresponding computational unit is invoked to generate a complete compiled artifact that can be directly loaded. Before being included in the scheme library, the name, shape, data type, and storage layout of the interval input and output tensors are verified item by item, the ownership and lifetime of the buffers are verified, and the necessary precision conversion nodes are verified.
[0064] Each scheme's metadata includes at least a scheme identifier, a mapping between computation graph partitions and computational units, available precision profiles, input / output tensor specifications, buffer configuration, applicable load range, and cost model index. Upon device startup, all or frequently used compilation artifacts are loaded, execution handles are established, and buffers are pre-allocated. During runtime, switching is completed through atomic replacement of execution handles, input / output routes, and buffer configurations, without requiring a re-invocation of the compiler.
[0065] For example, a pre-compiled heterogeneous execution scheme library may include: Table 1 plan Handling method Applicable status Option A The DSP or CPU performs trigger detection and preprocessing, while the NPU performs a small number of confirmation detections with low precision. Low load, low power consumption state Option B The NPU performs main inference, while the CPU / DSP performs preprocessing and post-processing. Medium load, balanced state Option C NPU and GPU share the detection and critical branch, while CPU / DSP performs preprocessing and postprocessing. High load, target influx state Option D Lowest power consumption scheme, reducing secondary channel QoS configuration High temperature or power-limited state
[0066] For the The image acquisition channel's quality of service configuration is represented as follows: ,in , and These are discrete resolution settings, frame rate settings, and precision settings. Each setting is then normalized to... , and The perceived quality gain is:
[0067] Equation (7)
[0068] in, ,and The sets of service quality levels are all discrete finite sets, which can be established based on the input size supported by the detection network, the frame rate allowed by the device, and the precision level supported by the compiler.
[0069] The system first assesses the semantic load globally. Given the predefined load range, a subset of suitable candidate solutions is selected from the solution library; then, the Cartesian product of the candidate solution subset and each limited quality of service level is enumerated. For each pre-compiled heterogeneous execution solution... The system calculates net utility:
[0070] Equation (8)
[0071] The constraints are:
[0072] Equation (9)
[0073] in, and These represent the estimated frame delay and estimated power consumption given by the cost model, respectively. During enumeration, combinations that do not meet the delay budget or dynamic power consumption budget are first excluded, and then the net utility of feasible combinations is compared. When the net utility of multiple combinations is the same, the combination with the lower estimated power consumption and the lower switching cost is selected in sequence, thus giving the finite optimization process a deterministic output.
[0074] The system further factores in the cost of switching between execution plans to determine the candidate execution plans:
[0075] Equation (10)
[0076] And calculate the switching gain:
[0077] Equation (11)
[0078] The cost of switching solutions is determined by offline switching tests and runtime monitoring results, and is expressed as:
[0079] Equation (12)
[0080] in, To normalize the preheating time, To normalize the amount of state transition data, To normalize cache or context reconstruction latency, and If the target solution has been preheated and the current buffer can be reused, the corresponding component is set to 0.
[0081] Example 6: Switching between gating and protection switching
[0082] In one embodiment, the system performs normal switching gating while the current execution scheme still meets hard constraints on power consumption and latency. If the switching gain... Greater than the hysteresis threshold And the current execution plan has a continuous number of resident frames. Greater than or equal to the minimum number of resident frames The system switches to the candidate execution plan. Otherwise, the system will continue with the current execution plan.
[0083] When the current execution scheme violates hard power or hard latency constraints, the system bypasses the hysteresis threshold and minimum dwell frame limit, and performs a protection switch. If a candidate execution scheme that meets the dynamic power and latency budget exists, the system switches to the candidate execution scheme that meets the constraints and has the highest net utility. If no candidate execution scheme meets the constraints, the system switches to the lowest power execution scheme and reduces the frame rate, resolution, or precision of the secondary image acquisition channels according to channel priority.
[0084] Example 7: Cost Model Update
[0085] In one embodiment, offline execution scheme is followed. Service quality configuration The cost model units are divided into candidate sensing zone area ratio intervals and target quantity intervals. Multiple benchmark tests are performed on each measurable combination, and the preset upper quantile statistics or maximum values of frame latency and power consumption are written into the corresponding unit as conservative initial values. For units not directly measured, interpolation between adjacent units is used; if adjacent data is insufficient, a conservative upper bound of the same execution scheme is used.
[0086] The runtime monitoring and feedback module collects the current execution plan, current service quality configuration, candidate sensing zone area ratio, target quantity, measured frame latency, measured power consumption, and plan switching cost, and uses this data as samples to update the cost model. For the lookup model, the model unit is located based on the current sample's plan, service quality level, and load range. For regression models, the same fields are used as input features.
[0087] The cost model can be either a lookup table model or a regression model. If a lookup table model is used, the model units can be determined according to the execution plan, service quality configuration, candidate sensing band area ratio range, and target quantity range. And perform an exponential moving average update on the model unit:
[0088] Equation (13)
[0089] in, This allows for online step size updates. In this way, the cost model can adaptively update based on the deployment environment, cooling conditions, and actual input scenarios.
[0090] Example 8: Edge Device Deployment Example
[0091] In one embodiment, the edge computing device is a board or edge box equipped with at least two types of heterogeneous computing units selected from NPU, GPU, DSP, and CPU. The edge computing device can connect to three partially overlapping field-of-view cameras and run the method described in this invention.
[0092] In one specific example, the edge computing device may employ an edge computing board with heterogeneous computing capabilities. The system can acquire multiple image data streams via an onboard camera interface, USB camera, or webcam; read temperature and power consumption information through a system interface; and invoke an NPU, GPU, or CPU to execute inference tasks in different scenarios through a pre-deployed inference framework. This specific board model is merely illustrative and does not constitute a limitation on the scope of protection of this invention.
[0093] In low-load scenarios, the system can adopt a low-power execution scheme and reduce the resolution or frame rate of some image acquisition channels; in scenarios with a surge of targets, the system can switch to a high-load execution scheme with NPU and GPU sharing the load and improve the quality of service configuration of relevant image acquisition channels; in scenarios with rising junction temperature, the system can tighten the dynamic power budget and prioritize the quality of service of high-priority image acquisition channels.
Claims
1. A multi-camera collaborative sensing method for heterogeneous computing power, applied to an edge computing device, the edge computing device comprising: Image acquisition channels and at least two types of heterogeneous computing units, Its characteristics are, The method includes: Obtain the The current frame group of the image acquisition channel; Perform low-resolution or low-frequency trigger detection on at least one image acquisition channel to obtain the region of interest in the corresponding image acquisition channel; For a target image acquisition channel that has a visual overlap relationship with the source image acquisition channel, the region of interest in the source image acquisition channel is mapped to the target image acquisition channel according to the pre-calibrated ground plane homography relationship, and a candidate sensing band of the target image acquisition channel is generated based on the mapping result. Confirmation detection is performed within the candidate sensing band, and independent detection is performed in non-overlapping areas to obtain the detection results of each image acquisition channel; The perceptual semantic load of each image acquisition channel is calculated based on the proportion of candidate sensing zone area, number of targets, influx rate of new targets, and intensity of motion of each image acquisition channel. Read the current junction temperature and current power consumption of the edge computing device, determine the dynamic power consumption budget based on the current junction temperature, and determine the power consumption hard constraint based on the current power consumption and the dynamic power consumption budget; Based on the perceptual semantic load, the dynamic power consumption budget, the latency budget, and the pre-compiled heterogeneous execution scheme library, candidate execution schemes and candidate service quality configurations are jointly determined from multiple pre-compiled heterogeneous execution schemes and the service quality configurations of each image acquisition channel. The selection of the candidate execution schemes and the arbitration of the service quality configurations of each image acquisition channel are jointly driven by the same perceptual semantic load. Each execution scheme in the pre-compiled heterogeneous execution scheme library is pre-compiled offline. The service quality configuration includes at least two of the following: resolution, frame rate, and discrete precision. Based on the switching cost, hysteresis threshold, and minimum number of resident frames, the candidate execution schemes are subject to switching gating to suppress frequent switching caused by load fluctuations; when the current execution scheme violates the hard power constraint or hard latency constraint, the switching gating is bypassed and the scheme is switched to a feasible execution scheme. According to the final determined execution scheme and quality of service configuration, inference is performed on the candidate sensing band or detection region on the heterogeneous computing unit, the collaborative sensing result is output, and the cost model used to estimate the frame latency and power consumption of each execution scheme is updated based on the measured frame latency and measured power consumption.
2. The method according to claim 1, characterized in that, The step of mapping the region of interest in the source image acquisition channel to the target image acquisition channel and generating candidate sensory bands includes: For each image acquisition channel Establish overlapping adjacency sets and only for those belonging to the overlapping adjacency set. The source image acquisition channel performs cross-view mapping; Source image acquisition channel Homogeneous coordinates of vertices in the region of interest Through the homography matrix of the ground plane Mapped to target image acquisition channel Then perform homogeneous normalization to obtain pixel coordinates. : target image acquisition channel The region of interest for self-checking this path and the overlapping adjacency set Find the union of the mapped regions and follow the tolerance. Morphological dilation is performed to obtain candidate receptive bands. : in, Image acquisition channel The region of interest obtained from this route's self-inspection For the reason Defined region mapping operator.
3. The method according to claim 1, characterized in that, Performing independent detection on non-overlapping regions includes: performing trigger detection on non-overlapping regions of each image acquisition channel at low frequency or low resolution; when the trigger detection hits the target, upgrading the corresponding non-overlapping region to a local confirmation detection region; when the scene state does not change significantly, reusing the candidate sensing band of the previous frame group or several previous frame groups, and re-performing the trigger detection according to a preset sentinel cycle.
4. The method according to claim 1, characterized in that, Perceptual semantic load of each image acquisition channel Calculate as follows: in, This represents the percentage of candidate sensing zone area. For the target quantity, The upper bound of the target quantity is normalized. For statistics window The ratio of the number of new targets appearing for the first time within a window to the total number of different targets within that window, with the denominator being at least 1 and the result limited to a certain range. ; For statistics window The average pixel displacement magnitude of each tracked target per unit time divided by the reference velocity Later cut off to If there is no effective target to track, the value is 0; ,and ; And based on the priority weight of each image acquisition channel Compute global-aware semantic load : in, ,and .
5. The method according to claim 1, characterized in that, The process of determining the dynamic power consumption budget based on the current junction temperature includes: in, , ,and The actual power consumption will be measured. and The comparison is used to determine whether the current execution plan violates the hard power constraint, and the estimated power consumption of the candidate execution plan is compared with... The feasibility of candidate execution plans is determined by comparison.
6. The method according to claim 1, characterized in that, The joint determination of candidate execution schemes and candidate quality of service configurations includes: Based on globally perceived semantic load Within the preset load range, a subset of candidate solutions is selected from the pre-compiled heterogeneous execution solution library; The first The quality of service configuration of a road is represented as , representing discrete resolution, frame rate, and precision levels respectively, and the perceived quality gain is calculated using normalized level values weighted with non-negative weights: Enumerate the Cartesian products of the subset of candidate solutions and each finite quality of service level, and calculate the load-weighted net utility under the constraints of latency budget and dynamic power budget: The switching cost is defined as a normalized weighted sum of warm-up latency, state transition amount, and cache or context rebuild overhead, and is included in the net utility of the candidate execution scheme; when multiple combinations have the same net utility, the combination with the lower estimated power consumption and the lower switching cost is selected in turn.
7. The method according to claim 6, characterized in that, The switching gating of the candidate execution schemes includes: when the current execution scheme satisfies the hard constraints on power consumption and latency, if the switching gain... Greater than the hysteresis threshold And the number of consecutive resident frames in the current execution scheme Greater than or equal to the minimum number of resident frames If the current execution scheme fails, the current execution scheme is used; otherwise, the current execution scheme is used. If the current execution scheme violates the hard power or hard latency constraints, the hysteresis threshold and minimum dwell frame limit are bypassed, and the candidate execution scheme that meets the budget and has the highest net utility is switched to. If there is no candidate execution scheme that meets the budget, the lowest power consumption execution scheme is switched to, and the frame rate, resolution or precision of the secondary image acquisition channel is reduced according to the priority weight of the image acquisition channel.
8. The method according to claim 1, characterized in that, The pre-compiled heterogeneous execution scheme library is established and used in the following manner: offline testing of the executability, latency, and power consumption of operators or computation graph partitions of the perceptron on at least two types of heterogeneous computing units, including NPU, GPU, DSP, and CPU; based on the test results, a finite number of computation graph mappings and discrete precision file combinations are formed, a complete executable compilation product is generated for each combination, and the consistency of input / output tensor interfaces, buffer ownership, storage layout, and precision conversion between partitions is verified; at startup, the compilation product and its metadata are loaded into memory and an execution handle is established; at runtime, the execution scheme is switched by switching the execution handle and the corresponding buffer configuration without recompiling the computation graph.
9. The method according to claim 1, characterized in that, The cost model is initialized and updated in the following way: offline, multiple benchmark tests are performed on combinations of execution scheme, quality of service configuration, candidate sensing band area ratio range and target number range. The hit model units are initialized with conservative statistics of measured frame latency and measured power consumption. For the missing units, the interpolation value of adjacent units or the upper bound value is used. At runtime, the current execution scheme, current quality of service configuration, current candidate sensing band area ratio, current target number, measured frame latency and measured power consumption are used as samples to perform exponential moving average updates on the hit cost model units.
10. A multi-camera collaborative sensing system for heterogeneous computing power, characterized in that, include: Multi-channel image acquisition layer, used to acquire The current frame group of the image acquisition channel; The trigger detection module is used to perform low-resolution or low-frequency trigger detection on at least one image acquisition channel to obtain the region of interest. The cross-view propagation and candidate sensing band module is used to generate candidate sensing bands for image acquisition channels with overlapping view relationships, based on the pre-calibrated homography relationship of the ground plane. The confirmation and supplementary detection module is used to perform confirmation detection within the candidate sensing band and to perform independent detection for non-overlapping areas. The perceptual semantic load estimation module is used to calculate the perceptual semantic load of each image acquisition channel based on the proportion of candidate perceptual zone area, number of targets, influx rate of new targets, and intensity of motion of each image acquisition channel. The junction temperature power consumption budget module is used to read the current junction temperature and current power consumption, determine the dynamic power consumption budget based on the current junction temperature, and determine the power consumption hard constraint based on the current power consumption and the dynamic power consumption budget. A pre-compiled heterogeneous execution scheme library is used to store a limited number of complete executable schemes and their metadata that are pre-compiled offline; The scheme selection and service quality budget arbitration module is used to jointly determine candidate execution schemes and candidate service quality configurations for each image acquisition channel under the constraints of dynamic power consumption budget and latency budget, using the perceived semantic load as a unified scheduling signal. The switching gating module is used to perform switching gating on the candidate execution schemes based on the scheme switching cost, hysteresis threshold and minimum number of resident frames, and to perform protective switching when the current execution scheme violates the power consumption hard constraint or the latency hard constraint. The heterogeneous execution layer includes at least two types of heterogeneous computing units such as NPU, GPU, DSP, and CPU, which are used to perform inference and output collaborative perception results according to the finally determined execution scheme and quality of service configuration. The runtime monitoring and feedback module is used to update the cost model based on the measured frame latency and measured power consumption.
11. An electronic device, characterized in that, The method includes a memory, a processor, and at least two types of heterogeneous computing units, wherein the memory stores a computer program, and the processor executes the computer program to implement the method according to any one of claims 1 to 9.
12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 9.
