A closed-loop ROI scheduling method for visual detection of a resource-constrained embedded platform

Abstract

The application discloses a closed-loop ROI scheduling method for visual detection of a resource-limited embedded platform, and comprises the following steps: step 1, obtaining an image frame collected by a camera as to-be-processed data; step 2, index reading and closed-loop control updating; step 3, predictive ROI tracking and priori ROI reliability calculation; step 4, multiplexing judgment; step 5, coarse detection mode selection; step 6, candidate ROI generation; step 7, ROI candidate pool construction and screening; step 8, ROI effectiveness check and hierarchical fallback; step 9, ROI cropping and cache multiplexing preparation; step 10, performing high-precision hierarchical detection on the ROI fine detection input data constructed in step 9 to obtain a current frame detection result; and step 11, outputting the current frame detection result, updating an ROI state cache, an operation index and a scheduling state, and entering a closed-loop ROI scheduling process of a next scheduling cycle. The application improves the stable operation capability of multiple visual tasks on an embedded platform.

Description

Technical Field

[0001] This invention belongs to the field of embedded vision processing and target detection technology, specifically relating to a closed-loop ROI scheduling method for vision detection on resource-constrained embedded platforms. Background Technology

[0002] With the rapid development of embedded vision processing technology, vision tasks such as color recognition, marker recognition, and lightweight AI detection have been widely applied in robot control, educational platforms, and industrial inspection. These scenarios typically require systems to perform real-time detection and result output on resource-constrained hardware platforms, thus placing high demands on processing latency, CPU utilization, operational stability, and detection quality.

[0003] In practical applications, although embedded vision systems typically perform only a single detection task at a time, limited processor computing power, storage bandwidth, and on-chip cache resources mean that performing high-frequency full-frame detection on every frame can easily lead to increased CPU utilization, increased end-to-end latency, and longer single-frame processing time, thus affecting the system's real-time response capability. This is especially true when the target position changes little or the scene remains stable for extended periods; continuous full-frame detection results in a large amount of repetitive computation, causing unnecessary resource consumption.

[0004] To reduce computational overhead, one existing strategy is to first identify candidate regions through low-cost detection, and then perform high-precision detection in those local regions. For example, Faster R-CNN is a typical two-stage object detection method. This method generates candidate regions through a Region Proposal Network (RPN), and then a subsequent detection network classifies and regresses the candidate regions, thereby reducing the computational overhead of the traditional candidate region generation process.

[0005] However, the aforementioned methods primarily generate candidate regions from the perspective of the target detection network structure. Their detection process typically follows a fixed model structure and inference flow, failing to establish a closed-loop feedback adjustment mechanism for end-to-end latency, CPU utilization, ROI hit rate, detection effectiveness, and backoff triggering during operation on resource-constrained embedded platforms. Furthermore, they do not dynamically switch between different detection modes such as skipping coarse detection, partial coarse detection, and full-frame coarse detection based on target stability and candidate region reliability. Therefore, on resource-constrained platforms, when the target position remains stable for extended periods, system load fluctuates, or ROI prediction fails, it remains difficult to simultaneously achieve real-time performance, detection quality, and resource utilization efficiency.

[0006] Therefore, there is an urgent need for a visual inspection method suitable for resource-constrained embedded platforms that can perform closed-loop ROI scheduling by combining system operating indicators, target stability, and candidate region reliability, thereby reducing redundant computation and resource consumption while ensuring detection quality. Summary of the Invention

[0007] This invention provides a closed-loop ROI scheduling method for visual inspection on resource-constrained embedded platforms. It quickly locates candidate regions through sparse sampling, combines ROI tracking and hierarchical fine inspection strategies, and introduces an indicator feedback and dynamic scheduling mechanism. Under end-to-end latency, CPU usage and other end-side constraints, it maintains detection quality and real-time performance, thereby improving the stable operation of various visual tasks on embedded platforms.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A closed-loop ROI scheduling method for visual inspection on resource-constrained embedded platforms includes the following steps: Step 1, Input Frame Acquisition: Acquire target scene images or videos containing the target to be detected from the camera, and use the image frames in the images or videos as data to be processed; Step 2, Indicator Reading and Closed-Loop Control Update: Read the running metrics of the previous scheduling cycle, where the previous scheduling cycle is the statistical cycle that has been completed before the current frame is processed; when the resource-constrained embedded vision inspection system updates frame by frame, the previous scheduling cycle corresponds to the previous frame; when the resource-constrained embedded vision inspection system uses window statistics, the previous scheduling cycle corresponds to the statistical window of the previous K frames. The operational metrics include at least the end-to-end latency, CPU utilization, ROI hit rate, detection effectiveness, and rollback trigger statistics of the previous scheduling cycle.

[0009] Based on the above operating indicators and the preset latency threshold, CPU utilization threshold and detection quality constraints, the scheduling control quantity of the current frame is updated; The scheduling control parameters include coarse detection step size, coarse detection interval, fine detection interval, ROI hold frame count, ROI expansion parameters, and global calibration interval; and the ROI state cache is obtained.

[0010] When the ROI state cache is empty or the current frame is the first frame, mark the current ROI state as invalid and select the coarse detection mode in step 3; Otherwise, predict the position and size of the ROI in the current frame based on the position and size of the ROI in the previous frame to obtain the predicted ROI of the current frame; when the predicted ROI satisfies the image boundary constraints and the minimum area constraints, it is determined as a valid predicted ROI; and combine the detection confidence, tracking consistency, hit statistics and consecutive loss count of the previous frame to calculate the prior ROI reliability of the current frame. Step 3, Reuse determination: Determine whether the detection result of the previous frame is reused in the current frame; If not reused, proceed to the coarse inspection mode selection process; select the coarse inspection mode for the current frame based on the prior ROI reliability, consecutive loss count, and the number of frames since the last global calibration. The coarse inspection modes include skip coarse inspection mode, local coarse inspection mode, and full frame coarse inspection mode.

[0011] Step 4, Candidate ROI generation: A candidate ROI set is generated according to the selected coarse detection mode; when the skip coarse detection mode is selected, the obtained effective predicted ROIs are used as candidate ROIs; when the local coarse detection mode or the full frame coarse detection mode is selected, sparse sampling or resolution-layered coarse detection is performed, and a candidate ROI set is generated. Among them, sparse sampling coarse detection is used to reduce the scanning density by increasing the sampling step size, and resolution layered coarse detection is used to quickly obtain candidate regions by reducing the resolution of the input image. The system selects one or a combination of them based on the current CPU utilization, end-to-end latency and target scale changes.

[0012] Step 5, ROI candidate pool construction and screening: The candidate ROI set and the effective predicted ROIs obtained in step 2 are merged, sorted, and budgeted to determine the ROIs to be further examined. Step 6, ROI validity check and tiered rollback: For ROIs to be finely inspected, geometric boundaries, ROI reliability, and budget constraints are checked to obtain valid ROIs. If the check fails, a tiered rollback is performed.

[0013] Step 7, ROI trimming and cache reuse preparation: For confirmed valid ROIs, trim them or create ROI view mappings, and use a cache reuse method to construct ROI fine-tuning input data.

[0014] Step 8, Stratified detailed inspection within the ROI: High-precision layered detection is performed on the ROI fine detection input data to obtain the detection result of the current frame; It then updates the ROI status cache, running metrics, and scheduling status, and enters the closed-loop ROI scheduling process for the next scheduling cycle.

[0015] The specific process of obtaining the image frame in step 1 is as follows: The OV2640 camera is initialized and configured through the target hardware platform. The initialization configuration includes at least clock configuration, resolution configuration, output format configuration, and frame buffer enable configuration. After initialization, the OV2640 camera acquires images of the target to be detected in the target scene, obtains the current frame image data containing the target to be detected, and records the frame number and acquisition timestamp of the current frame as input data for the subsequent closed-loop ROI scheduling process.

[0016] The specific process of reading indicators and updating closed-loop control in step 2 is as follows: Read the runtime metric vector from the previous scheduling cycle: The scheduling parameters are updated under preset objectives and constraints, and the optimization objective is expressed as follows: The constraints are satisfied: In the formula, The end-to-end delay of the previous scheduling cycle, This represents the CPU utilization rate in the previous scheduling cycle. The ROI hit rate of the previous scheduling cycle. This is the detection efficiency of the previous scheduling cycle. Statistics on rollback triggers from the previous scheduling cycle; For the time delay threshold, CPU utilization threshold This is the ROI hit rate threshold. To detect the effectiveness threshold, The fallback trigger threshold, α, β, γ, δ, η These are the weighting coefficients; Based on the above objectives and constraints, update the scheduling control variables for the current frame. : in, coarse_stride For the current frame coarse detection step size, coarse_interval For coarse inspection interval, fine inspection interval refine_interval To simplify intervals, roi_hold Maintain frame rate for ROI roi_pad For ROI edge expansion parameters, global_ calib_interval_t For global calibration interval, roi_min_hits_t The minimum number of consecutive hits required for an ROI to reach a steady state.

[0017] The specific process of step 2 is as follows: Step 2.1: Read the ROI state cache updated in the previous scheduling cycle. The ROI state cache includes at least the position, size, detection confidence, consecutive hit statistics, consecutive loss count, and detection results of the ROI in the previous frame. When the ROI state cache is empty or the current frame is the first frame, the current ROI state is marked as invalid, and the full-frame coarse detection mode is selected in step 3; when the ROI state cache is valid, the position and size of the ROI in the current frame are predicted based on the position and size of the ROI in the previous frame, and the predicted ROI is obtained, denoted as . ; Step 2.2: Based on the detection confidence, tracking consistency, consecutive hit statistics, and consecutive loss count of the previous frame, calculate the prior ROI reliability of the current frame. This is used for subsequent reuse determination and coarse inspection mode selection; The prior reliability of the current frame is represented as: In the formula, The prior ROI reliability for the current frame. The confidence level for detection or fine-tuning in the previous frame. To track consistency metrics, For consecutive hit statistics, For continuous loss counting, These are preset weighting coefficients.

[0018] In step 3, the reliability of the prior ROI of the current frame is calculated. Consecutive hit statistics read from the ROI state cache updated in the previous scheduling cycle. and consecutive loss count The number of times the detection results read from the scheduling state updated in the previous scheduling cycle are reused. And the ROI retention frame count obtained from closed-loop control updates. and global calibration interval To determine whether the detection result of the previous frame is reused in the current frame, the rules are as follows: The ROI hold frame count is used to limit the maximum number of frames that can be continuously reused for detection results; the global calibration interval is used to limit the maximum interval at which the system forces local coarse detection or full-frame coarse detection; when the prior ROI reliability is higher than the reuse threshold, the consecutive loss count is less than the loss threshold, the number of times the detection result is reused does not exceed the ROI hold frame count, and the number of frames since the last global calibration does not exceed the global calibration interval, the previous frame detection result is reused; otherwise, the coarse detection mode selection is entered.

[0019] The specific process of the reuse determination in step 3 is as follows: Based on prior reliability Consecutive hit statistics Continuous loss count Number of times test results can be reused and the number of ROI hold frames determined in step 2. and global calibration interval global_calib_interval_t The determination of whether the current frame reuses the detection result of the previous frame can be expressed as follows: In the formula, This is the result of the current frame reuse determination. The reuse threshold, The missing data count threshold, This is the number of frames since the last global calibration. This is the global calibration interval; when When the current frame does not reuse the detection results of the previous frame, proceed to step 3 to select the coarse detection mode; when When the current frame reuses the detection result of the previous frame as the detection result of the current frame, it skips the subsequent candidate ROI generation, ROI candidate pool construction and screening, ROI validity check, ROI pruning and ROI internal layer fine detection steps, and proceeds to step 8 for output and status update.

[0020] When determining that the detection result of the previous frame should not be reused in the current frame, the reliability of the ROI in the current frame is used as a basis. Continuous loss count and the number of frames since the last global calibration. The coarse detection mode for the current frame is selected, and its discriminant relationship is expressed as follows: In the formula, Select the result for the coarse detection mode of the current frame. and These are the high reliability threshold and the low reliability threshold, respectively. when When the reliability of the prior ROI in the current frame is high and the target state is stable, the coarse detection mode is skipped, and the effective predicted ROI obtained in step 3 is used as the candidate ROI, and then the process proceeds to step 5. when When the reliability of the prior ROI in the current frame is in the medium range, the local coarse detection mode is selected, and coarse detection is performed in the local area around the predicted ROI, and then the process proceeds to step 4 to generate a set of candidate ROIs. when When the reliability of the prior ROI in the current frame is low, the target is continuously lost, or the global calibration condition is reached, the full-frame coarse inspection mode is selected, and coarse inspection is performed in the entire map of the current frame, and then the process proceeds to step 4 to generate a candidate ROI set.

[0021] The specific process for generating candidate ROIs in step 4 is as follows: When step 3 selects to skip the coarse detection mode, the coarse detection calculation is not performed, and the obtained effective predicted ROI is used as a candidate ROI and added to the candidate ROI set of the current frame. When the local coarse detection mode is selected in step 3, the effective predicted ROI is used as the center, and the ROI expansion parameters are used to determine the expansion parameters. roi_pad Construct a local search region, and within the local search region, perform a coarse detection step size. coarse_stride Alternatively, a resolution-layered approach can be used to perform fast detection and generate a set of candidate ROIs; When step 3 selects the full-frame coarse inspection mode, the coarse inspection step size is applied to the entire current frame. coarse_stride Alternatively, a resolution-layered approach can be used to perform fast detection, generating a candidate ROI set; this candidate ROI set is denoted as... And record the coarse confidence score for each candidate ROI. Candidate area And position parameters.

[0022] The specific process of constructing and screening the ROI candidate pool in step 5 is as follows: Step 5.1: Construct a candidate pool based on the candidate ROI set generated in Step 4; when there is a valid predicted ROI in Step 3 and the valid predicted ROI is not yet included in the candidate ROI set generated in Step 4, merge the valid predicted ROI into the candidate pool, and perform overlap merging, deduplication and sorting on the ROIs in the candidate pool. Step 5.2: For each candidate ROI in the candidate pool, its comprehensive score is expressed as: In the formula, For the first The overall score of each candidate ROI. As a measure of consistency between candidate ROIs and effective predicted ROIs, This is an area penalty item. For the lost penalty item, These are weighting coefficients; when the candidate ROI comes from a valid predicted ROI and no coarse check is performed, It can be replaced by the confidence level detected in the previous frame or the prior ROI reliability in the current frame.

[0023] Under the constraint of the detailed inspection budget, the candidate ROI with the highest score or the score that meets the preset threshold is selected as the ROI to be detailed inspection.

[0024] The specific process of step 6, ROI validity check and graded rollback, is as follows: Step 6.1: Perform detailed inspection on the samples identified in Step 5. A validity check is performed, which includes ROI geometric boundary check, ROI reliability check, and budget constraint check. The validity determination relationship can be expressed as follows: In the formula, This is the result of determining the validity of the ROI to be further examined. The area of ​​ROI to be precisely inspected. and These are the minimum and maximum area thresholds for the ROI, respectively. The current frame image boundary region, The overall score of the ROI to be refined. As the validity threshold, For indicator functions; Step 6.2: When At that time, Once confirmed as a valid ROI, proceed to step 7; when When a tiered rollback is triggered, the rollback level is represented as follows: In the formula, The current frame's backoff level. This is a slight correction and regression. For partial rescan and rollback, To recapture the entire frame, The low reliability failure threshold, and < ; when = When increasing the ROI expansion parameter, roi_pad Alternatively, adjust the ROI geometric boundaries and re-check the validity. when = If necessary, expand the local coarse inspection area and return to step 4 to perform local coarse inspection and candidate ROI generation; when = If the condition is not met, the system will force a switch to full-frame coarse detection mode and return to step 3 to reselect the coarse detection mode.

[0025] The specific process of step 7, ROI pruning and cache reuse preparation, is as follows: Step 7.1: Crop the valid ROIs confirmed in Step 6 on the current frame image, or establish the corresponding ROI view mapping to obtain ROI image blocks or ROI mapping regions; Step 7.2: To reduce the overhead of frequent memory allocation and copying, a cache reuse method is used to construct the ROI fine-tuning input data. The cache reuse method includes at least one or more of the following: pre-allocated cache, double buffering, circular reuse, or zero-copy mapping operation.

[0026] The specific process of step 8, ROI stratification and fine-tuning, is as follows: Perform hierarchical fine detection on the ROI fine detection input data constructed in step 7, and perform high-precision detection only in the confirmed valid ROI areas to obtain the detection result of the current frame; update the consecutive hit statistics and consecutive loss count according to the detection result of the current frame.

[0027] The specific process of outputting and updating the state in step 8 is as follows: Output the detection results of the current frame, and update the running metrics, ROI status cache, and scheduling status; The operational metrics include at least one or more of the following: end-to-end latency, single-frame processing time, CPU utilization, ROI hit rate, detection effectiveness, fallback trigger statistics, and reuse trigger statistics. The ROI state cache includes at least the position, size, detection confidence, consecutive hit statistics, consecutive loss count, and detection result of the current frame ROI; The scheduling status includes at least the number of times the detection result is reused, the number of frames since the last global calibration, and the current frame backoff level. The updated operating metrics serve as the feedback basis for the closed-loop control update in step 2 of the next scheduling cycle, and the updated ROI state cache serves as the input basis for predictive ROI tracking and prior ROI reliability calculation in step 2 of the next scheduling cycle, thus forming a closed-loop ROI scheduling process across scheduling cycles.

[0028] The beneficial effects of this invention are: 1. This invention proposes a closed-loop ROI scheduling method for embedded single-task visual inspection. It uses operational metrics such as end-to-end latency, CPU utilization, ROI hit rate, detection effectiveness, and backoff trigger statistics as feedback variables to dynamically adjust scheduling control variables such as coarse detection step size, coarse detection interval, fine detection interval, ROI hold frame count, ROI expansion parameters, and global calibration interval. This transforms the detection process from being driven by fixed parameters to closed-loop control based on operational status feedback and resource constraints. Compared to existing schemes that rely on experience-based parameter tuning or fixed detection frequencies, this invention can balance detection quality and real-time performance while meeting edge-side resource constraints.

[0029] 2. This invention improves the timing efficiency of a single-task serial detection process by using predictive ROI tracking, prior ROI reliability calculation, reuse judgment, and coarse detection mode selection. The system can predict the current frame ROI and calculate the prior ROI reliability based on the ROI state cache updated in the previous scheduling cycle. When the target state is stable and meets the reuse conditions, the detection result of the previous frame can be directly reused and the system can proceed to output and state update. When the reuse conditions are not met but the predicted ROI is still relatively reliable, the coarse detection mode can be skipped, and the effective predicted ROI can be used as a candidate ROI. When the target state is unstable, continuously lost, or reaches the global calibration condition, the system automatically switches to local coarse detection or full-frame coarse detection. This reduces unnecessary high-frequency detection overhead across the entire frame, lowers end-to-end latency and its fluctuations, and improves the real-time processing capabilities on the embedded platform.

[0030] 3. This invention improves the reliability of the ROI scheduling process through ROI validity checks and a tiered backoff mechanism. Geometric boundaries, ROI reliability, and budget constraints are checked for ROIs to be precisely inspected. When an ROI slightly deviates, a mild correction backoff is performed; when a local candidate region is unreliable, a local rescan backoff is performed; and when an ROI's state is invalid or its reliability is too low, a full-frame recapture backoff is performed. This avoids the system relying on invalid ROIs for extended periods, ensuring the stability of the detection results.

[0031] 4. This invention reduces memory copying and bandwidth pressure by combining ROI pruning and cache reuse with in-ROI layered precision detection, thereby improving deployment efficiency on resource-constrained platforms. This invention performs high-precision layered detection only within confirmed valid ROI regions and uses pre-allocated cache, double buffering, circular reuse, or zero-copy mapping to construct ROI precision detection input data. This avoids the performance loss caused by full-frame high-frequency precision detection, frequent memory allocation and release, and large-scale data transfer, effectively reducing system jitter and improving long-term stable operation on MCUs, lightweight SoCs, and other edge embedded platforms. Attached Figure Description

[0032] Figure 1 This is a framework diagram of the closed-loop ROI scheduling system in an embodiment of the present invention.

[0033] Figure 2 This is a diagram of the coarse detection mode discrimination module in an embodiment of the present invention.

[0034] Figure 3 This is a diagram of the validity check and graded rollback module in an embodiment of the present invention.

[0035] Figure 4This refers to the ROI trimming and cache reuse and ROI fine-tuning module in this embodiment of the invention. Detailed Implementation

[0036] The present invention will now be described in further detail with reference to the accompanying drawings.

[0037] See Figure 1 A closed-loop ROI scheduling mechanism for visual inspection on resource-constrained embedded platforms includes the following steps: Step 1, Input Frame Acquisition: Acquire image frames captured by the camera as data to be processed; Step 2, Indicator Reading and Closed-Loop Control Update: The system reads the operational metrics from the previous scheduling cycle, which include at least end-to-end latency, CPU utilization, ROI hit rate, detection effectiveness, and backoff trigger statistics. Based on these operational metrics and preset latency thresholds, CPU utilization thresholds, and detection quality constraints, the system updates the scheduling control parameters for the current frame. These scheduling control parameters include coarse detection step size, coarse detection interval, fine detection interval, ROI hold frame count, ROI expansion parameters, and global calibration interval. Step 3, Predictive ROI tracking and prior ROI reliability calculation: Read the ROI state cache updated in the previous scheduling cycle. The ROI state cache includes at least the position, size, detection confidence, hit statistics, consecutive loss count, and detection results of the ROI in the previous frame. When the ROI state cache is empty or the current frame is the first frame, mark the current ROI state as invalid and select the full frame coarse detection mode in step 5.

[0038] Step 4, Reuse determination: Based on the prior ROI reliability, consecutive hit statistics, consecutive loss count, and number of times the detection result is reused obtained in step 3, as well as the ROI holding frame number and global calibration interval determined in step 2, it is determined whether the detection result of the previous frame is reused in the current frame; if the reuse condition is met, the detection result of the previous frame is reused and the process proceeds to step 11; otherwise, the process proceeds to step 5. Step 5, Select coarse inspection mode: When step 4 determines that the detection result of the previous frame is not reused in the current frame, the coarse detection mode selection process is entered. Based on the reliability of the prior ROI of the current frame, the consecutive loss count and the number of frames since the last global calibration calculated in step 3, the coarse detection mode of the current frame is selected. The coarse detection modes include skip coarse detection mode, local coarse detection mode and full frame coarse detection mode.

[0039] Step 6, Candidate ROI generation: Generate a candidate ROI set according to the coarse detection mode selected in step 5; when the skip coarse detection mode is selected, the effective predicted ROI obtained in step 3 is used as the candidate ROI; when the local coarse detection mode or the full frame coarse detection mode is selected, perform sparse sampling coarse detection or resolution hierarchical coarse detection, and generate a candidate ROI set. Step 7, ROI candidate pool construction and screening: The candidate ROI set generated in step 6 and the effective predicted ROI obtained in step 3 are merged, sorted and budgeted to determine the ROI to be further examined. Step 8, ROI validity check and tiered rollback: Perform geometric boundary, ROI reliability, and budget constraint checks on the ROI to be inspected, obtain the confirmed valid ROI, and perform tiered rollback when the check fails; Step 9, ROI trimming and cache reuse preparation: For the ROIs confirmed as valid in step 8, trim them or create ROI view mappings, and use a cache reuse method to construct ROI fine-tuning input data.

[0040] Step 10, Stratified detailed inspection within the ROI: High-precision layered detection is performed on the ROI fine detection input data constructed in step 9 to obtain the detection result of the current frame.

[0041] Step 11, Output and State Update: Output the detection result of the current frame, update the ROI status cache, running metrics and scheduling status, and enter the closed-loop ROI scheduling process of the next scheduling cycle.

[0042] See Figure 2 The specific process of steps 2 to 5 is as follows: Read the runtime metric vector from the previous scheduling cycle: The scheduling parameters are updated under preset objectives and constraints, and the optimization objective is expressed as follows: The constraints are satisfied: Based on the above objectives and constraints, update the scheduling control variables for the current frame. : Read the ROI state cache updated in the previous scheduling cycle. The ROI state cache includes at least the position, size, detection confidence, consecutive hit statistics, consecutive loss count, and detection results of the ROI in the previous frame. When the ROI state cache is empty or the current frame is the first frame, the current ROI state is marked as invalid, and the full-frame coarse detection mode is selected in step 5; when the ROI state cache is valid, the position and size of the ROI in the current frame are predicted based on the position and size of the ROI in the previous frame, and the predicted ROI is obtained, denoted as . .

[0043] Based on the detection confidence, tracking consistency, consecutive hit statistics, and consecutive loss count of the previous frame, calculate the prior ROI reliability of the current frame. This is used for the reuse determination in step 4 and the coarse inspection mode selection in step 5.

[0044] The prior reliability of the current frame is represented as: Based on prior reliability Consecutive hit statistics Continuous loss count Number of times test results can be reused and the number of ROI hold frames determined in step 2. and global calibration interval global_calib_interval_t The determination of whether the current frame reuses the detection result of the previous frame can be expressed as follows: when When the current frame does not reuse the detection results of the previous frame, proceed to step 5 to select the coarse detection mode; when When the current frame reuses the detection result of the previous frame as the detection result of the current frame, it skips the subsequent candidate ROI generation, ROI candidate pool construction and screening, ROI validity check, ROI pruning and ROI internal layer fine detection steps, and enters step 11 to output and update the status.

[0045] When determining in step 4 that the detection result of the previous frame is not reused in the current frame, the reliability of the ROI in the current frame is used as a basis. Continuous loss count and the number of frames since the last global calibration. The coarse detection mode for the current frame is selected, and its discriminant relationship can be expressed as follows: when When the reliability of the prior ROI in the current frame is high and the target state is stable, the coarse detection mode is skipped, and the effective predicted ROI obtained in step 3 is used as the candidate ROI, and then the process proceeds to step 7. when When the reliability of the prior ROI in the current frame is in the medium range, the local coarse detection mode is selected, and coarse detection is performed in the local area around the predicted ROI, and then the process proceeds to step 6 to generate a set of candidate ROIs. when When the reliability of the prior ROI in the current frame is low, the target is continuously lost, or the global calibration condition is reached, the full-frame coarse inspection mode is selected, and coarse inspection is performed in the entire map of the current frame, and then the process proceeds to step 6 to generate a candidate ROI set.

[0046] Through steps 2 to 5 above, this invention uses the ROI state cache updated in the previous scheduling cycle to predict the ROI of the current frame, and calculates the prior ROI reliability of the current frame based on detection confidence, tracking consistency, consecutive hit statistics and consecutive loss count, thereby determining whether the current frame can reuse the detection results of the previous frame, and when not reused, should select the skip coarse detection, partial coarse detection or full frame coarse detection mode.

[0047] The purpose of the above process is to adaptively select the detection path based on the target's stability and the system calibration status: when the target is stable and the ROI is reliable, redundant coarse detection and full-frame detection are reduced; when the target shows a slight shift, local coarse detection is performed only around the predicted ROI; when targets are continuously lost or the global calibration conditions are met, full-frame coarse detection is switched to recapture the target in a timely manner. This reduces end-to-end latency and CPU utilization, reduces redundant calculations caused by high-frequency full-frame detection, and avoids the system relying on failed ROIs for extended periods, thereby improving the real-time performance, stability, and detection reliability of vision inspection on resource-constrained embedded platforms.

[0048] See Figure 3 The specific process of steps 6 to 8 is as follows: When step 5 selects to skip the coarse detection mode, the coarse detection calculation is not performed. The effective predicted ROI obtained in step 3 is used as a candidate ROI and added to the candidate ROI set of the current frame. When the local coarse detection mode is selected in step 5, the effective predicted ROI obtained in step 3 is used as the center, and the ROI expansion parameters are used to determine the expansion parameters. roi_pad Construct a local search region, and within the local search region, perform a coarse detection step size. coarse_ stride Alternatively, a resolution-layered approach can be used to perform fast detection and generate a set of candidate ROIs; When the full-frame coarse inspection mode is selected in step 5, the coarse inspection step size is applied to the entire current frame. coarse_stride Alternatively, a resolution-layered approach can be used to perform fast detection, generating a candidate ROI set; this candidate ROI set is denoted as... And record the coarse confidence score for each candidate ROI. Candidate area And position parameters.

[0049] A candidate pool is constructed based on the candidate ROI set generated in step 6. When there is a valid predicted ROI in step 3 and the valid predicted ROI is not yet included in the candidate ROI set generated in step 6, the valid predicted ROI is merged into the candidate pool, and overlapping merging, deduplication, and sorting are performed on the ROIs in the candidate pool. For each candidate ROI in the candidate pool, its comprehensive score can be expressed as: Under the constraint of the detailed inspection budget, the candidate ROI with the highest score or the score that meets the preset threshold is selected as the ROI to be detailed inspection.

[0050] For the parts to be refined as determined in step 7 A validity check is performed, which includes ROI geometric boundary check, ROI reliability check, and budget constraint check. The validity determination relationship can be expressed as follows: when At that time, Once confirmed as a valid ROI, proceed to step 9; when When a tiered rollback is triggered, the rollback level can be represented as: when = When increasing the ROI expansion parameter, roi_pad Alternatively, adjust the ROI geometric boundaries and re-check the validity. when = If necessary, expand the local coarse inspection area and return to step 6 to perform local coarse inspection and candidate ROI generation; when = If the condition is not met, the system will force a switch to full-frame coarse detection mode and return to step 5 to reselect the coarse detection mode.

[0051] Through steps 6 to 8 above, this invention achieves a step-by-step screening and error correction mechanism from coarse detection mode to ROI to be finely detected. When the target is stable, this mechanism can reduce coarse detection calculations by effectively predicting ROIs; when the target is slightly offset, it can narrow the search range through local coarse detection; and when the target is lost or the ROI reliability is too low, it can recapture the target through full-frame coarse detection, thereby adaptively balancing detection accuracy and computational overhead under different target states.

[0052] Meanwhile, this invention introduces a comprehensive candidate pool score, a fine-check budget constraint, and an ROI validity check before candidate ROIs enter the fine-check phase. When a check fails, a tiered rollback is executed, enabling the system to promptly correct ROI offsets, expand local search areas, or trigger full-frame recapture. This reduces the probability of invalid ROIs entering the fine-check phase, minimizes unnecessary computational and memory consumption, and improves the real-time performance, stability, and reliability of the visual inspection process on resource-constrained embedded platforms.

[0053] See Figure 4 The specific process of steps 9 to 11 is as follows: For the ROIs confirmed as valid in step 8, crop them on the current frame image, or establish the corresponding ROI view mapping to obtain ROI image blocks or ROI mapping regions; To reduce the overhead of frequent memory allocation and copying, a cache reuse method is used to construct the ROI fine-tuning input data. The cache reuse method includes at least one or more of the following: pre-allocated cache, double buffering, circular reuse, or zero-copy mapping operation.

[0054] Layered precision detection is performed on the ROI precision detection input data constructed in step 9. High-precision detection is only performed within the confirmed valid ROI regions to obtain the detection result of the current frame. The consecutive hit statistics and consecutive loss count are updated based on the detection result of the current frame. Finally, the detection result of the current frame is output, and the running indicators, ROI state cache, and scheduling status are updated.

[0055] The operational metrics include at least one or more of the following: end-to-end latency, single-frame processing time, CPU utilization, ROI hit rate, detection effectiveness, fallback trigger statistics, and reuse trigger statistics. The ROI state cache includes at least the position, size, detection confidence, consecutive hit statistics, consecutive loss count, and current frame detection result of the ROI in the current frame.

[0056] The scheduling status includes at least the number of times the detection result is reused, the number of frames since the last global calibration, and the current frame backoff level. The updated operating metrics serve as the feedback basis for the closed-loop control update in step 2 of the next scheduling cycle, and the updated ROI state cache serves as the input basis for predictive ROI tracking and prior ROI reliability calculation in step 3 of the next scheduling cycle, thus forming a closed-loop ROI scheduling process across scheduling cycles.

[0057] Through steps 9-11, this invention completes a closed-loop process from valid ROIs to fine-tuning inputs, and from fine-tuning results to state feedback. This process constructs ROI fine-tuning input data through ROI pruning, ROI view mapping, and cache reuse, performing high-precision layered detection only within confirmed valid ROI regions, reducing the overhead of full-frame high-frequency fine-tuning, frequent memory allocation, and large-scale data copying. Furthermore, by outputting the current frame detection results and updating running metrics, ROI state cache, and scheduling status, the next scheduling cycle can continue to adjust the detection strategy based on feedback information, thereby reducing end-to-end latency and CPU utilization, and improving system real-time performance, stability, and closed-loop scheduling capabilities.

[0058] The effects of this invention will be further illustrated below with simulation experiments: 1. Experimental conditions Using VS Code software, a comparative experiment was conducted on the improved system architecture. The specific experimental conditions are shown in Table 1. Table 1 Experimental Environment Configuration Configuration detail Experimental Platform ESP32-S3 series chips Development framework ESP-IDF v5.5.1 Development tools VSCode, ESP-IDF Extension programming language C / C++ Host System Windows 11 Compilation tools CMake, Ninja, Xtensa cross-compilation toolchain Storage configuration 16 MB Flash, 8 MB PSRAM 2. Comparative analysis of experimental results To verify the effectiveness of the improved structure, ablation experiments were conducted on the coarse detection mode discrimination module, the validity check and hierarchical rollback module, the ROI pruning and cache reuse module, and the ROI fine detection module (referred to as Module 1, Module 2, and Module 3, respectively). The experimental results are shown in Tables 2 and 3. "√" indicates a module that is an improvement or introduction based on the baseline scheme, i.e., the traditional "coarse detection followed by fine detection" structure.

[0059] Table 2 Comparison of ablation test results under color detection task Module 1 Module 2 Module 3 CPU utilization rate / % End-to-end delay / ms Single frame processing time / ms Detection rate / % — — — 38.91 79.66 28.58 88.61 √ — — 26.21 69.86 21.54 90.24 — √ — 33.81 74.53 23.32 95.61 — — √ 28.62 67.64 19.64 92.61 √ √ — 28.49 70.32 22.85 96.48 √ — √ 21.42 60.28 15.24 94.52 — √ √ 22.75 68.43 16.32 95.67 √ √ √ 17.01 60.44 14.42 97.42 Table 3 Comparison of ablation experiment detection results under AI detection tasks Module 1 Module 2 Module 3 CPU utilization rate / % End-to-end delay / ms Single frame processing time / ms Detection rate / % — — — 98.00 295.11 196.47 83.42 √ — — 78.31 230.22 131.82 87.59 — √ — 90.11 270.68 170.52 94.32 — — √ 75.64 220.89 120.11 89.21 √ √ — 80.22 244.85 148.48 95.18 √ — √ 66.42 192.45 115.58 93.46 — √ √ 70.51 200.57 123.54 95.67 √ √ √ 64.80 184.48 95.22 96.52 As can be seen from the ablation experiment results in Tables 2 and 3, each of the improved modules proposed in this paper has improved performance.

[0060] In color detection tasks, the baseline solution had a CPU utilization of 38.91%, an end-to-end latency of 79.66 ms, a single-frame processing time of 28.58 ms, and a detection rate of 88.61%. After fully implementing the closed-loop ROI scheduling mechanism proposed in this invention, the CPU utilization, end-to-end latency, and single-frame processing time decreased by 55.3%, 23.9%, and 50.0%, respectively, while the detection rate increased by 10.2%. In AI detection tasks, the baseline solution achieved a CPU utilization of 98.00%, an end-to-end latency of 295.11 ms, a single-frame processing time of 196.47 ms, and a detection rate of 83.42%. After fully implementing the closed-loop ROI scheduling mechanism proposed in this invention, CPU utilization, end-to-end latency, and single-frame processing time decreased by 34.7%, 37.6%, and 51.4%, respectively, while the detection rate increased by 15.7%.

[0061] Based on the experimental results above, it can be seen that the overall system performance is improved to varying degrees after introducing modules 1, 2, and 3 respectively. Module 2 makes a significant contribution to improving the detection rate, indicating its outstanding role in improving detection quality. Modules 1 and 3 make significant contributions to reducing CPU utilization, end-to-end latency, and single-frame processing time, demonstrating their significant advantages in improving system real-time performance and resource utilization efficiency. This invention is not simply a matter of stacking multiple functional modules, but rather achieves a balance between detection quality, processing latency, and resource consumption through the collaborative cooperation between these modules. Therefore, the closed-loop ROI scheduling system framework proposed in this invention can effectively improve the overall detection performance and resource utilization of embedded vision inspection systems.

[0062] This invention enables closed-loop scheduling of the detection process using operational metrics under single-task serial execution conditions. It achieves dynamic acquisition of candidate regions through predictive ROI tracking and coarse detection mode discrimination, and performs validity checks and hierarchical rollback to ensure detection quality. Finally, it combines ROI pruning, caching, reuse, and ROI-based layered fine detection to reduce computational overhead, thereby improving overall detection performance and system stability while ensuring real-time performance.

Claims

1. A closed-loop ROI scheduling method for visual inspection on resource-constrained embedded platforms, characterized in that, Includes the following steps: Step 1: Acquire target scene images or videos containing the target to be detected from the camera, and use the image frames in the images or videos as data to be processed; Step 2: Read the running metrics from the previous scheduling cycle and obtain the ROI status cache; When the ROI state cache is empty or the current frame is the first frame, mark the current ROI state as invalid and select the coarse detection mode in step 3; Otherwise, the position and size of the ROI in the current frame are predicted based on the position and size of the ROI in the previous frame, and the predicted ROI of the current frame is obtained; when the predicted ROI satisfies the image boundary constraints and the minimum area constraints, it is determined as a valid predicted ROI; The prior ROI reliability of the current frame is calculated by combining the detection confidence, tracking consistency, hit statistics, and consecutive loss count of the previous frame. Step 3: Determine whether the detection result of the previous frame is reused in the current frame; If not reused, proceed to the coarse inspection mode selection process; select the coarse inspection mode for the current frame based on the prior ROI reliability, consecutive loss count, and the number of frames since the last global calibration. Step 4: Generate a set of candidate ROIs according to the selected coarse detection mode; Step 5: Merge, sort, and allocate budgets for the candidate ROI set and the effective predicted ROIs obtained in Step 2 to determine the ROIs to be refined. Step 6: Perform geometric boundary, ROI reliability, and budget constraint checks on the ROI to be inspected to obtain valid ROIs. If the check fails, perform a tiered rollback. Step 7: For confirmed valid ROIs, trim them or create ROI view mappings, and use a cache reuse method to construct ROI fine-tuning input data; Step 8: Perform high-precision layered detection on the ROI fine detection input data to obtain the detection result of the current frame; It then updates the ROI status cache, running metrics, and scheduling status, and enters the closed-loop ROI scheduling process for the next scheduling cycle.

2. The closed-loop ROI scheduling method for visual inspection on resource-constrained embedded platforms according to claim 1, characterized in that, The specific process of obtaining the image frame in step 1 is as follows: The OV2640 camera is initialized and configured through the target hardware platform. The initialization configuration includes at least clock configuration, resolution configuration, output format configuration, and frame buffer enable configuration. After initialization, the OV2640 camera acquires images of the target to be detected in the target scene, obtains the current frame image data containing the target to be detected, and records the frame number and acquisition timestamp of the current frame as input data for the subsequent closed-loop ROI scheduling process.

3. The closed-loop ROI scheduling method for visual inspection on resource-constrained embedded platforms according to claim 2, characterized in that, In step 2, the previous scheduling cycle is the statistical cycle that has been completed before the current frame is processed; when the resource-constrained embedded vision inspection system updates frame by frame, the previous scheduling cycle corresponds to the previous frame; when the resource-constrained embedded vision inspection system uses window statistics, the previous scheduling cycle corresponds to the statistical window of the previous K frames; the operating indicators include at least the end-to-end latency, CPU utilization, ROI hit rate, detection effectiveness rate, and backoff trigger statistics of the previous scheduling cycle; The ROI state cache includes at least the position, size, detection confidence, consecutive hit statistics, consecutive loss count, and detection results of the previous frame for the ROI. Based on the above operating indicators and the preset latency threshold, CPU utilization threshold and detection quality constraints, the scheduling control quantity of the current frame is updated; The scheduling control variables include coarse detection step size, coarse detection interval, fine detection interval, ROI hold frame count, ROI expansion parameters, and global calibration interval.

4. The closed-loop ROI scheduling method for visual inspection on resource-constrained embedded platforms according to claim 3, characterized in that, In step 2, When the ROI state cache is empty or the current frame is the first frame, the current ROI state is marked as invalid, and the full-frame coarse detection mode is selected in step 5; when the ROI state cache is valid, the position and size of the ROI in the current frame are predicted based on the position and size of the ROI in the previous frame, and the predicted ROI is obtained, denoted as . ; Based on the detection confidence, tracking consistency, consecutive hit statistics, and consecutive loss count of the previous frame, calculate the prior ROI reliability of the current frame. This is used for reuse determination in subsequent steps and selection of coarse inspection mode; The prior reliability of the current frame is represented as: In the formula, The prior ROI reliability for the current frame. The confidence level for detection or fine-tuning in the previous frame. To track consistency metrics, For consecutive hit statistics, For continuous loss counting, These are preset weighting coefficients.

5. A closed-loop ROI scheduling method for visual inspection on resource-constrained embedded platforms according to claim 4, characterized in that, In step 3, the reliability of the prior ROI of the current frame is calculated. Consecutive hit statistics read from the ROI state cache updated in the previous scheduling cycle. and consecutive loss count The number of times the detection results read from the scheduling state updated in the previous scheduling cycle are reused. And the number of ROI hold frames updated by closed-loop control. and global calibration interval To determine whether the detection result of the previous frame is reused in the current frame, the rules are as follows: The ROI hold frame count is used to limit the maximum number of frames that can be continuously reused for detection results; the global calibration interval is used to limit the maximum interval at which the system forces local coarse detection or full-frame coarse detection; when the prior ROI reliability is higher than the reuse threshold, the consecutive loss count is less than the loss threshold, the number of times the detection result is reused does not exceed the ROI hold frame count, and the number of frames since the last global calibration does not exceed the global calibration interval, the previous frame detection result is reused; otherwise, the coarse detection mode selection is entered.

6. A closed-loop ROI scheduling method for visual inspection on resource-constrained embedded platforms according to claim 5, characterized in that, The coarse detection modes include skip coarse detection mode, partial coarse detection mode, and full-frame coarse detection mode; When determining that the detection result of the previous frame should not be reused in the current frame, the reliability of the ROI in the current frame is used as a basis. Continuous loss count and the number of frames since the last global calibration. The coarse detection mode for the current frame is selected, and its discriminant relationship is expressed as follows: In the formula, Select the result for the coarse detection mode of the current frame. and These are the high reliability threshold and the low reliability threshold, respectively. when When the reliability of the prior ROI in the current frame is high and the target state is stable, the coarse detection mode is skipped, and the effective predicted ROI obtained in step 3 is used as the candidate ROI, and then the process proceeds to step 5. when When the reliability of the prior ROI in the current frame is in the medium range, the local coarse detection mode is selected, and coarse detection is performed in the local area around the predicted ROI, and then the process proceeds to step 4 to generate a set of candidate ROIs. when When the reliability of the prior ROI in the current frame is low, the target is continuously lost, or the global calibration condition is reached, the full-frame coarse inspection mode is selected, and coarse inspection is performed in the entire map of the current frame, and then the process proceeds to step 4 to generate a candidate ROI set.

7. A closed-loop ROI scheduling method for visual inspection on resource-constrained embedded platforms according to claim 6, characterized in that, In step 4, when the skip coarse detection mode is selected, the obtained effective predicted ROI is used as a candidate ROI; when the local coarse detection mode or the full-frame coarse detection mode is selected, sparse sampling or resolution-layered coarse detection is performed, and a candidate ROI set is generated. Among them, sparse sampling coarse detection is used to reduce the scanning density by increasing the sampling step size, and resolution layered coarse detection is used to quickly obtain candidate regions by reducing the resolution of the input image. The system selects one or a combination of them based on the current CPU utilization, end-to-end latency and target scale changes. The specific process for generating candidate ROIs in step 4 is as follows: When step 3 selects to skip the coarse detection mode, the coarse detection calculation is not performed, and the obtained effective predicted ROI is used as a candidate ROI and added to the candidate ROI set of the current frame. When the local coarse detection mode is selected in step 3, the effective predicted ROI is used as the center, and the ROI expansion parameters are used to determine the expansion parameters. roi_pad Construct a local search region, and within the local search region, perform a coarse detection step size. coarse_stride Alternatively, a resolution-layered approach can be used to perform fast detection and generate a set of candidate ROIs; When step 3 selects the full-frame coarse inspection mode, the coarse inspection step size is applied to the entire current frame. coarse_stride Alternatively, a resolution-layered approach can be used to perform fast detection, generating a candidate ROI set; this candidate ROI set is denoted as... And record the coarse confidence score for each candidate ROI. Candidate area And position parameters.

8. A closed-loop ROI scheduling method for visual inspection on resource-constrained embedded platforms according to claim 7, characterized in that, The specific process of constructing and screening the ROI candidate pool in step 5 is as follows: Step 5.1: Construct a candidate pool based on the candidate ROI set generated in Step 4; when there is a valid predicted ROI in Step 3 and the valid predicted ROI is not yet included in the candidate ROI set generated in Step 4, merge the valid predicted ROI into the candidate pool, and perform overlap merging, deduplication and sorting on the ROIs in the candidate pool. Step 5.2: For each candidate ROI in the candidate pool, its comprehensive score is expressed as: In the formula, For the first The overall score of each candidate ROI. As a measure of consistency between candidate ROIs and effective predicted ROIs, This is an area penalty item. For the lost penalty item, These are weighting coefficients; when candidate ROIs originate from valid predicted ROIs and no coarse check is performed, The confidence level of the previous frame or the reliability of the prior ROI in the current frame is used as a substitute. Under the constraint of the detailed inspection budget, the candidate ROI with the highest score or the score that meets the preset threshold is selected as the ROI to be detailed inspection.

9. A closed-loop ROI scheduling method for visual inspection on resource-constrained embedded platforms according to claim 8, characterized in that, The specific process of step 6, ROI validity check and graded rollback, is as follows: Step 6.1: Perform detailed inspection on the samples identified in Step 5. A validity check is performed, which includes ROI geometric boundary check, ROI reliability check, and budget constraint check. The validity determination relationship is expressed as follows: In the formula, This is the result of determining the validity of the ROI to be further examined. The area of ​​ROI to be precisely inspected. and These are the minimum and maximum area thresholds for the ROI, respectively. The current frame image boundary region, The overall score of the ROI to be refined. As the validity threshold, For indicator functions; Step 6.2: When At that time, Once confirmed as a valid ROI, proceed to step 7; when When a tiered rollback is triggered, the rollback level is represented as follows: In the formula, The current frame's backoff level. This is a slight correction and regression. For partial rescan and rollback, To recapture the entire frame, The low reliability failure threshold, and < ; when = When increasing the ROI expansion parameter, roi_pad Alternatively, adjust the ROI geometric boundaries and re-check the validity. when = If necessary, expand the local coarse inspection area and return to step 4 to perform local coarse inspection and candidate ROI generation; when = If the condition is not met, the system will force a switch to full-frame coarse detection mode and return to step 3 to reselect the coarse detection mode.

10. A closed-loop ROI scheduling method for visual inspection on resource-constrained embedded platforms according to claim 9, characterized in that, The specific process of step 7, ROI pruning and cache reuse preparation, is as follows: Step 7.1: Crop the valid ROIs confirmed in Step 6 on the current frame image, or establish the corresponding ROI view mapping to obtain ROI image blocks or ROI mapping regions; Step 7.2: Construct ROI fine-tuning input data using a cache reuse method, wherein the cache reuse method includes at least one or more of the following: pre-allocated cache, double buffering, circular reuse, or zero-copy mapping operation; The specific process of step 8, ROI stratification and fine-tuning, is as follows: Perform hierarchical fine detection on the ROI fine detection input data constructed in step 7, and perform high-precision detection only in the confirmed valid ROI areas to obtain the detection result of the current frame; update the consecutive hit statistics and consecutive loss count according to the detection result of the current frame; The specific process of outputting and updating the state in step 8 is as follows: Output the detection results of the current frame, and update the running metrics, ROI status cache, and scheduling status; The operational metrics include at least one or more of the following: end-to-end latency, single-frame processing time, CPU utilization, ROI hit rate, detection effectiveness, fallback trigger statistics, and reuse trigger statistics. The ROI state cache includes at least the position, size, detection confidence, consecutive hit statistics, consecutive loss count, and detection result of the current frame ROI; The scheduling status includes at least the number of times the detection result is reused, the number of frames since the last global calibration, and the current frame backoff level. The updated operating metrics serve as the feedback basis for the closed-loop control update in step 2 of the next scheduling cycle, and the updated ROI state cache serves as the input basis for predictive ROI tracking and prior ROI reliability calculation in step 2 of the next scheduling cycle, thus forming a closed-loop ROI scheduling process across scheduling cycles.

Images