Video target detection scheduling method and system based on task urgency

Abstract

The application discloses a video target detection scheduling method and system based on task urgency, and belongs to the technical field of image recognition, and comprises the following steps: forming a current evidence sequence, queue waiting duration and video frame arrival interval according to detection task state data; extracting a similar evidence continuation section in combination with historical alarm records to obtain a remaining confirmation duration and an effective frame number; generating a remaining waitable duration, a latest start reasoning time, a next frame to be detected and an extraction deadline, and generating a reasoning execution sequence, and calling a detection model to write a detection result. The application matches a historical similar alarm section through a current evidence sequence, generates a remaining waitable duration, a next frame extraction deadline and a reasoning execution sequence, achieves the technical effect of sorting by delay risk and keeping key change frame coverage, and solves the problems of fixed priority queuing and unified frame reduction in the prior art, which leads to scheduling basis deviation, insufficient key change frame coverage and unreasonable reasoning resource occupation.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a video target detection scheduling method and system based on task urgency. Background Technology

[0002] With the expansion of video surveillance in smart parks, urban governance, and industrial security scenarios, multiple cameras are continuously generating a large amount of image data. Video target detection and image recognition technologies can automatically identify people, vehicles, smoke, and equipment status, providing a foundation for real-time monitoring, event detection, and resource scheduling.

[0003] Existing systems typically involve a platform that connects to multiple camera video streams. Video frames are extracted from each stream according to preset cycles, inspection tasks, or alarm trigger conditions. Target detection models for personnel, vehicles, smoke, and equipment status are then used for image recognition, outputting target category, confidence level, detection box location, target appearance time, and alarm result. Video inspection systems also combine camera locations, task types, real-time operational data, inspection plans, or event levels to generate task priorities or scheduling sequences. Based on this scheduling sequence, different detection tasks are then placed into an inference queue to complete multi-channel video target detection and inspection task execution.

[0004] For example, Chinese invention patent CN120198785B discloses a video inspection task priority scheduling system that integrates real-time data. This system includes: deploying cameras and sensors in different monitoring areas; associating video frames and sensor data with their respective area numbers for storage; generating anomaly summary information based on the input video frames using a multimodal anomaly detection model; polling and inspecting video frames in the monitoring area according to the video task inspection priority; when anomalies in sensor data occur, obtaining the anomaly summary information of the current monitoring area's video frames; triggering the retrieval of video frames associated with the monitored targets in the associated monitoring areas based on the generation frequency of the anomaly summary information; extracting associated anomaly summary information using a monitoring target feature extraction module; and adjusting the video inspection task priority of the corresponding monitoring area according to the generation frequency of the anomaly summary information and the associated anomaly summary information.

[0005] The above-mentioned technology has at least the following technical problems: In existing technologies, multi-camera video surveillance platforms only queue tasks by task name, manual risk level, or fixed priority. This makes it difficult to determine the latest processing time for tasks that can be scheduled, considering the risk of missed detections for the same type of event under different conditions such as regional level, target movement speed, number of consecutive frames, confidence fluctuations, and queue waiting time. This leads to inaccurate criteria for tasks entering the inference queue. Furthermore, if fixed frame extraction or uniform frame reduction is used when the queue is congested, it can easily disrupt the effective detection frame coverage required for different events. For example, images of smoke or flames changing over time, images of people approaching and crossing a warning area, images of vehicles changing their stationary state, or images of equipment undergoing localized changes in state may be skipped. This distorts state data such as the number of consecutive frames, target change rate, and confidence fluctuations, ultimately resulting in scheduling bias, insufficient coverage of key change frames, and unreasonable use of inference resources. Summary of the Invention

[0006] To address the technical problems of scheduling bias, insufficient coverage of key change frames, and unreasonable inference resource consumption in existing technologies, this invention provides a video target detection scheduling method and system based on task urgency. The technical solution is as follows: On the one hand, a video object detection scheduling method based on task urgency is provided, including: Based on the detection task status data for each detection task, the current evidence sequence, queue waiting time, and video frame arrival interval are generated. Similar evidence continuation segments are extracted based on the current evidence sequence, detection task status data, and historical alarm records to obtain the remaining confirmation time and remaining valid frames of historical segments. The remaining waiting time and latest inference start time are generated based on the remaining confirmation time, queue waiting time, and video frame arrival interval. The next frame to be submitted and the next frame extraction deadline are generated based on the remaining waiting time, the remaining valid frames of historical segments, and the timestamps corresponding to the arrived frames. The current round of inference execution sequence is generated based on the next frame to be submitted, the latest inference start time, the detection task status data, and the current inference queue. The detection model is invoked according to the current round of inference execution sequence to detect the next frame to be submitted and the detection results are written.

[0007] On the other hand, a video target detection scheduling system based on task urgency is provided, including an evidence sequence formation module, a similar segment extraction module, a waiting time generation module, a frame extraction time generation module, an inference sequence generation module, and a detection result writing module.

[0008] The evidence sequence formation module is used to form the current evidence sequence, queue waiting time, and video frame arrival interval based on the detection task status data of each detection task.

[0009] The similar segment extraction module is used to extract similar evidence continuation segments based on the current evidence sequence, detection task status data, and historical alarm records, and to obtain the remaining confirmation time and the remaining number of valid frames of the historical segment.

[0010] The waiting time generation module is used to generate the remaining waiting time and the latest inference start time based on the remaining confirmation time of historical segments, queue waiting time, and video frame arrival interval.

[0011] The frame extraction time generation module is used to generate the next frame to be submitted for inspection and the next frame extraction deadline based on the remaining waiting time, the number of remaining valid frames in the historical segment, and the timestamps corresponding to the arrived frames.

[0012] The inference sequence generation module is used to generate the current round of inference execution sequence based on the next frame to be submitted for inspection, the latest start time of inference, the detection task status data, and the current inference queue.

[0013] The detection result writing module is used to call the detection model according to the current inference execution sequence, detect the next frame to be submitted for inspection, and write the detection result.

[0014] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: 1. The video target detection scheduling method based on task urgency provided by this invention forms a current evidence sequence based on the detection task status data of each detection task, and combines historical alarm records, queue waiting time and video frame arrival interval to obtain the remaining confirmation time, remaining waiting time and latest start time of inference for historical segments. This transforms the task urgency of different video states in a multi-camera video surveillance platform into time parameters that can be used for inference, and then arranges the execution order of detection tasks according to the actual delay risk of the target event. This effectively solves the problem in the prior art that it is difficult to reflect the changes in the risk of missed detection of similar events by queuing only according to a fixed priority.

[0015] 2. This invention generates the next frame to be submitted for inspection and the next frame extraction deadline, and combines it with the current inference queue to obtain the inference execution sequence of this round. This allows different detection tasks to determine the frame extraction rhythm according to the number of effective detection frames required for their historical similar events. In this way, key change frames are covered even when edge inference resources are limited. This effectively solves the problem in the prior art where fixed frame extraction or uniform frame reduction leads to the skipping of key frames for high-risk events and the occupation of inference computing power by low-risk tasks.

[0016] 3. This invention matches the current evidence sequence with similar evidence continuation segments in historical alarm records, and simultaneously extracts the remaining confirmation time and the remaining effective frames of historical segments to quantitatively evaluate the future evolution trend of the current task. This allows the task queuing time and frame extraction rhythm to be jointly constrained by the development process of similar events, providing a basis for prioritizing detection tasks and formulating frame extraction strategies. It enhances the correspondence between inference scheduling and the temporal changes of video events, and reduces the risk of missed events and alarm delays caused by excessive waiting time or insufficient frame extraction. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of a video target detection scheduling method based on task urgency provided in this application embodiment; Figure 2 A schematic diagram of the structure of a video target detection scheduling system based on task urgency provided in an embodiment of this application; Figure 3 A flowchart for extracting continuation segments of historical similar evidence provided in this application embodiment; Figure 4 A flowchart illustrating frame extraction control provided in an embodiment of this application. Detailed Implementation

[0019] Embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present disclosure are shown in the drawings, it should be understood that embodiments of the present disclosure may be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure.

[0020] It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure. In the description of the embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "this embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects.

[0021] The method provided in this embodiment is applied to a real-time target detection task scheduling scenario in a multi-camera video surveillance platform. The system accesses video streams from locations such as park entrances / exits, key passages, warehouse areas, production equipment areas, and parking areas, and configures GPUs, NPUs, or other inference resources on edge computing devices or servers. The target detection models in this embodiment are all already deployed in the video surveillance platform and are used to output target categories, detection box positions, and detection box confidence scores for video frames. They can be configured for a single detection task category or output multiple target categories. This embodiment does not limit the specific network structure and training process of the target detection model; in the following text, they are only referred to as the detection units being scheduled and invoked. The detection model identifier is a field in the detection task status data, used to record the name and version number of the target detection model called in this round of the detection task, and corresponds to the detection task category. This facilitates accurate recording of the scheduling basis even when different models have different inference times, different detection objects, or the model has been updated.

[0022] In this embodiment, a detection task refers to a processing task established for a detection target or a detection event in a video stream. For example, a video stream from the same entrance / exit can simultaneously establish a personnel intrusion detection task, a vehicle illegal parking detection task, and an area aggregation detection task; a video stream from the same equipment area can establish a device anomaly detection task and a smoke and fire recognition detection task. Detection task status data includes frame timestamps, detection task category, detection model identifier, camera area level, time the task entered the inference queue, detection results of inferred frames, and historical inference time of the model. The camera area level is written by the platform when the camera is deployed or configured for services, such as key area, ordinary area, or low-risk area, or it can be represented by a numerical level. The historical inference time of the model is updated by updating the historical actual inference time corresponding to the same detection model identifier. The initial value can be the median of the inference time of at least 20 inferences during the model's online testing phase, and subsequent updates are based on the time calculated from the actual start and end times of inference. Furthermore, this embodiment operates with a fixed scheduling cycle. At the arrival of each scheduling cycle, the system reads the detection task status data and the current inference queue of each detection task, and completes the video target detection scheduling according to the following steps.

[0023] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0024] like Figure 1 The diagram shown is a flowchart of a video target detection scheduling method based on task urgency provided in an embodiment of this application. The method includes the following steps: S1: Based on the detection task status data of each detection task, the current evidence sequence, queue waiting time, and video frame arrival interval are formed. Specifically, the system reads the most recent inferred frames for each detection task. If the platform is being deployed for the first time, or if a certain detection task has no similar historical alarm records, the evidence window frame number N is determined according to the number of frames that have been inferred in the video stream in the most recent observation period. The observation period is the time range within which the platform saves inferred frame evidence items. Specifically, the system reads the inferred frames of the detection task backward from the current scheduling cycle time as the end point, stopping when no earlier inferred frames are read. The time range between the earliest inferred frame read and the current scheduling cycle time is used as the observation period. If there are no inferred frames in the initial deployment stage, N first takes the lowest number of consecutive evidence frames and marks the detection task as a cold start candidate task; if there are already arrived frames at this time, the latest arrived frame is used as the frame to be submitted for inspection according to the cold start process. The system sets an evidence cache for each detection task. The evidence cache consists of several frame slots. Each frame slot stores the target category, task confidence, detection box center position, detection box width and height, frame timestamp, and empty frame marker corresponding to a reasoned frame. N must not be less than the minimum number of consecutive evidence frames configured by the platform for this detection task, nor greater than the number of evidence cache frame slots for this detection task.

[0025] In this embodiment, the minimum number of consecutive evidence frames is the minimum number of consecutive reasoned frames required to establish an alarm for the detection task, as defined in the platform's existing alarm confirmation rules. The number of evidence cache frame slots is obtained by dividing the number of evidence cache bytes allocated to a single detection task by the number of bytes occupied by a single frame slot and taking the integer part. The number of evidence cache bytes is the storage space reserved by the system for the detection task to store reasoned frame evidence items, measured in bytes. The number of bytes occupied by a single frame slot is the sum of the number of bytes for the target category field, task confidence field, detection box center position field, detection box width and height field, frame timestamp field, and empty frame marker field. Therefore, the system first counts the number of frames that have completed reasoning in the most recent observation period, and then compares this number with the minimum number of consecutive evidence frames and the number of evidence cache frame slots. If the number is less than the minimum number of consecutive evidence frames, then N is the minimum number of consecutive evidence frames; if the number is greater than the number of evidence cache frame slots, then N is the number of evidence cache frame slots; otherwise, N is the number of frames that have completed reasoning in the most recent observation period. If similar historical alarm records already exist, the number of reasoned frames from the first appearance of a suspected target to alarm confirmation in those records is read, and the median is taken as the evidence window frame number N. N is then limited based on the platform's configured minimum number of consecutive evidence frames and the number of bytes for evidence cache. This quantification rule ensures that N covers the main segments of consecutive evidence in the current event's formation process without causing excessive computational load for historical comparisons due to an overly long window.

[0026] For each inferred frame, the system retains only the detection result, task confidence, detection box center position, detection box width and height, and frame timestamp corresponding to the detection task category. After the target detection model completes inference on the inferred frame, it outputs the target category, detection box position, and detection box confidence. The system reads the target category, detection box position, and detection box confidence that match the detection task category from the model output. The target category and detection box position form the detection result corresponding to that frame, and the detection box confidence is written into the current evidence sequence as the task confidence. If multiple detection boxes belonging to the same detection task category exist within the same frame, the detection box with the highest confidence is selected as the task detection box corresponding to that frame, and the task confidence is taken as the confidence of that detection box. The detection box center position is obtained by calculating the center point from the coordinates of the top-left corner and the bottom-right corner of the detection box, and the detection box width and height are obtained by the difference between the horizontal and vertical coordinates of the bottom-right corner and the top-left corner. If no corresponding target is detected in the frame, the frame is recorded as an empty frame, the task confidence is recorded as 0, and the detection box center position and detection box width and height are recorded as null values. The system arranges the target category, task confidence, detection box center position, detection box width and height, and frame timestamp of the above frames from earliest to latest according to the frame timestamp, forming the current evidence sequence. If the number of inferred frames for a certain detection task is less than N, the existing frames are written into the current evidence sequence in chronological order, and the detection task is marked as a cold start candidate task. It will directly enter the cold start process when there is insufficient historical fragments.

[0027] The queue waiting time is obtained by subtracting the time the task entered the inference queue from the current time. If a detection task has not yet joined the current inference queue, the starting time for the waiting period is the time when the task most recently generated a frame to be submitted for inspection; if no frame to be submitted for inspection has been generated, the starting time is the current scheduling cycle time. The video frame arrival interval includes the latest video frame arrival interval and the median of the frame arrival intervals in the current evidence sequence. The latest video frame arrival interval is obtained by subtracting the timestamps of the two most recent arrived frames from the same video stream, and the median of the frame arrival intervals in the current evidence sequence is obtained by taking the median of the timestamp intervals of adjacent frames in the current evidence sequence. Thus, the current evidence sequence, queue waiting time, video frame arrival interval, and corresponding detection task status data are obtained for each detection task.

[0028] S2: Extract similar evidence continuation segments based on the current evidence sequence, detection task status data, and historical alarm records to obtain the remaining confirmation time and the remaining valid frames of the historical segments. For example... Figure 3The diagram shows a flowchart for extracting historical similar evidence continuation segments according to an embodiment of this application. Historical alarm records are written by the platform during historical alarm confirmation, including the detection task category, detection model identifier, camera area level, evidence sequence fragments from the first appearance of a suspected target to alarm confirmation, confirmation time, and number of valid detection frames. The detection task category, detection model identifier, and camera area level in the historical alarm records are fields for historical events that have already been confirmed, while the detection task category, detection model identifier, and camera area level in the detection task status data are fields for the current detection task. Both fields have the same name but correspond to historical events and the current detection task, respectively, and are used in this step to filter comparable historical alarm records. The number of valid detection frames refers to the number of inferred frames used in the platform's existing alarm confirmation rules to form judgments of continuous appearance, continuous dwell, status change, or alarm establishment from the first appearance of a suspected target to alarm confirmation. The alarm confirmation rules are conventional alarm rules pre-configured by the video target detection platform for different detection task categories. This embodiment only reads the alarm confirmation results and the number of inferred frames involved in the alarm confirmation.

[0029] The system first filters historical alarm records that are identical to the current detection task in terms of detection task category, detection model identifier, and camera area level. If the number of filtered records is less than M, the camera area level restriction is lifted, and only historical alarm records with the same detection task category and detection model identifier are retained. If the number of filtered records is still less than M after this operation, the current detection task enters cold start processing. The frame extraction control module takes the latest arrived frame as the frame to be sent for inspection, the system sets the latest inference start time to the current time, and uses the cold start processing result as the input of S5 for this round of sorting. Cold start processing is used to avoid waiting for historical statistical results when there are insufficient historical alarm records, ensuring that new tasks, new areas, or new models can still immediately obtain an inference opportunity. The preset number M is used to ensure the stability of historical segment statistics, and its value is the number of historical alarm records that can form a median statistic, and is not less than 3. When there are fewer than M historical alarm records that meet the conditions, the remaining time is no longer estimated based on historical segments, but cold start processing is initiated.

[0030] Before entering the cold start process, the system extracts a fragment of the evidence sequence prior to alarm confirmation for each selected historical alarm record. The length of the evidence sequence fragment is the same as the number of evidence window frames N in S1, used to enable frame-by-frame comparison between the current evidence sequence and the historical evidence sequence. If the number of frames that can be extracted before alarm confirmation in a historical alarm record is greater than N, then the N frames closest to the alarm confirmation frame are extracted; if the number of frames that can be extracted is equal to N, then the fragment is used directly; if the number of frames that can be extracted is less than N, then the record does not participate in the comparison of similar fragments in this round.

[0031] The system aligns each historical evidence sequence fragment with the current evidence sequence frame by frame, comparing frames 1 to N sequentially. The comparison includes whether the target appearance state is consistent, the task confidence difference, the detection box center displacement difference, and the frame arrival interval difference. The target appearance state is indicated by whether it is an empty frame; consistency is defined as both the current and historical frames being empty frames or both detecting the target, otherwise it is defined as inconsistent. Inconsistency is marked as 1, and consistency as 0. The task confidence difference is the absolute difference between the task confidence of the current frame and the task confidence of the historical frame under the same frame number. The detection box center displacement difference reflects changes in target motion. It is calculated by first determining the center movement distance of the detection boxes in adjacent frames of the current evidence sequence and the center movement distance of the detection boxes in adjacent frames of the historical evidence sequence, then taking the absolute value of the difference. When empty frames are involved, the center displacement difference of the detection boxes corresponding to that frame is recorded as the normalized maximum value of 1. The frame arrival interval difference is the absolute difference between the frame arrival interval of the current evidence sequence and the frame arrival interval of the historical evidence sequence under the same adjacent frame number.

[0032] To enable the accumulation of different units of measurement, the system normalizes the differences between each item to a range of 0 to 1. Specifically, if the task confidence is already within the range of 0 to 1, the absolute difference between the confidence levels of two tasks is directly used as the normalized value; the difference in the center displacement of the detection box is divided by the length of the image diagonal, and if it exceeds 1, it is counted as 1, where the length of the image diagonal is calculated from the image width and image height in the video frame resolution; the difference in frame arrival intervals is divided by the median of the frame arrival intervals in the current evidence sequence, and if it exceeds 1, it is counted as 1. When the median of the frame arrival intervals in the current evidence sequence is 0, the arrival interval of the latest video frame is used as the denominator; if the arrival interval of the latest video frame is still 0, the platform scheduling cycle is used as the denominator; the inconsistency flag for the target's appearance status is itself set to 0 or 1. The system accumulates the normalized differences from frame 1 to frame N in frame order to obtain the similarity difference between the historical evidence sequence segment and the current evidence sequence. Accumulating in frame order preserves the temporal relationship between target appearance, target movement, confidence fluctuations, and changes in video frame arrival, avoiding the loss of evidence change process by only comparing average values.

[0033] After calculating the similarity difference for all historical evidence sequence segments, the system calculates the median of all similarity differences and selects evidence sequence segments with similarity differences no greater than this median as historical segments. Smaller similarity differences indicate a closer similarity between the historical and current evidence sequences. Segments with similarity differences no greater than the median are retained as historical segments with higher similarity, while those with significant deviations are excluded. For each historical segment, the time difference between the last frame and the alarm confirmation frame is read as the remaining confirmation time for that historical segment. Between the last frame and the alarm confirmation frame, the number of inferred frames used by existing alarm confirmation rules to form alarm confirmation results is counted as the remaining valid detection frames. Inferred frames used to form alarm confirmation results include those involved in continuous occurrence judgments, sustained dwell judgments, state change judgments, or other alarm establishment judgments. The system takes the median of the remaining confirmation time and the remaining valid detection frames for each historical segment to obtain the remaining confirmation time and the remaining valid frames for that historical segment.

[0034] S3: Based on the remaining confirmation time of historical segments, queue waiting time, and video frame arrival interval, the system generates the remaining waiting time and the latest inference start time. For detection tasks obtained through cold start processing, the system directly uses the latest inference start time set in S2. For detection tasks that have not entered cold start processing, the system subtracts the queue waiting time from the remaining confirmation time of historical segments to determine the time that the detection task has consumed while waiting in the queue.

[0035] Next, it is determined whether the arrival interval of the latest video frame is greater than the median arrival interval of frames in the current evidence sequence. If it is greater, the median arrival interval of frames in the current evidence sequence is subtracted from the arrival interval of the latest video frame to obtain the video frame arrival gap. This gap represents the time increase in the interval between adjacent arriving frames relative to the regular frame arrival interval of the current evidence sequence. The coverage of subsequent frames that can be used for continuous evidence judgment may decrease, so this video frame arrival gap is directly subtracted from the time obtained in the previous step; otherwise, the video frame arrival gap is not subtracted. When the above calculation result is greater than 0, the remaining waiting time is recorded as the calculation result; otherwise, it is directly recorded as 0. The smaller the remaining waiting time, the closer the detection task is to the time window or alarm confirmation window when it cannot continue to form continuous judgment, and the higher the task urgency. The system uses the current time as the time base, adds the remaining waiting time to the current time, and obtains the latest time to start inference. Thus, the task urgency is converted into a time parameter that can be directly used for sorting the inference queue, avoiding fixed sorting based solely on task name or manual risk level.

[0036] S4: Generate the next frame to be submitted for inspection and the next frame extraction deadline based on the remaining waiting time, the remaining valid frames in the historical segment, and the timestamps corresponding to the arrived frames. For example... Figure 4 The diagram shows a frame extraction control flowchart provided in this application embodiment. The timestamps corresponding to the arrived frames are obtained from the original video frames received by the system. Specifically, when the remaining valid frames of a historical segment are greater than zero, the remaining waiting time is evenly distributed according to the remaining valid frames of the historical segment to obtain the target time interval between two adjacent frame extractions. This target time interval is then added to the timestamp of the previous submitted frame to obtain the cutoff time for the next frame extraction. When the remaining valid frames of a historical segment are zero, the cutoff time for the next frame extraction is set to the timestamp of the previous submitted frame. Subsequently, all arrived frames with timestamps later than the timestamp of the previous submitted frame are checked in chronological order. If there is an arrived frame with a timestamp no later than the cutoff time for the next frame extraction, the arrived frame with the closest timestamp and no later than the cutoff time for the next frame extraction is selected as the next frame to be submitted for inspection. If there is no arrived frame with a timestamp no later than the cutoff time for the next frame extraction, the arrived frame with the earliest timestamp and later than the cutoff time for the next frame extraction is selected as the next frame to be submitted for inspection. The timestamp of the previous submitted frame is the timestamp of the most recent frame submitted to the current inference queue for this detection task. If this detection task enters S4 for the first time and there is no timestamp of the previous submitted frame yet, then if there are already inferred frames in the current evidence sequence, the timestamp of the last inferred frame in the current evidence sequence is used as the timestamp of the previous submitted frame; if there are no inferred frames in the current evidence sequence, the latest arrived frame is used as the next frame to be submitted and is included in S5 as a cold start task. The new frame is an arrived frame with a timestamp later than the timestamp of the previous submitted frame.

[0037] When the remaining valid frames of a historical segment do not meet the multi-frame segmentation condition, the system will use the latest arrived frame as the next frame to be submitted for inspection. The multi-frame segmentation condition is that the remaining valid frames of the historical segment can form at least two extraction time periods within the remaining waiting time. Specifically, when the remaining valid frames of the historical segment are greater than 1 and the remaining waiting time is greater than 0, the multi-frame segmentation condition is met. When the remaining valid frames of the historical segment are less than or equal to 1, or the remaining waiting time is equal to 0, the multi-frame segmentation condition is not met. In this case, the latest arrived frame will be directly taken to ensure that urgent tasks enter the inference queue as soon as possible.

[0038] When the remaining valid frames of a historical segment meet the multi-frame segmentation condition, the frame extraction control module divides the remaining waiting time into equal-length time intervals based on the remaining valid frames of the historical segment. For example, when the remaining valid frames of a historical segment are K, the length of each extraction time interval is the remaining waiting time divided by K. The system adds one extraction time interval length forward from the timestamp of the previous submitted frame to determine the extraction deadline for the next frame. It then selects the frame whose timestamp is no later than the extraction deadline and is closest to that deadline from the already arrived frames in the video stream as the next frame to be submitted. If no new frame arrives before the extraction deadline for the next frame, no new frame for this detection task is added to the current inference queue, and the remaining waiting time is deducted based on the arrival interval of the latest video frame in the next scheduling cycle. Thus, the system determines the frame extraction rhythm of each detection task based on the number of valid detection frames that still need to be retained before alarm confirmation in the historical segment, thereby balancing inference resource usage and coverage of key change frames.

[0039] S5: Generate the current round of inference execution sequence based on the next frame to be submitted for inspection, the latest inference start time, the detection task status data, and the current inference queue. First, determine whether the detection task has entered the current round of sorting. In this step, the system uses the detection task status data to identify the current detection task record. Specifically, it reads the historical inference time of the corresponding model according to the detection model identifier; it determines whether the detection task has reached the next frame extraction deadline based on the frame timestamp of the next frame to be submitted for inspection; and it updates the queue waiting time in the next scheduling cycle based on the time the task entered the inference queue. The detection results of the inferred frames are retained with the detection task record for the next cycle to form the current evidence sequence. For detection tasks that have not entered the cold start process, they are only included in the current round of sorting if they have reached the next frame extraction deadline and generated the next frame to be submitted for inspection; detection tasks that have not reached the next frame extraction deadline are not included in the current round of sorting and are recorded as detection tasks not included in the current round of sorting. For detection tasks obtained from the cold start process, since the latest arrived frame has been used as the frame to be submitted for inspection and the latest inference start time is the current time, they are directly included in the current round of sorting.

[0040] For detection tasks included in this round of sorting, they are arranged from earliest to latest according to their latest inference start time; that is, the earlier the latest inference start time, the higher the detection task ranks in this round of sorting. The sorted order of detection tasks is used as the inference execution sequence for this round. After sorting, the system accumulates the historical inference time of the model corresponding to each detection task at the front of the inference execution sequence to obtain the estimated start inference time for each detection task. Since tasks in the same inference queue occupy the same inference resources sequentially, two inference tasks cannot start simultaneously from the same inference queue. Furthermore, the historical inference time of different detection models may vary. Therefore, the estimated start inference time reflects the actual executable pace of the current inference queue.

[0041] In the current inference queue, only one frame awaiting inference for the same detection task is retained. If the detection task already has an old frame that has not yet been inferred, and an updated frame awaiting inference is generated in this cycle, the old frame that has not yet been inferred is replaced with the updated frame, and the replaced old frame is recorded. The replacement does not change the video stream itself, only the inference input that has not yet been executed. Since the updated frame is closer to the deadline for extracting the next frame, it is more representative of the subsequent state of the current evidence sequence. If the expected start time of inference for a detection task is later than the latest start time of inference, it is still executed according to the latest start time of inference, and the time difference between the expected start time of inference and the latest start time of inference is recorded. This time difference is used to update the queue waiting time in the next scheduling cycle, so that the queue overdue affects the calculation of the remaining waiting time and the latest start time of inference in the next round, thereby changing the task urgency expressed by the time parameter in the next round.

[0042] S6: The system invokes the target detection model according to the current inference execution sequence, detects the next frame to be submitted for inspection, and writes the detection result. Specifically, starting from the first item of the current inference execution sequence, the system sequentially reads the detection task and the corresponding next frame to be submitted for inspection, invokes the corresponding target detection model according to the detection model identifier in the detection task status data, performs target detection on the corresponding next frame to be submitted for inspection, and outputs the target category, confidence level, and detection box position. The platform records the actual start time of inference, the actual end time of inference, the target category corresponding to the detection task category, the confidence level, the detection box position, and the alarm confirmation result. Among them, the alarm confirmation result is generated by the platform's existing alarm rules. This part belongs to the conventional alarm confirmation logic in the target detection system. In this embodiment, only the alarm confirmation is read, and the number of inferred frames that participated in the alarm confirmation is read and written to the corresponding detection task record to form the basis for subsequent scheduling.

[0043] After completing this round of inference, the detection results are appended to the current evidence sequence of the corresponding detection task according to the frame timestamp, and the most recent N inferred frames are retained. The actual inference time is obtained by subtracting the actual start time from the actual end time of inference. The platform updates the historical inference time of the model corresponding to the detection model identifier based on the actual inference time. Specifically, when the same detection model identifier already has an actual inference time record, the historical inference time of the model is taken as the median of the most recent actual inference times saved by the system; if there is no actual inference time record, the initial value obtained during the model's online testing phase continues to be used. Using the median can reduce the impact of single device load fluctuations or single video frame complexity anomalies on the historical inference time of the model. If a detection task triggers an alarm confirmation, the system extracts the evidence sequence segment from the frame where the suspected target first appears to the alarm confirmation frame from the current evidence sequence of the corresponding detection task, calculates the confirmation time and the number of valid detection frames, and writes it into the historical alarm record. The confirmation time is obtained by subtracting the timestamp of the frame where the suspected target first appears from the alarm confirmation frame timestamp. The number of valid detection frames is counted according to the aforementioned rules, that is, the number of inference frames that have been used to form alarm confirmation results in the existing alarm confirmation rules of the platform.

[0044] Through the above implementation process, the platform transforms the detection task status in multiple video streams into the current evidence sequence. It then estimates the remaining confirmation time and the remaining number of valid frames in historical segments using similar evidence continuations from historical alarm records. This, combined with queue waiting time and video frame arrival intervals, generates the remaining waiting time and the latest inference start time. Thus, the urgency of detection tasks such as personnel intrusion, smoke and fire detection, vehicle parking violations, area clustering, and equipment malfunctions is no longer determined solely by task category or manual intervention level, but rather by current evidence changes, historical similar alarm processes, video frame arrival status, and inference queue waiting time. Furthermore, frame extraction control no longer uses fixed frame extraction or uniform frame reduction, but instead determines the next frame extraction cutoff time based on the remaining number of valid frames in historical segments. This makes it easier to include key change frames such as changes in smoke and fire diffusion textures, personnel trajectories before and after crossing boundaries, vehicle stopping and starting states, and sudden changes in equipment states. Ultimately, video target detection scheduling can prioritize detection tasks with shorter remaining waiting times and higher risks of delayed or missed detection under limited inference resources, reducing the probability of low-urgency tasks occupying the current inference queue for extended periods and improving key frame coverage and alarm confirmation stability.

[0045] like Figure 2 The diagram shown is a structural schematic of a video target detection scheduling system based on task urgency provided in this application embodiment, including: an evidence sequence formation module, a similar segment extraction module, a waiting time generation module, a frame extraction time generation module, an inference sequence generation module, and a detection result writing module.

[0046] The evidence sequence formation module is used to form the current evidence sequence, queue waiting time, and video frame arrival interval based on the detection task status data of each detection task.

[0047] The similar segment extraction module is used to extract similar evidence continuation segments based on the current evidence sequence, detection task status data, and historical alarm records, and to obtain the remaining confirmation time and the remaining number of valid frames of the historical segment.

[0048] The waiting time generation module is used to generate the remaining waiting time and the latest inference start time based on the remaining confirmation time of historical segments, queue waiting time, and video frame arrival interval.

[0049] The frame extraction time generation module is used to generate the next frame to be submitted for inspection and the next frame extraction deadline based on the remaining waiting time, the number of remaining valid frames in the historical segment, and the timestamps corresponding to the arrived frames.

[0050] The inference sequence generation module is used to generate the current round of inference execution sequence based on the next frame to be submitted for inspection, the latest start time of inference, the detection task status data, and the current inference queue.

[0051] The detection result writing module is used to call the detection model according to the current inference execution sequence, detect the next frame to be submitted for inspection, and write the detection result.

[0052] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the above functions can be divided into different functional modules to complete all or part of the functions described above.

[0053] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0054] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units, located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0055] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0056] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the solution, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0057] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video target detection scheduling method based on task urgency, characterized in that, Includes the following steps: S1: Based on the detection task status data of each detection task, form the current evidence sequence, queue waiting time, and video frame arrival interval; S2: Extract similar evidence continuation segments based on the current evidence sequence, detection task status data and historical alarm records to obtain the remaining confirmation time of historical segments and the remaining number of valid frames of historical segments; S3: Generate the remaining waiting time and the latest inference start time based on the remaining confirmation time of the historical segment, the queue waiting time and the video frame arrival interval; S4: Generate the next frame to be submitted for inspection and the next frame extraction deadline based on the remaining waiting time, the remaining valid frames of the historical segment and the timestamps of the arrived frames; S5: Generate the current round of inference execution sequence based on the next frame to be sent for inspection, the latest start time of inference, the detection task status data, and the current inference queue; S6: Call the detection model according to the current inference execution sequence, detect the next frame to be sent for inspection, and write the detection result.

2. The video target detection scheduling method based on task urgency as described in claim 1, characterized in that, include: The detection task status data includes frame timestamps, detection task category, detection model identifier, camera area level, time when the task enters the inference queue, detection results of inferred frames, and historical inference time of the model. The current evidence sequence is formed by the most recent inferenced frames. Each inferenced frame retains the detection result, task confidence, detection box center position, detection box width and height and frame timestamp corresponding to the detection task category. The detection result includes the target category corresponding to the detection task category. If no corresponding target is detected, it is recorded as an empty frame.

3. The video target detection and scheduling method based on task urgency as described in claim 1, characterized in that, The extraction of similar evidence continuation segments includes: Filter historical alarm records that are identical to the current detection task in terms of detection task category, detection model identifier, and camera area level; When the number of filters is less than the preset number, the camera area level restriction is removed, and historical alarm records with the same detection task category and detection model identifier are retained. If the number of frames filtered is still less than the preset number after the above operations, a cold start process is initiated, where the latest arriving frame is taken as the frame to be sent for inspection, and the latest start time for inference is set to the current time.

4. The video target detection scheduling method based on task urgency as described in claim 1, characterized in that, The process of obtaining the remaining confirmed duration of the historical segment and the remaining valid frames of the historical segment includes: Before entering the cold start process, extract the evidence sequence fragments from the historical alarm records before alarm confirmation, align the evidence sequence fragments with the current evidence sequence according to frame order, compare whether the target appearance status is consistent, and calculate the task confidence difference, the detection box center displacement difference, and the frame arrival interval difference respectively. The comparison results of the target occurrence status are converted into inconsistency markers. The task confidence difference, the detection box center displacement difference, and the frame arrival interval difference are normalized to the same numerical range and then accumulated in frame order to obtain the similarity difference. Evidence sequence segments with similarity differences not greater than the median of all similarity differences are selected as historical segments. The time difference between the last frame of the historical segment and the alarm confirmation frame is read as the remaining time. The number of valid detection frames participating in alarm confirmation between the last frame of the segment and the alarm confirmation frame is counted as the remaining valid detection frames. The median of the remaining time and the remaining valid detection frames of the selected historical segments are taken to obtain the remaining confirmation time and the remaining valid frames of the historical segments.

5. The video target detection and scheduling method based on task urgency as described in claim 1, characterized in that, The generation of the remaining waiting time and the latest start time for inference includes: For detection tasks that have not entered the cold start process, subtract the queue waiting time from the remaining confirmation time of the historical segment; When the arrival interval of the latest video frame is greater than the median arrival interval of frames in the current evidence sequence, subtract the difference between the two. If the above calculation result is less than or equal to zero, the remaining waiting time will be recorded as zero; otherwise, the deducted result will be recorded as the remaining waiting time. Using the current moment as the time base and the remaining waiting time as the time offset, the latest time to start inference is determined.

6. The video target detection scheduling method based on task urgency as described in claim 1, characterized in that, The generation of the next frame to be submitted for inspection and the cutoff time for extracting the next frame include: When the number of remaining valid frames in a historical segment does not meet the conditions for multi-frame segmentation, the latest arrived frame will be used as the next frame to be sent for inspection. When the remaining valid frames of a historical segment meet the conditions for multi-frame segmentation, the remaining waiting time is divided into equal-length time periods according to the remaining valid frames of the historical segment. The next frame extraction deadline is determined from the timestamp of the previous frame to the next frame extraction deadline, and the arrived frame whose timestamp is not later than and closest to the next frame extraction deadline is selected as the next frame to be sent for inspection. The multi-frame segmentation condition is that at least two extraction time periods can be formed within the remaining waiting time based on the remaining valid frames of the historical segment. If no new frames arrive, do not add new frames for this detection task to the current inference queue.

7. The video target detection scheduling method based on task urgency as described in claim 1, characterized in that, The generated current round of inference execution sequence includes: Detection tasks that have reached the deadline for extracting the next frame, as well as detection tasks obtained from cold start processing, are included in this round of sorting. Detection tasks that have not reached the deadline for extracting the next frame are not included in this round of sorting. For the detection tasks included in this round of sorting, they are arranged from earliest to latest according to the latest start time of inference. The order of the sorted detection tasks is used as the inference execution sequence for this round, and the detection tasks not included in this round of sorting are recorded.

8. The video target detection scheduling method based on task urgency as described in claim 7, characterized in that, The generated current round of inference execution sequence also includes: The estimated start time of inference is obtained by adding the historical inference time of the model corresponding to the preceding detection task according to the current inference execution sequence; For the same detection task, one uninferred frame is kept in the current inference queue and replaced with the updated frame generated in this cycle. If the expected start time of inference is later than the latest start time of inference, then record the time difference between the expected start time of inference and the latest start time of inference.

9. The video target detection scheduling method based on task urgency as described in claim 1, characterized in that, The process of performing the detection and writing the detection results includes: Record the actual start time of inference, the actual end time of inference, the target category corresponding to the detection task category, the confidence level, the detection box position, and the alarm confirmation result; The results of this round of testing will be added to the current evidence sequence for the corresponding testing task; Update the historical inference time of the detection model corresponding to the actual inference time; When an alarm confirmation is triggered, the evidence sequence fragment, confirmation time, and number of valid detection frames between the first appearance of the suspected target and the alarm confirmation frame are extracted from the current evidence sequence of the corresponding detection task, and written into the historical alarm record.

10. A system applying the video target detection scheduling method based on task urgency as described in any one of claims 1-9, characterized in that, include: The module includes: evidence sequence formation module, similar fragment extraction module, waiting time generation module, frame extraction time generation module, inference sequence generation module, and detection result writing module. The evidence sequence forming module is used to form the current evidence sequence, queue waiting time and video frame arrival interval based on the detection task status data of each detection task. The similar segment extraction module is used to extract similar evidence continuation segments based on the current evidence sequence, detection task status data and historical alarm records, and to obtain the remaining confirmation time of the historical segment and the remaining number of valid frames of the historical segment. The waiting time generation module is used to generate the remaining waiting time and the latest inference start time based on the remaining confirmation time of the historical segment, the queue waiting time and the video frame arrival interval. The frame extraction time generation module is used to generate the next frame to be submitted for inspection and the next frame extraction deadline based on the remaining waiting time, the remaining valid frames of the historical segment and the timestamps corresponding to the arrived frames. The inference sequence generation module is used to generate the current round of inference execution sequence based on the next frame to be sent for inspection, the latest inference start time, the detection task status data, and the current inference queue. The detection result writing module is used to call the detection model according to the current inference execution sequence, detect the next frame to be sent for inspection, and write the detection result.

Images