Multi-stream collaborative scheduling dynamic frame extraction method, system and device and storage medium

Abstract

The invention discloses a multi-stream collaborative scheduling dynamic frame extraction method, system and device and a storage medium, and relates to the field of computer vision technology and monitoring video technology, and the method comprises the steps: determining a process number and a thread number based on a monitoring end network resource sensing parameter obtained by a processor executing scheduling, constructing a task group, and sending the task group to a server; and distributing the monitoring objects to the task groups according to a greedy strategy based on the detection time and the distribution weight of the monitoring objects. And based on a frame taking task priority queue in the constructed task group, executing a frame taking task according to the bandwidth occupancy rate of the monitoring end and a frame taking task waiting time threshold value, and controlling a frame taking process in combination with a code stream switching condition judged by frame taking time consumption and a retry strategy of stream taking failure. A combined sampling sequence is generated based on a multi-algorithm period of the same camera which has executed a frame taking task, frame extraction is executed at a combined point, and repeated frame taking is reduced through a combined coverage interval. According to the method provided by the invention, the video stream transmission stability, the picture quality and the network utilization rate are improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision technology and surveillance video technology, and specifically to a multi-stream collaborative scheduling dynamic frame extraction method, system, device and storage medium. Background Art

[0002] In modern industrial production environments, the deployment density and coverage of surveillance cameras continue to expand. This exponential growth in data volumes not only places enormous pressure on storage systems but also poses severe challenges to data transmission networks. Traditional video transmission architectures are experiencing significant performance bottlenecks when faced with such massive data flows.

[0003] In recent years, breakthroughs in deep learning technology have enabled significant progress in the application of machine vision in industrial inspection. In modern industrial production, machine vision algorithms have expanded from simple product appearance inspection to complex production process monitoring, equipment status analysis, and human behavior recognition. In particular, algorithms such as object detection and anomaly recognition based on convolutional neural networks play an irreplaceable role in quality control and production safety. However, these advanced algorithms typically require high-definition video streams as input and place high demands on the integrity and continuity of video frames, further exacerbating data transmission pressures.

[0004] Unlike commercial network environments, network infrastructure at industrial sites is often constrained by multiple factors, including cost, environmental factors, and compatibility. Most industrial sites still utilize 100M-level network bandwidths, and must simultaneously support multiple services, including video surveillance, equipment control, and data acquisition. Under these bandwidth-constrained conditions, existing technologies struggle to simultaneously meet the requirements for real-time transmission of multiple channels of high-definition video and the application of machine vision algorithms. Of particular note, when multiple detection algorithms need to process video streams from different scenarios in parallel, the conflict over network bandwidth allocation becomes even more pronounced. Common solutions, such as reducing video resolution or frame rate, directly impact the accuracy and reliability of the detection algorithm. This compromise makes it difficult to meet the stringent detection accuracy requirements in industrial scenarios.

[0005] Against the backdrop of a surge in industrial surveillance video data and growing demand for machine vision inspection, how to overcome the constraints of limited bandwidth and achieve efficient inspection of large quantities of surveillance videos has become a key technical challenge that needs to be urgently addressed in the process of industrial intelligent transformation. Summary of the Invention

[0006] In view of the above-mentioned problems, the present invention is proposed.

[0007] Therefore, the technical problem solved by the present invention is: the amount of industrial monitoring video data has increased sharply and the demand for machine vision detection has increased. Traditional technical solutions have shown obvious performance bottlenecks in the face of huge data streams, and it is difficult to simultaneously meet the real-time transmission of multiple channels of high-definition video and the application requirements of machine vision algorithms.

[0008] To solve the above technical problems, the present invention provides the following technical solutions: a multi-stream collaborative scheduling dynamic frame extraction method, including determining the number of processes and threads based on the network resource perception parameters of the monitoring end obtained by the processor executing scheduling, constructing task groups, and allocating the monitoring objects to the task groups according to a greedy strategy based on the detection time and allocation weight of the monitoring objects.

[0009] Based on the priority queue of the frame acquisition task within the established task group, the frame acquisition task is executed according to the bandwidth occupancy rate of the monitoring end and the waiting time threshold of the frame acquisition task. The frame acquisition process is controlled by combining the stream switching conditions judged by the frame acquisition time consumption and the retry strategy for stream acquisition failure.

[0010] A merged sampling sequence is generated based on multiple algorithm cycles of the same camera that has executed the frame acquisition task, and frame extraction is performed at the merge point, reducing repeated frame acquisition by merging the coverage interval.

[0011] The frame acquisition process is controlled by combining the code stream switching condition based on the frame acquisition time judgment and the retry strategy for failed stream acquisition, including: When the frame acquisition time exceeds the preset number of times continuously and the time each time is greater than the frame acquisition time threshold, the processor executing the scheduling switches the video stream of the current camera from the main code stream to the sub-code stream and records the switching status in the task execution log.

[0012] For tasks that fail to obtain the stream, the waiting time for the next retry is calculated using the exponential backoff method, and the frame acquisition request is re-initiated within the maximum number of retries specified. After the maximum number of retries is exceeded, the processor executing the scheduling will mark the task as failed and terminate the task scheduling.

[0013] As a preferred solution of the multi-stream collaborative scheduling dynamic frame extraction method described in the present invention, the construction of task groups includes calculating the number of processes based on the number of CPU cores of the processor executing the scheduling, calculating the number of threads based on the bandwidth upper limit of the monitoring end, forming task groups with threads as units, and performing statistics based on the detection time of each monitored object collected by the monitoring end, taking the maximum value of the monitoring object association algorithm cycle for the detection time, and using the detection time as the allocation weight, and allocating the monitored objects to the task groups by executing a greedy strategy through the processor executing the scheduling. When the maximum times of the candidate groups are equal, the task groups are selected in combination with the principle of dispersion of the stations to which the monitored objects belong.

[0014] As a preferred solution of the multi-stream collaborative scheduling dynamic frame extraction method of the present invention, wherein: The method of allocating monitored objects to task groups according to a greedy strategy based on the detection time and allocation weight of the monitored objects includes: after the processor executing scheduling obtains the detection time of all monitored objects, the detection time is used as a sorting basis to select the monitored objects in sequence and allocate them to task groups, and each time the task group with the smallest current cumulative time is selected as the target group. When the cumulative time of the task groups is equal, the processor executing scheduling calculates the distribution of the stations to which the monitored objects belong within the task group and compares them, selects the task group with a dispersed distribution of stations, and records the allocation results as a task group allocation table.

[0015] As a preferred solution of the multi-stream collaborative scheduling dynamic frame extraction method described in the present invention, the execution of the frame extraction task according to the monitoring end bandwidth occupancy rate and the frame extraction task waiting time threshold includes arranging the frame extraction tasks in the task group into a priority queue according to the preset task priority parameters. When the monitoring end bandwidth occupancy rate exceeds the preset bandwidth occupancy rate threshold, only the low-priority tasks with a priority not higher than the high-priority threshold are scheduled to perform the frame extraction operation. For the low-priority tasks, the frame extraction is delayed when the waiting time does not exceed the detection period. When the waiting time exceeds the detection period, the frame extraction request of the task is discarded, and the processor that executes the scheduling records the discard log.

[0016] As a preferred embodiment of the multi-stream collaborative scheduling dynamic frame extraction method described in the present invention, wherein: the stream switching condition combined with the frame extraction time judgment and the retry strategy for stream extraction failure control frame extraction process includes, during the continuous frame extraction task execution process, the processor executing the scheduling records the start time and end time for each frame extraction task, calculates the time consumption of the frame extraction task, and compares the time consumption result with the preset frame extraction time consumption threshold. When the number of times that the time consumption in the continuous frame extraction task exceeds the frame extraction time consumption threshold reaches the preset continuous number threshold, the processor executing the scheduling switches the video stream code stream format of the current camera from the main code stream to the sub-code stream before the next frame extraction cycle arrives, and immediately updates the code stream status information and switching trigger condition record in the task execution log after the switching is completed. During the execution of the frame extraction task, when a stream extraction failure is detected, the processor executing the scheduling calculates the next retry waiting time according to the preset exponential backoff algorithm, and the waiting time for each retry is the product of the previous retry waiting time and the backoff multiplier until the preset maximum retry waiting time threshold is reached, and the video stream request of the camera is suspended during the retry interval. During the retry process, the processor that executes the schedule verifies the number of retries that the current task has executed before initiating each retry. When the cumulative number of retries reaches the preset maximum number of retries, the frame acquisition task is immediately terminated and the task status is updated to failure.

[0017] As a preferred embodiment of the multi-stream collaborative scheduling dynamic frame extraction method described in the present invention, generating a merged sampling sequence based on multiple algorithm cycles of the same camera that has executed a frame extraction task includes: the processor executing the scheduling obtains all algorithm sampling cycles configured for the same camera, calculates the greatest common divisor and least common multiple of all sampling cycles, and generates a unified sampling time sequence based on the greatest common divisor if an integer divisibility relationship exists; otherwise, separate sampling time sequences are generated. The time sequences are merged to generate a merged sampling sequence containing all sampling time points.

[0018] As a preferred solution of the multi-stream collaborative scheduling dynamic frame extraction method described in the present invention, the method of reducing repeated frame extraction by merging coverage intervals includes: the processor executing scheduling identifies the coverage relationship between adjacent time points in the merged sampling sequence, marks the coverage relationship as a merged coverage interval, performs a frame extraction operation at the starting time point of the merged coverage interval, and updates the status of the coverage interval after the extraction is completed.

[0019] Another object of the present invention is to provide a multi-stream collaborative scheduling dynamic frame extraction system, which can be used through priority queues and dynamic frame extraction modules to execute frame extraction tasks based on the priority queues of frame extraction tasks within the established task group according to the bandwidth occupancy rate of the monitoring end and the waiting time threshold of the frame extraction task, and control the frame extraction process in combination with the code stream switching conditions for judging the frame extraction time consumption and the retry strategy for failed stream extraction, thereby solving the problem that the amount of industrial monitoring video data has increased sharply and the demand for machine vision detection is growing. Traditional technical solutions have shown obvious performance bottlenecks in the face of huge data streams, and it is difficult to simultaneously meet the real-time transmission of multiple high-definition videos and the application requirements of machine vision algorithms.

[0020] As a preferred solution of the multi-stream collaborative scheduling dynamic frame extraction system described in the present invention, it includes: a task grouping and greedy allocation module, a priority queue and dynamic frame extraction module, and a merged sampling and coverage optimization module.

[0021] The task grouping and greedy allocation module is used to determine the number of processes and threads based on the monitoring end network resource perception parameters obtained by the processor executing scheduling, construct task groups, and allocate the monitoring objects to task groups according to the greedy strategy based on the detection time and allocation weight of the monitoring objects.

[0022] The priority queue and dynamic frame acquisition module are used to execute the frame acquisition task based on the frame acquisition task priority queue within the constructed task group according to the monitoring end bandwidth occupancy rate and the frame acquisition task waiting time threshold, and control the frame acquisition process in combination with the code stream switching conditions judged by the frame acquisition time consumption and the retry strategy for stream acquisition failure.

[0023] The merged sampling and coverage optimization module is used to generate a merged sampling sequence based on multiple algorithm cycles of the same camera that has performed the frame acquisition task, and perform frame extraction at the merge point, thereby reducing repeated frame acquisition by merging coverage intervals.

[0024] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a step of a multi-stream collaborative scheduling dynamic frame extraction method.

[0025] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a multi-stream collaborative scheduling dynamic frame extraction method.

[0026] Beneficial effects of the present invention: The multi-stream collaborative scheduling dynamic frame extraction method provided by the present invention obtains real-time network resource occupancy by performing bandwidth monitoring and network status acquisition at the video stream input end, thereby realizing dynamic perception of the network environment by the input end. The purpose of this step is to provide real-time, accurate basic data such as network bandwidth and delay for subsequent frame extraction strategies, avoiding frame loss or freeze problems caused by fixed frame extraction strategies when network status fluctuates. This can ensure the continuity and stability of video transmission, and ultimately achieve the beneficial effect of adaptively adjusting the frame extraction strategy under different network conditions and improving the quality of video transmission.

[0027] By combining network bandwidth prediction with video content feature analysis, a dynamic frame extraction strategy is generated, matching the frame extraction frequency with the rate of change of video content. This step considers both network transmission capacity and the criticality of the video content, prioritizing the retention of high-information frames when bandwidth is insufficient, while maintaining a high frame rate when bandwidth is sufficient. This effectively improves the fidelity of key scenes and avoids information loss caused by excessive frame extraction in important scenes, ultimately ensuring core image quality and user viewing experience within limited network resources.

[0028] By implementing a coordinated optimization of dynamic frame extraction and video stream encoding, the frame extraction strategy is directly applied to the video encoding process, achieving orderly screening and compression encoding of video frames before transmission. This step controls the generation of redundant data during the encoding phase, reducing the amount of invalid bitstream transmission at the source. It also forms a closed-loop feedback mechanism with changes in network status, allowing for synchronous optimization of encoding parameters and frame extraction strategies. This not only reduces network load but also alleviates decoding pressure at the receiving end, ultimately achieving the beneficial effect of balancing smoothness, clarity, and resource utilization in the end-to-end transmission link. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 A schematic diagram of priority selection for a dynamic frame extraction method for multi-stream collaborative scheduling provided in Example 1 of the present invention.

[0031] Figure 2 A schematic diagram of the on-site network deployment of a multi-stream collaborative scheduling dynamic frame extraction method provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0032] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0033] Example 1, with reference to Figure 1 , as an embodiment of the present invention, provides a multi-stream collaborative scheduling dynamic frame extraction method, comprising: S1: Determine the number of processes and threads based on the network resource perception parameters of the monitoring end obtained by the processor executing the scheduling, build task groups, and assign the monitoring objects to the task groups according to the greedy strategy based on the detection time and allocation weight of the monitoring objects.

[0034] The number of processes is calculated based on the number of CPU cores of the processor executing scheduling, and the number of threads is calculated based on the bandwidth upper limit of the monitoring end. Task groups are formed using threads as units, and statistics are collected based on the detection time of each monitored object collected by the monitoring end. The detection time takes the maximum value of the monitoring object association algorithm cycle, and the detection time is used as the allocation weight. The processor executing scheduling executes a greedy strategy to allocate the monitored objects to the task group.

[0035] Furthermore, the number of processes and threads is calculated by the scheduling processor reading the number of CPU cores C of the processor itself. To reduce the process context switching overhead and ensure that each process can stably occupy core resources, the scheduling processor calculates the number of processes according to the one-third rule: that is, the number of CPU cores is divided by 3 and rounded down to get the number of processes.

[0036] The number of threads is calculated by the scheduling processor, which reads the upper limit of available bandwidth (B) (in Mbps) at the monitoring station. To ensure that single-threaded traffic does not consume bandwidth and facilitates inter-group balancing, the scheduling processor calculates the number of threads using the tenth rule: dividing the upper limit of available bandwidth by 10 and rounding down to the nearest integer.

[0037] The task grouping is constructed by creating an equal number of task groups in units of threads for the processor that performs scheduling. Each group corresponds to a parallel frame extraction execution unit, which is used to carry the frame extraction tasks of multiple monitoring objects.

[0038] A preferred specific solution for task grouping includes the system automatically determining the number of processes and threads based on resource-aware parameters to balance load stability and maximize resource utilization. The calculation formula is as follows: The number of processes is determined by the number of CPU cores. The optimization rule is to divide the number of processes by 3, rounded up to the nearest integer. The division by 3 is based on experimental best practices to ensure that each process fully utilizes CPU resources without incurring context switching overhead.

[0039] Number of threads: Determined by the bandwidth limit. The optimization rule is to round the number of threads to the bandwidth limit divided by 10. The factor of 10 is obtained through bandwidth load testing.

[0040] Each thread corresponds to a task group, which is used to carry out frame extraction tasks for multiple monitoring objects. The task group is defined as: ; in, It is A collection of monitoring objects for a thread group. Indicates the total number of threads.

[0041] For example, in a typical scenario with 32 CPU cores and a bandwidth limit of 50 Mbps, P = 10 and T = 5.

[0042] The system creates 10 processes, each process creates 5 threads, for a total of 50 threads (task grouping).

[0043] The detection time is determined by the processor that performs scheduling traversing the detection algorithm set configured for each monitoring object, and taking the maximum value of the detection cycle as the detection time of the monitoring object.

[0044] Furthermore, the detection time is the basic parameter for intelligent allocation. The detection time of each monitored object is maintained in the monitoring device management program. For monitoring objects , its time The detection cycle is determined based on all associated detection algorithms: Set monitoring object The configured detection algorithm set is , each algorithm The detection cycle is (Unit: seconds).

[0045] The time used by the monitoring object Defined as:

[0046] That is, the maximum period is used as the time benchmark. The time statistics are stored in the time matrix middle.

[0047] in, Indicates the monitored object No. A detection algorithm. Indicates the monitored object The number of associated detection algorithms. Indicates the total number of monitored objects.

[0048] For example, if a camera is configured with face recognition (cycle 3s) and behavior analysis (cycle 5s), 5s.

[0049] Furthermore, the specific allocation process of the greedy strategy involves the scheduling processor summarizing the detection times of all monitored objects and generating a record for each monitored object, including the station ID and detection time. To prioritize monitored objects with longer detection times, the scheduling processor generates an allocation list in descending order of detection time. The current monitored object is retrieved sequentially, and the scheduling processor tentatively adds it to each task group, calculating the cumulative time of the group after the group is added and the maximum cumulative time of all groups. Among all candidate groups, the group that minimizes the maximum cumulative time of all groups is selected as the first choice. If multiple groups reach the same minimum value, the station dispersion principle is used to further identify the parallel candidates.

[0050] When the maximum time of candidate groups is equal, the task group is selected based on the dispersion principle of the monitoring object's stations. After the scheduling processor obtains the detection time of all monitoring objects, it uses the detection time as the sorting criterion, selects the monitoring objects in turn, and assigns them to task groups. For each assignment, the task group with the lowest cumulative time is selected as the target group.

[0051] The allocation algorithm is based on time statistics and a greedy algorithm. The goal is to minimize the maximum time of all task groups. The input of the algorithm is the time matrix and the number of group targets, and the output is a group set. .

[0052] Monitoring object list Time weight (i.e. time to weight). Grouping Initially empty set ( ). For each monitoring object in L Find candidate groups: compute potential maximum time weight of adding i to each group ; If is empty, then = 1 .

[0053] Select candidate groups: let the set of candidate group indices be , be the k value with minimum , typically there are multiple candidate groups (e.g. first candidate , second candidate , etc.).

[0054] If adding to the first candidate group results in a larger proportion of the same station monitoring in than the second candidate group, then select the second candidate group . If the remaining candidate groups result in the same outcome, then select the first candidate group. Add to the selected group .

[0055] The selection rule of the greedy algorithm can be expressed as: ; where denotes the time weight of monitoring object . denotes the th group. denotes the number of allocated monitoring objects in group . and denote the specific set of candidate groups.

[0056] Further, the preferred solution of the task grouping according to the station dispersion principle specifically comprises, when there are multiple candidate groups that have the same impact on the maximum cumulative time weight of all groups after adding the current monitoring object (tied for best), the station dispersion principle is used for selection.

[0057] ​The processor executing scheduling counts the number of assigned monitoring objects at the same station as the current monitoring object in each candidate group, or their proportion to the total number within the group. Candidate groups with smaller statistical values ​​are preferentially selected to ensure that the monitoring objects at the same station are dispersed as much as possible into different groups. If there is still a tie, the processor executing scheduling selects the first candidate group according to a fixed stability rule (set as a fixed group sequence number in the present invention) to ensure allocation certainty. By introducing a constraint that prioritizes the smaller number / proportion of objects at the same station among the tied candidates, the distribution of stations across groups is more balanced, reducing the risk of sudden congestion caused by the clustering of objects at the same station within a particular group.

[0058] Furthermore, a preferred solution of using detection time as a sorting basis to sequentially select monitoring objects and assign them to task groups specifically includes: Establish an allocation list: The processor that performs scheduling first sorts all monitored objects from large to small according to the detection time to form a list to be allocated.

[0059] Loop Allocation: Starting from the beginning of the list, each monitored object is removed one by one, and a candidate grouping process is performed. The group that minimizes the global maximum cumulative time is selected. If there is a tie, the tie is broken according to the principle of dispersion. Once the target group is determined, it is immediately added to the group, and the cumulative time and station distribution statistics within the group are updated.

[0060] Continuous Update: After each monitoring object is assigned, the scheduling processor will update the cumulative time of each group and the number / ratio of each station in each group, so that the subsequent object exploration data and allocation decisions are always based on the latest status.

[0061] Record structure: After completing the round of allocation, the processor that executes the scheduling generates a task group allocation table, which lists the following items: monitoring object ID, target group ID, cumulative time after the group allocation, and the number of identical stations in the group, which is directly referenced by subsequent frame scheduling.

[0062] When the cumulative time of the task groups is equal, the processor executing the scheduling calculates the distribution of the stations to which the monitored objects belong in the task groups and compares them, selects the task groups with dispersed stations, and records the allocation results as a task group allocation table.

[0063] Furthermore, dispersion means that within the target group, the proportion of objects belonging to the same station as the object to be assigned is lower.

[0064] It should be noted that S1 first determines the number of processes and threads based on the number of CPU cores and the bandwidth limit through the processor that executes the schedule, constructs task groups, and uses the maximum value of the monitoring object detection algorithm cycle as the detection time. A greedy strategy is used to prioritize the allocation of objects with high detection time to balance the load of each group. In the case of parallel deployment, the principle of site dispersion is introduced, and priority is given to groups with a lower proportion of the same site to reduce the risk of concentrated congestion. During the allocation process, S1 continuously updates the group status and records the task group allocation table to provide accurate data support for subsequent scheduling.

[0065] S2: Based on the priority queue of the frame acquisition task within the established task group, the frame acquisition task is executed according to the bandwidth occupancy rate of the monitoring end and the waiting time threshold of the frame acquisition task. The frame acquisition process is controlled by combining the stream switching conditions determined by the frame acquisition time and the retry strategy for failed stream acquisition.

[0066] The frame-fetching tasks in the task group are arranged into a priority queue according to the preset task priority parameters. When the bandwidth occupancy rate of the monitoring end exceeds the preset bandwidth occupancy rate threshold, only low-priority tasks whose priority is not higher than the high-priority threshold are scheduled to perform frame-fetching operations. For low-priority tasks, frame fetching is delayed when the waiting time does not exceed the detection period. When the waiting time exceeds the detection period, the frame-fetching request of the task is discarded, and the processor that executes the scheduling records the discard log.

[0067] Furthermore, the processor that executes the scheduling sets the task priority parameter to an integer from 1 to 3. The smaller the value, the higher the priority. Level 1 tasks correspond to critical security monitoring services, level 2 tasks correspond to important security detection services, and level 3 tasks correspond to general detection services. The preset threshold of the bandwidth occupancy rate of the monitoring end is set to 70%. When the bandwidth occupancy rate exceeds this, the execution of low-priority tasks is restricted. The high-priority threshold is set to 2, that is, when the bandwidth occupancy rate is higher than 70%, only tasks with a priority value of 1 or 2 are executed. Reference for priority selection Figure 1 .

[0068] When the bandwidth utilization rate is higher than 70%, only high-priority tasks are executed. Low-priority task queuing delay: If the waiting time of a low-priority task exceeds its detection period, the task in that round is abandoned (the frame is discarded) and a log is recorded.

[0069] Otherwise, execute all tasks, expressed as: ; in, Indicates the The Boolean result of whether a task is executed. Indicates the bandwidth usage of the monitoring end. Indicates the The Boolean result of whether a task is executed. Indicates the high priority threshold.

[0070] During the execution of continuous frame grabbing tasks, the processor that executes the scheduling records the start time and end time for each frame grabbing task, calculates the time consumption of the frame grabbing task, and compares the time consumption result with a preset frame grabbing time consumption threshold.

[0071] Furthermore, the detection cycle for each frame capture task is determined by its configured detection algorithm. For example, the detection cycle for open flame detection can be set to 3 seconds, for hardhat detection to 5 seconds, and for perimeter security to 10 seconds. The frame capture time threshold is set to 150 milliseconds. If the time from the start of a single frame capture to the acquisition of a complete video frame exceeds this value, the frame capture time is considered exceeded.

[0072] When the number of times that the time consumed in continuous frame acquisition tasks exceeds the frame acquisition time threshold reaches the preset continuous number threshold, the processor executing the scheduling will switch the video stream format of the current camera from the main stream to the sub-stream before the next frame acquisition cycle arrives, and immediately update the stream status information and switching trigger condition record in the task execution log after the switch is completed.

[0073] When the frame acquisition time increases, the system automatically switches the video stream type (such as switching from the main stream to a sub-stream) to reduce bandwidth usage. The switching strategy is based on timeout detection: Switching condition: Set the frame time sequence (Unit: ms). If you take frames three times in a row If it is greater than 150ms, the switching action is triggered: ; Among them, 150ms is the program setting value. Indicates the trigger condition for stream switching, which is expressed as three consecutive times >150ms.

[0074] Furthermore, the consecutive times threshold is set to 3. If three consecutive frame acquisitions each take longer than 150 milliseconds, the scheduling processor switches the current camera's video stream format from main stream to sub-stream before the next frame acquisition cycle begins. The switching process involves sending a stream parameter update command to the camera or access point to adjust the stream configuration from main stream parameters to sub-stream parameters. Upon receiving a confirmation response from the device, the internally recorded stream type status is updated and the stream type, triggering conditions, and switching time before and after the switch are recorded in the task execution log.

[0075] During the execution of the frame acquisition task, when a failure in stream acquisition is detected, the processor executing the scheduling calculates the next retry waiting time according to the preset exponential backoff algorithm. The waiting time for each retry is the product of the previous retry waiting time and the backoff multiplier until the preset maximum retry waiting time threshold is reached, and the camera's video stream request is suspended during the retry interval.

[0076] Further, the stream acquisition failure refers to any of the following situations occurring in the process of the processor performing scheduling initiating a video stream request to the camera and preparing to receive data: connection establishment stage timeout or authentication failure. No video frame data is received within the specified video frame reception timeout. The received video frame fails three times in a row in the data integrity check, and fails to obtain a valid frame in the retry.

[0077] When the stream acquisition failure is detected, the processor performing scheduling calculates the waiting time for the next retry. The waiting time for the first retry is 10 seconds, and the waiting time for each subsequent retry is twice the waiting time of the previous retry. For example, 10 seconds for the first time, 20 seconds for the second time, and 40 seconds for the third time. During the waiting period, the sending of frame acquisition requests to the camera is suspended to reduce bandwidth pressure.

[0078] During the retry process, the processor performing scheduling verifies the number of retries performed by the current task before initiating each retry. When the cumulative number of retries reaches the preset maximum number of retries, the frame acquisition task is immediately terminated, and the task status is updated to failure.

[0079] Further, the maximum retry waiting time threshold is set to 40 seconds, i.e., when the retry waiting time calculation result exceeds 40 seconds, the waiting time is no longer increased. The maximum number of retries is set to 3, i.e., after three consecutive retries fail, the processor performing scheduling immediately terminates the frame acquisition task, and records the failure reason, the number of retries, and the last waiting time in the task execution log.

[0080] It should be noted that S2 dynamically controls the execution of the frame acquisition task based on the grouped tasks through priority queues and bandwidth occupancy, ensuring that high-priority tasks are processed first. In combination with the judgment of task waiting time and detection period, invalid frame acquisition is avoided. When the continuous frame acquisition time consumption exceeds the threshold, the stream is actively switched to reduce bandwidth occupancy, and when the stream acquisition fails, the retry frequency is controlled through exponential backoff retry, and finally the task is terminated after exceeding the maximum number of times or the waiting time and the log is recorded, realizing the stability and adaptability of task scheduling, providing double protection of bandwidth and execution efficiency for the overall frame extraction process.

[0081] S3: Generate a merged sampling sequence based on the same camera of the executed frame acquisition task, and perform frame extraction at the merging point to reduce repeated frame acquisition by merging the coverage interval.

[0082] The processor performing scheduling obtains all algorithm sampling periods configured for the same camera, calculates the greatest common divisor and the least common multiple of all sampling periods, generates a unified sampling time sequence according to the greatest common divisor if there is a division relationship, or generates respective sampling time sequences if there is no division relationship. Merge the time sequences to generate a merged sampling sequence containing all sampling time points.

[0083] Furthermore, the processor that performs scheduling first obtains the sampling periods of all detection algorithms configured on the same camera. These sampling periods are usually fixed time intervals between algorithms, such as 3 seconds for algorithm A, 6 seconds for algorithm B, and 9 seconds for algorithm C. The existence of a divisibility relationship means that among these sampling periods, at least one sampling period can be divided by another sampling period, and the result of the divisibility is an integer. For example, 6 seconds can be divided by 3 seconds, and 9 seconds can be divided by 3 seconds. At this time, the processor that performs scheduling will calculate the greatest common divisor of all sampling periods, and use the greatest common divisor as a unified sampling time step to generate a unified sampling time series covering all algorithms. For example, if the greatest common divisor in the above example is 3 seconds, the generated sampling time series is 0 seconds, 3 seconds, 6 seconds, 9 seconds, 12 seconds... and so on.

[0084] If there's no integer divisibility relationship, meaning the result of dividing any two sampling periods is not an integer (for example, 4 seconds for algorithm A, 6 seconds for algorithm B, and 10 seconds for algorithm C), the scheduling processor generates a separate sampling time sequence for each algorithm. For example, the sequence for the 4-second algorithm is 0 seconds, 4 seconds, 8 seconds, and so on; the sequence for the 6-second algorithm is 0 seconds, 6 seconds, 12 seconds, and so on; and the sequence for the 10-second algorithm is 0 seconds, 10 seconds, 20 seconds, and so on. These sequences are then merged to create a combined sampling sequence containing all sampling time points. The combined time points are then sorted in ascending order to eliminate duplicates.

[0085] The processor that performs scheduling identifies the coverage relationship between adjacent time points in the merged sampling sequence, marks the coverage relationship as a merged coverage interval, performs a frame extraction operation at the starting time point of the merged coverage interval, and updates the status of the coverage interval after the extraction is completed.

[0086] After obtaining the merged sampling sequence, the scheduling processor analyzes whether there is an overlap between adjacent time points in the sequence. This means that two or more algorithms need to access exactly the same frame data at the same time, or the time interval is less than the minimum acceptable frame interval. The range of time points that are determined to be overlapping is called the merged coverage interval.

[0087] Within each merged coverage interval, the processor executing the schedule only performs a frame extraction operation once at the first time point in the interval. The specific execution method of frame extraction is: when the time point is reached, the original video frame data of the corresponding time point is requested from the camera or video stream cache. After receiving the complete frame data, it is copied to the buffer of each algorithm task that requires this frame, so that the frame can be used by multiple algorithms at the same time, thereby avoiding repeated frame extraction requests for the same video stream in the same time period. After the extraction is completed, the processor executing the schedule will mark the coverage interval as processed and update the execution status of the interval in the task scheduling record, so that the completed interval can be skipped in the subsequent scheduling process to reduce repeated operations.

[0088] Set monitoring object Configure multiple algorithms, each with a detection cycle . The goal is to generate a combined sampling sequence , minimize the total number of sampling points: If the cycle If there is an inclusion relationship (i.e. one period is a multiple of another period), the merge is based on the least common multiple: ; in, is the greatest common divisor, is the least common multiple.

[0089] Otherwise, the sequence is calculated independently for each period and the sampling points are reduced by merging the covering intervals.

[0090] Algorithm Description: Enter a list of algorithm cycles: ; Calculate the base interval: ; Generate a sequence of sampling points, if All elements satisfy divisibility ,but: ; Otherwise, calculate each The sampling point sequence is then merged: ; Where T is the monitoring period.

[0091] The frame extraction task only performs At the point in time, the system records the merge log.

[0092] Example 1: Cycle 5s, If 10s, , , (5s interval).

[0093] Example 2: 3s, is 7s, no common factors, then Calculate independently and combine: , covering the needs of both models.

[0094] It should be noted that S3 leverages the sampling cycle relationships of multiple algorithms on the same camera to unify or merge sampling time points, reducing repeated frame acquisition operations. By determining the divisibility relationship between the sampling cycles, a unified sampling sequence can be generated under the greatest common denominator. If the sampling cycles are not divisible, multiple sequences are merged and duplicates are removed. The merged sequence is analyzed for coverage relationships, and frame extraction is performed only once at the beginning of the coverage interval. The results are distributed to multiple algorithm tasks. This not only avoids bandwidth waste caused by repeated stream acquisition, but also ensures that each algorithm can obtain data on demand, thereby improving frame acquisition efficiency and resource utilization.

[0095] Example 2 is an embodiment of the present invention, which provides a dynamic frame extraction method for multi-stream collaborative scheduling. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0096] A new energy power plant deployed an industrial AI visual inspection product, with nearly a thousand surveillance camera terminals connected throughout the plant. Supporting multiple scene inspection tasks simultaneously while the dedicated 50M bandwidth of the site video network was temporarily immutable was a significant challenge. The experimental conditions were as follows: Test platform: Uses multi-camera video stream input with a resolution of 1920×1080 and a frame rate of 30fps.

[0097] Network environment: Bandwidth limit 100Mbps, latency 50ms.

[0098] Hardware configuration: Processor model Intel Xeon E5-2620v4, memory 64GB.

[0099] Algorithm Configuration: Three detection algorithms are available, including open flame detection, hard hat detection, and perimeter security. The open flame detection cycle is 3 seconds, accounting for 20% of the total; the hard hat detection cycle is 5 seconds, accounting for 50% of the total; and the perimeter security cycle is 10 seconds, accounting for 30% of the total.

[0100] The priority configuration is: open flame detection priority 1, hard hat detection priority 2, and perimeter security priority 3.

[0101] The experiment set up three sampling strategies for comparison: Independent sampling strategy: Each algorithm takes frames independently according to its own sampling period and does not merge them.

[0102] Fixed-period unified sampling strategy: frames are taken using the greatest common divisor as the unified sampling period.

[0103] The multi-stream collaborative scheduling strategy of the present invention generates a merged sampling sequence according to the steps of the present invention, and performs a frame extraction within the merged coverage interval.

[0104] The experimental results are shown in Table 1.

[0105] Table 1 Comparison between our invention and traditional solutions Index Traditional solution Invention solution Promotion effect Total bandwidth occupation 1.8 Gbps 45 Mbps Reduced by 97.5% Average detection delay 500 ms 80 ms Reduced by 84% High priority task delay 200 ms 50 ms Reduced by 75% CPU utilization 40% 85% Increased by 112.5% Task discard rate 15% 2% Reduced by 86.7% Hardware cost Need special server cluster Single T4 + 32-core server Reduced by 30%

[0106] The data in Table 1 demonstrates that the proposed multi-stream collaborative scheduling method is effective in reducing task abandonment rates and bandwidth usage, while also optimizing task processing latency. This demonstrates that, in multi-algorithm, multi-cycle video analysis tasks, this method can effectively improve overall system efficiency and reduce resource consumption.

[0107] Example 3 is an embodiment of the present invention, which provides a multi-stream collaborative scheduling dynamic frame extraction system, including a task grouping and greedy allocation module, a priority queue and dynamic frame extraction module, and a merged sampling and coverage optimization module.

[0108] The task grouping and greedy allocation module is used to determine the number of processes and threads based on the network resource perception parameters of the monitoring end obtained by the processor executing the scheduling, build task groups, and allocate the monitoring objects to the task groups according to the greedy strategy based on the detection time and allocation weight of the monitoring objects.

[0109] The priority queue and dynamic frame acquisition module is used to execute the frame acquisition task based on the frame acquisition task priority queue within the established task group, according to the monitoring end bandwidth occupancy rate and the frame acquisition task waiting time threshold, and control the frame acquisition process in combination with the stream switching conditions judged by the frame acquisition time consumption and the retry strategy for stream acquisition failure.

[0110] The merged sampling and coverage optimization module is used to generate a merged sampling sequence based on the multi-algorithm cycle of the same camera that has performed the frame acquisition task, and perform frame extraction at the merge point, reducing repeated frame acquisition by merging the coverage interval.

Claims

1. A dynamic frame extraction method for multi-stream collaborative scheduling, characterized in that: include: The number of processes and threads is determined based on the network resource perception parameters of the monitoring end obtained by the processor executing the schedule, and task groups are constructed. The monitoring objects are assigned to the task groups according to the greedy strategy based on the detection time and allocation weight of the monitoring objects. Based on the priority queue of the frame acquisition task within the established task group, the frame acquisition task is executed according to the bandwidth utilization rate of the monitoring end and the waiting time threshold of the frame acquisition task. The frame acquisition process is controlled by combining the stream switching conditions based on the frame acquisition time judgment and the retry strategy for stream acquisition failure. Generate a merged sampling sequence based on multiple algorithm cycles of the same camera that has already executed the frame acquisition task, and perform frame extraction at the merge point, reducing repeated frame acquisition by merging the coverage interval; The frame acquisition process is controlled by combining the code stream switching condition based on the frame acquisition time judgment and the retry strategy for failed stream acquisition, including: When the frame acquisition time exceeds the preset number of times and each time the time is greater than the frame acquisition time threshold, the processor executing the scheduling switches the video stream of the current camera from the main stream to the sub-stream and records the switching status in the task execution log; For tasks that fail to obtain the stream, the waiting time for the next retry is calculated using the exponential backoff method, and the frame acquisition request is re-initiated within the maximum number of retries specified. After the maximum number of retries is exceeded, the processor executing the scheduling will mark the task as failed and terminate the task scheduling.

2. The multi-stream collaborative scheduling dynamic frame extraction method according to claim 1, wherein: The construction task grouping includes: The number of processes is calculated based on the number of CPU cores of the processor executing the scheduling, and the number of threads is calculated based on the bandwidth limit of the monitoring end. Task groups are formed based on threads. The detection time of each monitored object collected by the monitoring end is counted. The detection time is the maximum value of the monitoring object association algorithm cycle. The detection time is used as the allocation weight, and the monitoring objects are assigned to task groups using a greedy strategy implemented by the processor executing the scheduling. When the maximum time of candidate groups is equal, the task group is selected based on the dispersion principle of the sites to which the monitored objects belong.

3. The multi-stream collaborative scheduling dynamic frame extraction method according to claim 1 or 2, characterized in that: The method of allocating the monitored objects to task groups according to the greedy strategy based on the detection time and allocation weight of the monitored objects includes: After the scheduling processor obtains the detection time of all monitored objects, it uses the detection time as the sorting basis, selects the monitored objects in turn and assigns them to task groups. Each time the task group with the smallest current cumulative time is selected as the target group; When the cumulative time of the task groups is equal, the processor executing the scheduling calculates the distribution of the stations to which the monitored objects belong in the task groups and compares them, selects the task groups with dispersed stations, and records the allocation results as a task group allocation table.

4. The multi-stream coordinated scheduling dynamic frame extraction method according to claim 3, wherein: The frame fetching task is executed according to the bandwidth occupancy rate of the monitoring terminal and the frame fetching task waiting time threshold, including: The frame-fetching tasks in the task group are arranged into a priority queue according to the preset task priority parameters. When the bandwidth occupancy rate of the monitoring end exceeds the preset bandwidth occupancy rate threshold, only low-priority tasks whose priority is not higher than the high-priority threshold are scheduled to perform frame-fetching operations. For low-priority tasks, frame fetching is delayed when the waiting time does not exceed the detection period. When the waiting time exceeds the detection period, the frame-fetching request of the task is discarded, and the processor that executes the scheduling records the discard log.

5. The multi-stream collaborative scheduling dynamic frame extraction method according to any one of claims 1, 2 or 4, characterized in that: The frame acquisition process is controlled by combining the code stream switching condition based on the frame acquisition time judgment and the retry strategy for failed stream acquisition, including: During the execution of the continuous frame-taking tasks, the processor executing the scheduling records the start time and end time of each frame-taking task, calculates the time consumption of the frame-taking task, and compares the time consumption result with the preset frame-taking time consumption threshold; When the number of times that the time consumed in consecutive frame-taking tasks exceeds the frame-taking time threshold reaches a preset consecutive number threshold, the processor executing the scheduling will switch the video stream format of the current camera from the main stream to the sub-stream before the next frame-taking cycle arrives, and immediately update the stream status information and switching trigger condition record in the task execution log after the switch is completed; During the execution of the frame acquisition task, if a failure in acquiring the stream is detected, the processor executing the scheduling will calculate the next retry waiting time according to the preset exponential backoff algorithm. The waiting time for each retry is the product of the previous retry waiting time and the backoff multiplier until the preset maximum retry waiting time threshold is reached. The camera's video stream request will be suspended during the retry interval. During the retry process, the processor that executes the schedule verifies the number of retries that the current task has executed before initiating each retry. When the cumulative number of retries reaches the preset maximum number of retries, the frame acquisition task is immediately terminated and the task status is updated to failure.

6. The multi-stream coordinated scheduling dynamic frame extraction method according to claim 5, characterized in that: The multi-algorithm cycle generation of the merged sampling sequence based on the same camera that has executed the frame acquisition task includes: The processor that executes the scheduling obtains the sampling periods of all algorithms configured for the same camera, calculates the greatest common divisor and least common multiple of all sampling periods, and generates a unified sampling time series based on the greatest common divisor if an integer divisibility relationship exists. If no integer divisibility relationship exists, separate sampling time series are generated separately. Merge the time series to generate a merged sampling series containing all sampling time points.

7. The multi-stream coordinated scheduling dynamic frame extraction method according to any one of claims 1, 2, 4 or 6, characterized in that: The reducing of repeated frame acquisition by merging coverage intervals includes: The processor that performs scheduling identifies the coverage relationship between adjacent time points in the merged sampling sequence, marks the coverage relationship as a merged coverage interval, performs a frame extraction operation at the starting time point of the merged coverage interval, and updates the status of the coverage interval after the extraction is completed.

8. A multi-stream collaborative scheduling dynamic frame extraction system, using the multi-stream collaborative scheduling dynamic frame extraction method according to any one of claims 1 to 7, characterized in that: It includes task grouping and greedy allocation modules, priority queue and dynamic frame acquisition modules, merged sampling and coverage optimization modules; The task grouping and greedy allocation module is used to determine the number of processes and threads based on the monitoring end network resource perception parameters obtained by the processor executing the scheduling, construct task groups, and allocate the monitored objects to the task groups according to the greedy strategy based on the detection time and allocation weight of the monitored objects; The priority queue and dynamic frame acquisition module is used to execute the frame acquisition task based on the frame acquisition task priority queue within the established task group according to the monitoring end bandwidth occupancy rate and the frame acquisition task waiting time threshold, and control the frame acquisition process in combination with the code stream switching condition determined by the frame acquisition time consumption and the retry strategy for failed stream acquisition; The merged sampling and coverage optimization module is used to generate a merged sampling sequence based on multiple algorithm cycles of the same camera that has performed the frame acquisition task, and perform frame extraction at the merge point, thereby reducing repeated frame acquisition by merging coverage intervals.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the multi-stream cooperative scheduling dynamic frame extraction method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-stream cooperative scheduling dynamic frame extraction method according to any one of claims 1 to 7 are implemented.

Images