A multi-stream cooperative scheduling dynamic frame extraction method, system, device and storage medium

Abstract

The application discloses a kind of multi-flow cooperative scheduling dynamic frame extraction method, system, equipment and storage medium, it is related to computer vision technique, monitoring video technical field, including based on the process number and thread number determined by the monitoring terminal network resource perception parameter obtained by execution scheduling processor, constructs task grouping, based on the detection time of monitoring object and allocation weight according to the greedy strategy monitoring object is allocated to task grouping.Based on the frame extraction task priority queue in the task grouping that has been constructed, according to the bandwidth occupancy of monitoring terminal and frame extraction task waiting time threshold executes frame extraction task, and combine the code stream switching condition of frame extraction time consumption judgment and retry strategy control frame extraction process of stream failure.Based on the same camera of the frame extraction task that has been executed, multiple algorithm cycles generate merge sample sequence, and frame extraction is executed at merge point, and the repeated frame extraction is reduced by merge coverage interval.The method described in the application improves the stability of video stream transmission, picture quality and network utilization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and monitoring video, in particular to a multi-stream cooperative scheduling dynamic frame extraction method, system, device and storage medium. BACKGROUND

[0002] In modern industrial production environments, the deployment density and coverage range of monitoring cameras continue to expand. The exponential growth of data not only puts a huge pressure on storage systems, but also poses a serious challenge to data transmission networks. Traditional video transmission architecture has shown obvious performance bottlenecks in the face of such a large data stream.

[0003] In recent years, breakthroughs in deep learning technology have enabled significant progress in the application of machine vision in industrial detection. In modern industrial production, machine vision algorithms have expanded from simple product appearance detection to complex production process monitoring, equipment state analysis, personnel behavior recognition, and other dimensions. In particular, target detection and anomaly recognition algorithms based on convolutional neural networks play an irreplaceable role in quality control, safety production, and other aspects. However, these advanced algorithms usually require high-definition video streams as input, and have high requirements for the integrity and continuity of video frames, which further exacerbates the pressure on data transmission.

[0004] Unlike commercial network environments, network infrastructure in industrial sites is often limited by factors such as cost, environment, compatibility, and others. Most industrial sites still use network bandwidths of hundreds of megabits, and need to simultaneously carry video monitoring, device control, data acquisition, and other services. Under such bandwidth-limited conditions, existing technologies are difficult to meet the real-time transmission of multiple high-definition videos and the application requirements of machine vision algorithms. In particular, when multiple detection algorithms need to process video streams of different scenes in parallel, the allocation of network bandwidth becomes more prominent. Common solutions such as reducing video resolution or frame rate will directly affect the accuracy and reliability of detection algorithms, and this compromise solution is difficult to meet the strict requirements of detection accuracy in industrial scenarios.

[0005] In the context of the rapid growth of industrial monitoring video data and the increasing demand for machine vision detection, how to break through the constraints of limited bandwidth and achieve efficient detection of large amounts of monitoring video has become a key technical problem that needs to be solved in the process of industrial intelligent transformation. SUMMARY

[0006] In view of the above problems, the present application is proposed.

[0007] Therefore, the technical problem solved by the present application is that the amount of industrial monitoring video data is increasing rapidly, the demand for machine vision detection is growing, and the traditional technical solution has shown obvious performance bottlenecks in the face of large data streams, and it is difficult to simultaneously meet the real-time transmission of multiple high-definition videos and the application requirements of machine vision algorithms.

[0008] To solve the above technical problems, the present application provides the following technical solutions: a multi-stream cooperative scheduling dynamic frame extraction method, including determining the number of processes and the number of threads based on the monitoring end network resource perception parameters obtained by the processor executing scheduling, constructing task groups, and distributing monitoring objects to task groups based on the detection time of the monitoring objects and the allocation weight according to the greedy strategy.

[0009] Based on the frame extraction task priority queue in the constructed task group, the frame extraction task is executed according to the monitoring end bandwidth occupancy rate and the frame extraction task waiting time threshold, and the frame extraction process is controlled in combination with the code stream switching condition determined by the frame extraction time and the retry strategy of the stream extraction failure.

[0010] Based on the multi-algorithm cycle of the same camera of the executed frame extraction task, a merged sampling sequence is generated, and frame extraction is performed at the merging point to reduce repeated frame extraction by merging the coverage interval.

[0011] The frame extraction process controlled in combination with the code stream switching condition determined by the frame extraction time and the retry strategy of the stream extraction failure includes,

[0012] When the frame extraction time continuously exceeds the preset number of times and the time consumption of each time is greater than the frame extraction time threshold, the processor executing scheduling switches the video stream of the current camera from the main code stream to the sub-code stream, and records the switching state in the task execution log.

[0013] For the task of stream extraction failure, the waiting time of the next retry is calculated in an exponential backoff manner, the frame extraction request is reinitiated within the limited maximum number of retries, and after exceeding the maximum number of retries, the processor executing scheduling marks the task as failed and terminates the scheduling of the task.

[0014] As a preferred scheme of the multi-stream cooperative scheduling dynamic frame extraction method of the present application, wherein: the task group construction includes calculating the number of processes based on the number of CPU cores of the processor executing scheduling, calculating the number of threads based on the upper limit of the bandwidth of the monitoring end, forming task groups as units based on threads, and based on the detection time of each monitoring object collected by the monitoring end, the maximum value of the monitoring object associated algorithm cycle is taken as the detection time, the detection time is taken as the allocation weight, and the monitoring objects are distributed to the task groups by the processor executing scheduling according to the greedy strategy. When the maximum time of the candidate group is equal, the task group is selected in combination with the dispersity principle of the station to which the monitoring object belongs.

[0015] As a preferred scheme of the multi-stream cooperative scheduling dynamic frame extraction method, wherein:

[0016] The monitoring object detection time and the distribution weight are used to distribute the monitoring objects to the task groups according to the greedy strategy, which includes that after the processor performing the scheduling acquires the detection time of all the monitoring objects, the detection time is used as the sorting basis to select the monitoring objects and distribute them to the task groups in sequence, and each time the distribution is performed, the task group with the minimum cumulative time is selected as the target group. When the cumulative time of the task groups is equal, the processor performing the scheduling compares the distribution of the stations to which the monitoring objects belong in the task groups, selects the task group with the scattered station distribution, and records the distribution result as the task group distribution table.

[0017] As a preferred scheme of the multi-stream cooperative scheduling dynamic frame extraction method, wherein: the frame extraction task is performed according to the monitoring end bandwidth occupation rate and the frame extraction task waiting time threshold value, which includes that the frame extraction tasks in the task groups are arranged into a priority queue according to a preset task priority parameter, when the monitoring end bandwidth occupation rate exceeds a preset bandwidth occupation rate threshold value, only the low-priority tasks with a priority not higher than a high-priority threshold value are scheduled to perform the frame extraction operation, for the low-priority tasks, when the waiting time does not exceed the detection period, the frame extraction is delayed, when the waiting time exceeds the detection period, the frame extraction request of the task is discarded, and the processor performing the scheduling records the discarding log.

[0018] As a preferred scheme of the multi-stream cooperative scheduling dynamic frame extraction method, wherein: the code stream switching condition and the retry strategy for the stream extraction failure control the frame extraction process, which includes that the processor performing the scheduling records the start time and the end time for each frame extraction task in the continuous frame extraction task execution process, calculates the time consumption of the frame extraction task, compares the time consumption result with a preset frame extraction time consumption threshold value, when the time consumption of the continuous frame extraction task exceeds the frame extraction time consumption threshold value for a preset continuous number of times, the processor performing the scheduling switches the video stream code stream format of the current camera from the main code stream to the sub-code stream before the next frame extraction period arrives, and updates the code stream state information and the switching trigger condition record in the task execution log immediately after the switching is completed. In the frame extraction task execution process, when the stream extraction failure is detected, the processor performing the scheduling calculates the next retry waiting time according to a preset exponential backoff algorithm, the waiting time for each retry is the product of the previous retry waiting time and the backoff rate, until the preset maximum retry waiting time threshold value is reached, and the video stream request of the camera is suspended within the retry interval. In the retry process, the processor performing the scheduling verifies the number of retries of the current task before each time the retry is initiated, when the cumulative number of retries reaches a preset maximum number of retries, the frame extraction task is immediately terminated, and the task state is updated to failure.

[0019] As a preferred scheme of the multi-stream cooperative scheduling dynamic frame extraction method, wherein: the multi-algorithm cycle based on the same camera of the executed frame extraction task includes that the processor for executing scheduling acquires all algorithm sampling cycles configured by the same camera, calculates the greatest common divisor and the least common multiple of all sampling cycles, generates a unified sampling time sequence according to the greatest common divisor if there is a whole division relationship, and generates a respective sampling time sequence if there is no whole division relationship. The time sequence is merged to generate a merged sampling sequence containing all sampling time points.

[0020] As a preferred scheme of the multi-stream cooperative scheduling dynamic frame extraction method, wherein: the repeated frame extraction reduction through the merged coverage interval includes that the processor for executing scheduling identifies the coverage relationship between adjacent time points in the merged sampling sequence, marks the coverage relationship as a merged coverage interval, performs a frame extraction operation at the starting time point of the merged coverage interval, and updates the state of the coverage interval after the extraction is completed.

[0021] Another object of the present application is to provide a multi-stream cooperative scheduling dynamic frame extraction system, which can be used for frame extraction tasks based on the priority queue of the constructed task grouping in the dynamic frame extraction module, performing frame extraction tasks according to the bandwidth occupancy rate of the monitoring end and the waiting time threshold of the frame extraction task, and combining the code stream switching condition judged by the frame extraction time and the retry strategy of the stream failure control frame extraction process, solving the problem that the traditional technical scheme has shown obvious performance bottleneck in the face of huge data stream, and it is difficult to meet the application requirements of real-time transmission of multi-channel high-definition video and machine vision algorithm.

[0022] As a preferred scheme of the multi-stream cooperative scheduling dynamic frame extraction system, wherein: it comprises a task grouping and greedy allocation module, a priority queue and dynamic frame extraction module, a merged sampling and coverage optimization module.

[0023] The task grouping and greedy allocation module is used for determining the number of processes and threads based on the network resource perception parameters of the monitoring end acquired by the processor for executing scheduling, constructing a task grouping, and allocating the monitoring objects to the task grouping according to the greedy strategy based on the detection time of the monitoring objects and the allocation weight.

[0024] The priority queue and dynamic frame extraction module is used for executing frame extraction tasks based on the priority queue of the frame extraction tasks in the constructed task grouping, and combining the code stream switching condition judged by the frame extraction time and the retry strategy of the stream failure control frame extraction process according to the bandwidth occupancy rate of the monitoring end and the waiting time threshold of the frame extraction task.

[0025] The merging sampling and coverage optimization module is used for generating a merging sampling sequence based on a multi-algorithm cycle of the same camera which has performed a frame taking task, and performing frame extraction at a merging point, and reducing repeated frame taking through a merging coverage interval.

[0026] A computer device comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the steps of the multi-stream cooperative scheduling dynamic frame extraction method.

[0027] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the multi-stream cooperative scheduling dynamic frame extraction method.

[0028] The beneficial effects of the present application: the multi-stream cooperative scheduling dynamic frame extraction method provided by the present application realizes dynamic perception of the network environment at the input end by performing bandwidth monitoring and network state acquisition at the video stream input end, and obtains real-time network resource occupation, which provides real-time and accurate network bandwidth and delay and other basic data for subsequent frame extraction strategies, avoiding the frame loss or lag problem caused by fixed frame extraction strategies when the network state fluctuates. This can guarantee the continuity and stability of video transmission, and ultimately achieves the beneficial effects of adaptive adjustment of frame extraction strategies and improvement of video transmission quality under different network conditions.

[0029] By combining network bandwidth prediction and video content feature analysis, a dynamic frame extraction strategy is generated, and the matching between the frame extraction frequency and the video content change rate is realized. The role of this step is to consider both the network transmission capacity and the key degree of video content, so that high information frames are preferentially retained when the bandwidth is insufficient, and the high frame rate is maintained as much as possible when the bandwidth is sufficient. This can effectively improve the fidelity of key pictures and avoid information loss due to excessive frame extraction in important scenes, ultimately achieving the beneficial effects of ensuring core picture quality and user viewing experience under limited network resources.

[0030] By performing dynamic frame extraction and video stream encoding linkage optimization, the frame extraction strategy is directly applied to the video encoding process, and the ordered selection and compression encoding of video frames before transmission are realized. The role of this step is to control the generation of redundant data in the encoding stage, to reduce the transmission amount of invalid code streams from the source, and to form a closed-loop feedback mechanism with network state changes, so that the encoding parameters and the frame extraction strategy are optimized synchronously. This not only reduces the network load, but also reduces the decoding pressure of the receiving end, ultimately achieving the beneficial effects of balancing smoothness, clarity and resource utilization in the end-to-end transmission link. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0032] Figure 1 A multi-flow cooperative scheduling dynamic frame extraction method priority selection schematic diagram provided for the embodiment 1 of the present application.

[0033] Figure 2 A field network deployment situation schematic diagram of a multi-flow cooperative scheduling dynamic frame extraction method provided for the embodiment 2 of the present application. DETAILED DESCRIPTION

[0034] In order to make the above-mentioned objects, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification. Obviously, the described embodiments are only some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort should belong to the protection scope of the present application.

[0035] Embodiment 1, refer to Figure 1 For an embodiment of the present application, a multi-flow cooperative scheduling dynamic frame extraction method is provided, comprising:

[0036] S1: determining the number of processes and the number of threads based on the monitoring terminal network resource perception parameters obtained by the processor performing scheduling, constructing task groups, and distributing the monitoring objects to the task groups according to the greedy strategy based on the detection time of the monitoring objects and the allocation weight.

[0037] The number of processes is calculated based on the number of CPU cores of the processor performing scheduling, the number of threads is calculated based on the upper limit of the bandwidth of the monitoring terminal, the threads are used as units to form task groups, and the detection time of each monitoring object collected by the monitoring terminal is counted. The detection time is the maximum value of the algorithm cycle associated with the monitoring object. The detection time is used as the allocation weight, and the monitoring objects are distributed to the task groups by the processor performing scheduling according to the greedy strategy.

[0038] Further, the number of processes and the number of threads calculation includes that the number of processes is calculated by the processor performing scheduling reading the number of CPU cores C of the processor itself. In order to reduce the process context switching overhead and ensure that each process can stably occupy the core resources, the processor performing scheduling calculates the number of processes according to the one-third rule, that is, the number of processes is obtained by dividing the number of CPU cores by 3 and rounding down.

[0039] The number of threads is calculated by the processor executing the scheduling, which reads the maximum available bandwidth B (in Mbps) of the monitoring station. To ensure that single-threaded bandwidth access does not consume bandwidth and to facilitate bandwidth balancing between groups, the processor executing the scheduling calculates the number of threads according to a 10 rule, that is, by dividing the maximum available bandwidth by 10 and rounding down to the nearest integer.

[0040] The task grouping process creates an equal number of task groups for the execution scheduler, with each group corresponding to a parallel frame-fetching execution unit, which is used to carry the frame-fetching tasks of multiple monitored objects.

[0041] A preferred approach to task grouping involves the system automatically determining the number of processes and threads based on resource awareness parameters to balance load stability and maximize resource utilization. The calculation formula is as follows:

[0042] Number of processes: Determined by the number of CPU cores. The optimization rule is that the number of processes is the number of CPU cores divided by 3 and then rounded down. The coefficient of dividing by 3 is based on best practices obtained from experiments to ensure that each process fully utilizes CPU resources without incurring context switching overhead.

[0043] Number of threads: Determined by the bandwidth limit. The optimization rule is that the number of threads is the bandwidth limit divided by 10 and rounded down. The coefficient for dividing by 10 is obtained through bandwidth load testing.

[0044] Each thread corresponds to a task group, which is used to carry out frame extraction tasks for multiple monitored objects. The task group is defined as follows:

[0045] ;

[0046] in, It is the first A collection of monitored objects for each thread group. This indicates the total number of threads.

[0047] For example, in a typical scenario: with 32 CPU cores and a maximum bandwidth of 50Mbps, P=10 and T=5.

[0048] The system creates 10 processes, each process creates 5 threads, for a total of 50 threads (task grouping).

[0049] The detection time is determined by the processor executing the scheduling process traversing the set of detection algorithms configured for each monitored object and taking the maximum value of the detection period as the detection time for that monitored object.

[0050] Furthermore, the detection time is a fundamental parameter for intelligent allocation. The detection time for each monitored object is maintained within the monitoring equipment management program. For the monitored object Its usage time The detection period of each detection algorithm is determined based on all associated detection algorithms:

[0051] The monitoring object is set as The configured set of detection algorithms is Each algorithm The detection period of each algorithm is (unit: seconds).

[0052] The use time of the monitoring object is defined as:

[0053]

[0054] That is, the maximum period is used as the use time reference. The use time statistics are stored in the use time matrix .

[0055] Wherein, represents the th detection algorithm of the monitoring object . represents the number of detection algorithms associated with the monitoring object . represents the total number of monitoring objects.

[0056] For example, if a camera is configured with face recognition (period 3s) and behavior analysis (period 5s), then is 5s.

[0057] Further, the specific allocation process of the greedy strategy includes that the processor performing scheduling aggregates the detection use time of all monitoring objects, generates a record for each monitoring object, and includes the identification of the corresponding station and the detection use time. In order to preferentially process monitoring objects with larger detection use time, the processor performing scheduling generates an allocation list in descending order of detection use time. The current monitoring object is taken out in order, and the processor performing scheduling tries to add it to each task group in turn, calculates the cumulative use time of the group after the addition and the maximum cumulative use time of all groups. Among all candidate groups, the group that makes the maximum cumulative use time of all groups the smallest is selected as the first choice. If there are multiple groups that reach the same minimum value, the station dispersion principle is used to further screen the parallel candidates.

[0058] When the maximum use time of the candidate groups is equal, the task group is selected according to the dispersion principle of the station to which the monitoring object belongs. After the processor performing scheduling obtains the detection use time of all monitoring objects, the detection use time is used as the basis for sorting, and the monitoring objects are selected in turn and allocated to the task groups. Each time the task group with the smallest cumulative use time is selected as the target group.

[0059] The allocation algorithm is implemented based on time statistics and a greedy algorithm, with the goal of minimizing the maximum time for all task groups. The algorithm takes a time matrix and the target number of groups as input, and outputs a set of groups. .

[0060] List of monitored objects Time weighting (i.e., time is converted to weight). Grouping Initially an empty set ( For each monitored object in L. Find candidate groups: Calculate which group i should be added to. Potential maximum time:

[0061] ;

[0062] like If empty, then = .

[0063] Selecting candidate groups: Let the candidate group index set be... , For the smallest (the value of k), usually There are multiple candidate groups (such as the first candidate) Second candidate wait).

[0064] If After being added to the first candidate group, the number of monitoring stations at the same site is If the proportion of the first candidate group is greater than that of the second candidate group, then the second candidate group is selected. If the results are the same after adding the remaining candidate groups, then the first candidate group is selected. Add to selected group .

[0065] The selection rule of the greedy algorithm can be expressed as:

[0066] ;

[0067] in, Indicates the monitored object The weight of time used. Indicates the first Grouped. Indicates grouping The number of objects that have been assigned to be monitored. and This represents the specific set of candidate groups.

[0068] Further, a preferred solution of the task group selection principle based on the dispersion of the station to which the monitored object belongs specifically includes, when there are multiple candidate groups that have the same impact on the maximum cumulative use time of all groups after adding the current monitored object (tie), the station dispersion principle is used for selection.

[0069] The processor performing the scheduling respectively counts the number of allocated monitored objects in each candidate group that are the same station as the current monitored object or the proportion of the number of allocated monitored objects in the total number of the group. The candidate group with a smaller statistical value is preferentially selected to ensure that the monitored objects of the same station are as dispersed as possible in different groups. If there is still a tie, the processor performing the scheduling selects the first candidate group according to a fixed stability rule (the fixed group order number set by the present application) to ensure allocation certainty. By introducing the constraint of the smaller number of objects of the same station or the smaller proportion in the tie candidate, the distribution of stations across groups is more balanced, and the risk of sudden congestion caused by the clustering of objects of the same station in a group is reduced.

[0070] Further, a preferred solution of the task group selection principle based on the detection time as the basis for sorting, selecting monitored objects in turn and allocating to the task group specifically includes,

[0071] Establishing an allocation list: the processor performing the scheduling first sorts all monitored objects in descending order of detection time to form a list to be allocated.

[0072] Cyclic allocation: starting from the head of the list, monitored objects are taken out one by one, and the candidate group trial process is performed to select the group that minimizes the global maximum cumulative use time. If there is a tie, the tie-breaking decision process is performed according to the dispersion principle. Once the target group is determined, it is immediately written into the group, and the cumulative use time of the group and the distribution statistics of the stations in the group are updated.

[0073] Continuous updating: after allocating each monitored object, the processor performing the scheduling updates the cumulative use time of each group and the number / proportion of each station in each group, so that the trial data and allocation decision of subsequent objects are always based on the latest state.

[0074] Recording structure: after completing the current round of allocation, the processor performing the scheduling generates a task group allocation table, which lists the monitored object identifier, target group identifier, cumulative use time of the group after allocation, and the number of objects of the same station in the group for direct reference in subsequent frame scheduling.

[0075] When the cumulative use time of the task group is equal, the processor performing the scheduling compares the distribution of the stations to which the monitored objects belong in the task group, selects the task group with a dispersed station distribution, and records the allocation result as a task group allocation table.

[0076] Further, the dispersion refers to a lower proportion of objects of the same station as the object to be allocated in the target group.

[0077] It should be noted that S1 first determines the number of processes and threads according to the number of CPU cores and the upper limit of the bandwidth, constructs a task grouping, and takes the maximum value of the monitoring object detection algorithm period as the detection time, and uses the greedy strategy to preferentially allocate high detection time objects to balance the load of each grouping. In the parallel case, the field station dispersion principle is introduced, and the grouping with lower proportion of the same field station is preferentially allocated to reduce the risk of congestion. S1 continuously updates the grouping state and records the task grouping allocation table during the allocation process to provide accurate data support for subsequent scheduling.

[0078] S2: Based on the frame taking task priority queue in the constructed task grouping, the frame taking task is executed according to the monitoring end bandwidth occupancy rate and frame taking task waiting time threshold, and the frame taking process is controlled in combination with the code stream switching condition judged by the frame taking time and the retry strategy of the stream taking failure.

[0079] The frame taking tasks in the task grouping are arranged into a priority queue according to the preset task priority parameter. When the monitoring end bandwidth occupancy rate exceeds the preset bandwidth occupancy rate threshold, only low-priority tasks with a priority not higher than a high-priority threshold are scheduled to perform frame taking operations. For low-priority tasks, when the waiting time does not exceed the detection period, the frame taking is delayed, and when the waiting time exceeds the detection period, the frame taking request of the task is discarded, and the executing processor records the discarding log.

[0080] Further, the executing processor sets the task priority parameter to an integer of 1 to 3, and the smaller the value, the higher the priority. Among them, the first-level task corresponds to the key safety monitoring business, the second-level task corresponds to the important safety detection business, and the third-level task corresponds to the general detection business. The preset threshold of the monitoring end bandwidth occupancy rate is set to 70%. When the bandwidth occupancy rate exceeds, the execution of low-priority tasks is limited. The high-priority threshold is set to 2, that is, in the case where the bandwidth occupancy rate is higher than 70%, only tasks with a priority value of 1 or 2 are executed. The priority selection refers to Figure 1 .

[0081] When the bandwidth occupancy rate is higher than 70%, only high-priority tasks are executed. Low-priority tasks are queued and delayed: if the waiting time of a low-priority task exceeds its detection period, the task of this round is discarded (frame is discarded) and a log is recorded.

[0082] Otherwise, all tasks are executed, denoted as:

[0083] ;

[0084] Wherein, represents the Boolean determination result of whether the th task is executed. represents the monitoring end bandwidth occupancy rate. represents the The Boolean result indicating whether a task has been executed. This indicates a high-priority threshold.

[0085] During the execution of continuous frame fetching tasks, the processor that performs the scheduling records the start and end times of each frame fetching task, calculates the time consumed by the frame fetching task, and compares the time consumed with a preset frame fetching time threshold.

[0086] Furthermore, the detection period for each frame-fetching task is determined by its configured detection algorithm. For example, the detection period for open flame detection can be set to 3 seconds, for safety helmet detection to 5 seconds, and for perimeter security to 10 seconds. The frame-fetching time threshold is set to 150 milliseconds, meaning that if the time from the start of a single frame-fetching operation to obtaining complete video frame data exceeds this value, it is considered that the frame-fetching time has exceeded the limit.

[0087] When the number of consecutive frame-fetching tasks that take longer than the frame-fetching time threshold reaches a preset consecutive number threshold, the processor executing the scheduling will switch the current camera's video stream bitstream format from the main bitstream to the sub-bitstream before the next frame-fetching cycle arrives, and immediately update the bitstream status information and switching trigger condition record in the task execution log after the switch is completed.

[0088] When frame fetching time increases, the system automatically switches the video stream type (e.g., switching from the mainstream stream to a sub-stream) to reduce bandwidth usage. The switching strategy is based on timeout detection.

[0089] Switching conditions: Set the frame time sequence (Unit: ms). If three frames are captured consecutively... If the time exceeds 150ms, a handover action will be triggered.

[0090] ;

[0091] 150ms is a program setting value. This indicates the trigger condition for stream switching, which can be formally represented as three consecutive... >150ms.

[0092] Furthermore, the consecutive capture threshold is set to 3. When the time taken for three consecutive frame captures is greater than 150 milliseconds, the processor executing the scheduling will switch the current camera's video stream bitstream format from the main bitstream to the sub-bitstream before the start of the next frame capture cycle. The switching process is as follows: a bitstream parameter update command is sent to the camera or access terminal to adjust the bitstream configuration from the main bitstream parameter to the sub-bitstream parameter. After receiving the device's confirmation response, the internally recorded bitstream type status is updated, and the bitstream type, triggering conditions, and switching time before and after the switch are recorded in the task execution log.

[0093] When it is detected that the stream acquisition fails during the execution of the frame acquisition task, the processor performing the scheduling calculates the next retry waiting time according to a preset exponential backoff algorithm. The waiting time for each retry is the product of the waiting time for the previous retry and a backoff rate, until a preset maximum retry waiting time threshold is reached, and the video stream request of the camera is suspended during the retry interval.

[0094] Further, the stream acquisition failure refers to any of the following situations occurring during the process in which the processor performing the scheduling initiates a video stream request to the camera and prepares to receive data: a connection establishment stage timeout or an authentication failure. No video frame data is received within a specified video frame reception timeout. The received video frame fails three times in succession in data integrity verification, and no valid frame is obtained in the retry.

[0095] When it is detected that the stream acquisition fails, the processor performing the scheduling calculates the waiting time for the next retry. The waiting time for the first retry is 10 seconds, and the waiting time for each subsequent retry is twice the waiting time for the previous retry. For example, 10 seconds for the first retry, 20 seconds for the second retry, and 40 seconds for the third retry. During the waiting period, the frame acquisition request to the camera is suspended to reduce the bandwidth pressure.

[0096] During the retry process, the processor performing the scheduling verifies the number of retries performed by the current task before initiating each retry. When the cumulative number of retries reaches a preset maximum number of retries, the frame acquisition task is immediately terminated, and the task status is updated to failure.

[0097] Further, the maximum retry waiting time threshold is set to 40 seconds, that is, when the calculation result of the retry waiting time exceeds 40 seconds, the waiting time is no longer increased. The maximum number of retries is set to 3 times, that is, after three consecutive retries fail, the processor performing the scheduling immediately terminates the frame acquisition task, and records the failure reason, the number of retries, and the last waiting time in the task execution log.

[0098] It should be noted that S2 dynamically controls the execution of the frame acquisition task on the basis of the grouped tasks through the priority queue and the bandwidth occupancy rate, ensures that high-priority tasks are processed first, avoids invalid frame acquisition in combination with the judgment of the task waiting time and the detection period, actively switches the stream when the continuous frame acquisition time consumption exceeds the threshold to reduce the bandwidth occupancy, controls the retry frequency through the exponential backoff retry when the stream acquisition fails, and finally terminates the task and records the log when the maximum number of times or the waiting time is exceeded, thereby realizing the stability and adaptability of the task scheduling, and providing double protection of the bandwidth and execution efficiency for the overall frame extraction process.

[0099] S3: a merged sampling sequence is generated based on the same camera of the executed frame acquisition task according to a plurality of algorithms, and frame extraction is performed at the merging point to reduce repeated frame acquisition through the merging coverage interval.

[0100] The processor performing scheduling acquires all algorithm sampling periods configured for the same camera, calculates the greatest common divisor and least common multiple of all sampling periods, generates a unified sampling time sequence according to the greatest common divisor if there is a division relationship, and generates respective sampling time sequences if there is no division relationship. The time sequences are merged to generate a merged sampling sequence containing all sampling time points.

[0101] Further, the processor performing scheduling first acquires the sampling periods of all detection algorithms configured on the same camera. These sampling periods are usually fixed time intervals between algorithms, for example, algorithm A is 3 seconds, algorithm B is 6 seconds, and algorithm C is 9 seconds. The division relationship exists means that at least one sampling period can be divided by another sampling period, and the result of division is an integer. For example, 6 seconds can be divided by 3 seconds, and 9 seconds can be divided by 3 seconds. At this time, the processor performing scheduling calculates the greatest common divisor of all sampling periods, and takes the greatest common divisor as a unified sampling time step to generate a unified sampling time sequence covering all algorithms. For example, in the above example, the greatest common divisor is 3 seconds, and the generated sampling time sequence is 0 seconds, 3 seconds, 6 seconds, 9 seconds, 12 seconds, and so on.

[0102] If there is no division relationship, that is, the result of division of any two sampling periods is not an integer (for example, algorithm A is 4 seconds, algorithm B is 6 seconds, and algorithm C is 10 seconds), the processor performing scheduling generates a corresponding sampling time sequence for each algorithm respectively. For example, the sequence of the 4-second algorithm is 0 seconds, 4 seconds, 8 seconds, and so on, the sequence of the 6-second algorithm is 0 seconds, 6 seconds, 12 seconds, and so on, and the sequence of the 10-second algorithm is 0 seconds, 10 seconds, 20 seconds, and so on. Then, these sequences are merged to obtain a merged sampling sequence containing all sampling time points, and the merged time points are rearranged in ascending order.

[0103] The processor performing scheduling identifies the coverage relationship between adjacent time points in the merged sampling sequence, marks the coverage relationship as a merged coverage interval, performs frame extraction at the start time point of the merged coverage interval, and updates the state of the coverage interval after the extraction is completed.

[0104] Further, after obtaining the merged sampling sequence, the processor performing scheduling analyzes whether there is a coverage relationship between adjacent time points in the sequence, that is, two or more algorithms need to access frame data that is completely consistent in time or has a time interval less than an acceptable minimum frame interval threshold. The time point range determined to be covered is referred to as a merged coverage interval.

[0105] In each merged coverage interval, the processor performing scheduling only performs frame extraction operation once at the first time point of the interval. The specific implementation of frame extraction is as follows: when the time point is reached, the camera or video stream buffer is requested for the original video frame data of the corresponding time point. After receiving the complete frame data, it is copied to the algorithm task buffer of each algorithm that needs this frame, so that the frame can be used by multiple algorithms at the same time, thereby avoiding repeated frame request for the same video stream in the same time period. After extraction is completed, the processor performing scheduling marks the coverage interval as processed and updates the execution state of the interval in the task scheduling record, so as to skip the completed interval in the subsequent scheduling process and reduce repeated operations.

[0106] Suppose there are monitoring objects Configure multiple algorithms, each with a detection period Generate a merged sampling sequence Minimize total sampling points:

[0107] If the periods There is a containing relationship (i.e. one period is a multiple of the other), then merge based on the least common multiple:

[0108] ;

[0109] Where, GCD is the greatest common divisor, LCM is the least common multiple.

[0110] Otherwise, the sequence is calculated independently based on each period, and the sampling points are reduced by merging coverage intervals.

[0111] Algorithm description:

[0112] Input algorithm period list:

[0113] ;

[0114] Calculate the base interval:

[0115] ;

[0116] Generate a sampling point sequence, if All elements of satisfy Divide Then:

[0117] ;

[0118] Otherwise, calculate the sampling point sequence of each Then merge:

[0119] ;

[0120] wherein T is a monitoring period.

[0121] The frame extraction task only executes The system records a merge log.

[0122] Example 1: Period is 5s, is 10s, then , , (5s interval).

[0123] Example 2: is 3s, is 7s, and has no common factor, then is independently calculated and merged: , covering the requirements of two models.

[0124] It should be noted that S3 utilizes the sampling period relationship of multiple algorithms of the same camera, unifies or merges the sampling time points, and reduces repeated frame extraction operations. By judging the division relationship between the sampling periods, a unified sampling sequence can be generated under the greatest common divisor, and when it is not an integer, multiple sequences are merged and de-duplicated. The merged sequence is analyzed for coverage relationship, and frame extraction is only performed once at the first point in the coverage interval, and the result is distributed to multiple algorithm tasks, which not only avoids bandwidth waste caused by repeated stream extraction, but also ensures that each algorithm can obtain data as needed, thereby improving frame extraction efficiency and resource utilization.

[0125] Embodiment 2, as an embodiment of the present application, provides a multi-stream cooperative scheduling dynamic frame extraction method. In order to verify the beneficial effects of the present application, economic benefit calculation and simulation experiments are used for scientific demonstration.

[0126] A new energy power plant deploys an industrial AI vision detection product, and nearly a thousand monitoring camera terminals are connected in the whole plant. Under the condition that the video network special line bandwidth of 50M is temporarily invariable, it is necessary to support multiple scene detection tasks at the same time, which is a very big challenge. The experimental conditions are as follows:

[0127] Test platform: multi-camera video stream input, resolution 1920x1080, frame rate 30fps.

[0128] Network environment: bandwidth upper limit 100Mbps, delay 50ms.

[0129] Hardware configuration: processor model Intel Xeon E5-2620v4, memory 64GB.

[0130] Algorithm configuration: three detection algorithms, algorithm types include open fire detection, safety helmet detection and perimeter security, open fire detection period 3 seconds, proportion 20%. Safety helmet detection period 5 seconds, proportion 50%. Perimeter security period 10 seconds, proportion 30%.

[0131] The priority configuration is that the priority of open fire detection is 1, the priority of safety helmet detection is 2, and the priority of perimeter security is 3.

[0132] Three sampling strategies are set up for comparison in the experiment.

[0133] Independent sampling strategy: each algorithm independently takes frames according to its own sampling period without merging.

[0134] Fixed period unified sampling strategy: take frames with the greatest common divisor as the unified sampling period.

[0135] The multi-stream cooperative scheduling strategy of the application: generate a merged sampling sequence according to the steps of the application, and perform frame extraction in the merged coverage interval.

[0136] The experimental results are shown in Table 1.

[0137] Table 1 Comparison of the application and the traditional scheme

[0138] Index Traditional solution Invention solution Promotion effect Total bandwidth occupation 1.8 Gbps 45 Mbps Reduced by 97.5% Average detection delay 500 ms 80 ms Reduced by 84% High priority task delay 200 ms 50 ms Reduced by 75% CPU utilization 40% 85% Increased by 112.5% Task discard rate 15% 2% Reduced by 86.7% Hardware cost Need special server cluster Single T4 + 32-core server Reduced by 30%

[0139] As can be seen from the data in Table 1, the multi-stream cooperative scheduling method proposed by the application has good effect in reducing task discard rate and reducing bandwidth occupation, and the task processing delay is also optimized. This shows that in the multi-algorithm and multi-period video analysis task, the method can effectively improve the overall efficiency of the system and reduce resource consumption.

[0140] Embodiment 3 is an embodiment of the application, which provides a multi-stream cooperative scheduling dynamic frame extraction system, including a task grouping and greedy allocation module, a priority queue and dynamic frame extraction module, a merged sampling and coverage optimization module.

[0141] The task grouping and greedy allocation module is used to determine the number of processes and threads based on the monitoring end network resource perception parameters obtained by the processor for execution scheduling, construct task groups, and allocate monitoring objects to task groups according to the detection time of monitoring objects and allocation weights based on the greedy strategy.

[0142] The priority queue and dynamic frame extraction module is used to execute frame extraction tasks based on the frame extraction task priority queue in the constructed task group, according to the monitoring end bandwidth occupation rate and frame extraction task waiting time threshold, and combines the code stream switching condition judged by frame extraction time and the retry strategy of stream failure to control the frame extraction process.

[0143] The merging sampling and coverage optimization module is configured to generate a merging sampling sequence based on a same camera's multi-algorithm cycle of the executed frame-taking task, and perform frame extraction at a merging point, and reduce repeated frame-taking by merging coverage intervals.

Claims

1. A multi-stream coordinated scheduling dynamic frame extraction method, characterized in that, The application relates to a method for scheduling a monitoring terminal to acquire video streams. The method comprises the following steps: a processor acquiring monitoring terminal network resource sensing parameters based on execution scheduling determines the number of processes and the number of threads, constructs a task group, and allocates monitoring objects to the task group based on the detection time of the monitoring objects and the allocation weight according to a greedy strategy. The construction of the task group comprises the following steps: the number of processes is calculated based on the number of CPU cores of the processor for execution scheduling, the number of threads is calculated based on the upper limit of the bandwidth of the monitoring terminal, the threads are used as units to form the task group, the detection time of each monitoring object collected by the monitoring terminal is counted, the maximum value of the monitoring object associated algorithm cycle is taken as the detection time, the detection time is taken as the allocation weight, and the monitoring objects are allocated to the task group by the processor for execution scheduling according to the greedy strategy. When the maximum time of the candidate groups is equal, the task group is selected according to the dispersity principle of the station to which the monitoring object belongs. Based on the frame taking task priority queue in the constructed task group, the frame taking task is executed according to the bandwidth occupation rate of the monitoring terminal and the frame taking task waiting time threshold, and the frame taking process is controlled according to the code stream switching condition determined by the frame taking time and the retry strategy of the stream taking failure. Based on the same camera of the executed frame taking task, a merged sampling sequence is generated according to multiple algorithm cycles, and frame extraction is performed at the merging point, and the repeated frame taking is reduced by the merging coverage interval. The frame taking process is controlled according to the code stream switching condition determined by the frame taking time and the retry strategy of the stream taking failure. When the frame taking time continuously exceeds the preset number of times and the time consumption of each time is greater than the frame taking time threshold, the processor for execution scheduling switches the video stream of the current camera from the main code stream to the sub code stream, and records the switching state in the task execution log. For the task of stream taking failure, the waiting time of the next retry is calculated according to the exponential backoff mode, the frame taking request is reinitiated within the limited maximum number of retries, and the task is marked as failed and the scheduling of the task is terminated by the processor for execution scheduling when the maximum number of retries is exceeded.

2. The multi-stream coordinated scheduling dynamic frame-skipping method of claim 1, wherein: The monitoring objects are allocated to the task group based on the detection time of the monitoring objects and the allocation weight according to the greedy strategy. After the processor for execution scheduling acquires the detection time of all the monitoring objects, the detection time is taken as the sorting basis, the monitoring objects are selected in sequence and allocated to the task group, and the task group with the minimum cumulative time is selected as the target group during each allocation. When the cumulative time of the task group is equal, the distribution of the stations to which the monitoring objects in the task group belong is compared by the processor for execution scheduling, the task group with the dispersed station distribution is selected, and the allocation result is recorded as the task group allocation table.

3. The multi-stream coordinated scheduling dynamic frame-skipping method of claim 2, wherein: The frame taking task in the task group is arranged into a priority queue according to the preset task priority parameter, when the bandwidth occupation rate of the monitoring terminal exceeds the preset bandwidth occupation rate threshold, only the low-priority task with the priority not higher than the high-priority threshold is scheduled to execute the frame taking operation, for the low-priority task, when the waiting time does not exceed the detection period, the frame taking is delayed, when the waiting time exceeds the detection period, the frame taking request of the task is discarded, and the discarding log is recorded by the processor for execution scheduling. ​ 4. The multi-stream co-scheduling dynamic frame-skipping method of any one of claims 1 or 3, wherein: The code stream switching condition based on the frame taking time consumption judgment and the retry strategy of the stream taking failure control the frame taking process, In the continuous frame taking task execution process, the processor performing scheduling records the start time and the end time for each frame taking task, calculates the frame taking time consumption, and compares the frame taking time consumption with a preset frame taking time consumption threshold; When the number of times that the frame taking time consumption exceeds the frame taking time consumption threshold in the continuous frame taking tasks reaches a preset continuous number threshold, the processor performing scheduling switches the video stream code stream format of the current camera from the main code stream to the sub code stream before the next frame taking period arrives, and updates the code stream state information and the switching trigger condition record in the task execution log immediately after the switching is completed; In the frame taking task execution process, when the stream taking failure is detected, the processor performing scheduling calculates the next retry waiting time according to a preset exponential backoff algorithm, the waiting time of each retry is the product of the previous retry waiting time and the backoff rate, until the preset maximum retry waiting time threshold is reached, and the video stream request of the camera is suspended during the retry interval; In the retry process, the processor performing scheduling verifies the number of retries that the current task has performed before each retry is initiated, and when the cumulative number of retries reaches a preset maximum number of retries, the frame taking task is terminated immediately, and the task state is updated to failure.

5. The multi-stream coordinated scheduling dynamic frame-skipping method of claim 4, wherein: The combined sampling sequence generated based on the multiple algorithm periods of the same camera of the executed frame taking tasks includes The processor performing scheduling obtains all algorithm sampling periods configured by the same camera, calculates the greatest common divisor and the least common multiple of all the sampling periods, generates a unified sampling time sequence according to the greatest common divisor if there is a whole division relationship, or generates respective sampling time sequences if there is no whole division relationship; The time sequences are combined to generate a combined sampling sequence containing all sampling time points.

6. The multi-stream co-scheduling dynamic frame-skipping method of any one of claims 1, 3 or 5, wherein: The repeated frame taking is reduced by merging the coverage intervals, which includes The processor performing scheduling identifies the coverage relationship between adjacent time points in the combined sampling sequence, marks the coverage relationship as a merged coverage interval, performs frame extraction at the start time point of the merged coverage interval, and updates the state of the coverage interval after the extraction is completed.

7. A multi-stream co-scheduled dynamic frame-skipping system employing the multi-stream co-scheduled dynamic frame-skipping method of any one of claims 1-6. The task grouping and greedy allocation module, the priority queue and dynamic frame taking module, and the combined sampling and coverage optimization module are included. The task grouping and greedy allocation module is configured to determine the number of processes and the number of threads based on the network resource perception parameters obtained by the processor performing scheduling, construct task groups, and allocate the monitoring objects to the task groups according to the greedy strategy based on the detection time and the allocation weight of the monitoring objects. The priority queue and dynamic frame taking module is configured to perform the frame taking task according to the monitoring end bandwidth occupancy rate and the frame taking task waiting time threshold based on the frame taking task priority queue in the constructed task groups, and control the frame taking process by combining the code stream switching condition based on the frame taking time consumption judgment and the retry strategy of the stream taking failure. The combined sampling and coverage optimization module is configured to generate a combined sampling sequence based on the multiple algorithm periods of the same camera of the executed frame taking tasks, and perform frame extraction at the merging point to reduce repeated frame taking.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the multi-stream coordinated scheduling dynamic frame extraction method in any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the multi-stream coordinated scheduling dynamic frame extraction method in any one of claims 1 to 6.

Images