Multi-thread-based video processing system and frame extraction interval dynamic adjustment method thereof

Abstract

The invention relates to a multi-thread-based video processing system and a frame extraction interval dynamic adjustment method thereof. The method comprises the following steps: acquiring the number N of active cameras accessed to the system; obtaining average ambient light intensity, and determining single-frame processing time T based on the ambient light intensity in combination with pre-configured single-frame processing basic time; and a first frame extraction interval t1 = N * T / (G * a) is calculated and determined, G is the total computing power of hardware, and a is a computing power coefficient. Compared with the prior art, the number of the active cameras and the average ambient light intensity are introduced, so that the frame extraction interval can be influenced in advance when the access number of the cameras is changed and the time elements are changed, and feedback adjustment can be performed without waiting for the factors to influence the lagging of the occupancy rate of the processor, so that the problem of reflection lagging can be solved on one hand; and the fluctuation of the overall GPU occupancy rate can also be reduced.

Description

Technical Field

[0001] The present invention relates to the field of video surveillance, and in particular, to a multi-threaded video processing system and a method for dynamically adjusting the frame extraction interval thereof. Background Art

[0002] With the wide application of video surveillance systems, the demand for real-time video processing is increasing day by day. In multi-camera surveillance scenarios, real-time analysis and recognition of videos are key tasks. However, due to the large amount of video data, real-time processing faces huge challenges.

[0003] In this regard, generally, a certain frame extraction interval is set to reduce the frame rate of the original video to ensure that the GPU can process the video smoothly. For example, Chinese Patent CN 112506653A discloses a method and device for adjusting the frame extraction frame rate, which dynamically adjusts the frame extraction interval by obtaining the CPU occupancy rate and combining the processing time of the current video frame. However, such existing technologies still have the following problems:

[0004] 1. Lack of adaptability for multiple cameras. When the number of cameras increases or decreases, only feedback adjustment can be made according to the CPU occupancy rate, resulting in hysteresis. When the number of cameras increases, there will be a period of lag.

[0005] 2. It adjusts the frame extraction interval by feeding back the occupancy rate of multiple cores of the final CPU. First, there are large fluctuations. Secondly, many current image processing systems use GPUs as the computing cores, and GPUs have a large number of cores, resulting in a large amount of calculation in the adjustment process itself and occupying more resources. Summary of the Invention

[0006] The purpose of the present invention is to provide a multi-threaded video processing system and a method for dynamically adjusting the frame extraction interval thereof.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A multi-threaded video processing system and a method for dynamically adjusting the frame extraction interval thereof, including:

[0009] Obtain the number N of active cameras accessing the system;

[0010] Obtain the average ambient light intensity, and determine the single-frame processing time T based on the ambient light intensity and in combination with the pre-configured single-frame processing basic time;

[0011] Calculate and determine the first frame extraction interval t1 = N * T / (G * a), where G is the total hardware computing power and a is the computing power coefficient.

[0012] The computing power coefficient is obtained based on the GPU occupancy rate and the computing power basic coefficient.

[0013] The computing power coefficient is specifically as follows:

[0014] a = a0 * f1(P)

[0015] Where: a0 is the computing power base coefficient, P is the sequence composed of the GPU occupancy rates at the most recent consecutive m sampling moments, f1(·) is the first mapping relationship, and f1(P) is the tuning coefficient corresponding to the sequence P.

[0016] The first mapping relationship is obtained from the first model.

[0017] The determination process of the tuning coefficient includes:

[0018] Calculate the mean, variance, range, and increasing or decreasing trend of the sequence P;

[0019] If the absolute value of the increasing or decreasing trend is less than the pre-configured first threshold, set the adjustment coefficient to 1; otherwise, input the mean, variance, and range of the sequence P into the first model to obtain the adjustment coefficient α;

[0020] Update the tuning coefficient based on the adjustment coefficient: f1(P) = f3(P) * α, where f3(·) is the third mapping relationship.

[0021] The third mapping relationship is obtained by fitting.

[0022] The single-frame processing time is specifically as follows:

[0023] T = T0 * f2(cd)

[0024] Where: T0 is the single-frame processing base time, cd is the average ambient light intensity, and f2(·) is the second mapping relationship.

[0025] The second mapping relationship is obtained by fitting.

[0026] A multi-threaded video processing system and its frame extraction interval dynamic adjustment device, including a memory, a processor, and a program stored in the memory. When the processor executes the program, the above method is implemented.

[0027] A storage medium, on which a program is stored. When the program is executed, the above method is implemented.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. By introducing the number of active cameras and the average ambient light intensity, the frame extraction interval can be affected in advance when the number of cameras accessing the system changes and at different times, without waiting for the occupancy rate of the processor to be affected by these factors and then adjusting through feedback. On the one hand, the problem of lag in response can be solved, and on the other hand, the fluctuation of the overall GPU occupancy rate can also be reduced.

[0030] 2. The computing power coefficient is based on the sequence of GPU occupancy rates at consecutive m sampling times. By calculating the mean, variance, range, and increasing or decreasing trend of the sequence P, when an obvious increasing or decreasing trend occurs, the adjustment coefficient is enlarged or reduced, thereby affecting the computing power coefficient. This can make a response earlier when the computing power fluctuates, improve the stability of the system, and maintain the GPU within a stable load range. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the main steps of the method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] The present invention will be described in detail below with reference to the drawings and specific embodiments. This embodiment is implemented on the premise of the technical solution of the present invention, and gives detailed implementation manners and specific operation processes, but the protection scope of the present invention is not limited to the following embodiments.

[0033] A multi-threaded video processing system and its method for dynamically adjusting the frame extraction interval, as Figure 1 shown, includes:

[0034] Obtain the number N of active cameras accessing the system;

[0035] Obtain the average ambient light intensity, and determine the single-frame processing time T based on the ambient light intensity in combination with the pre-configured basic single-frame processing time;

[0036] Calculate and determine the first frame extraction interval t1 = N * T / (G * a), where G is the total hardware computing power and a is the computing power coefficient.

[0037] In this way, by introducing the number of active cameras and the average ambient light intensity, the frame extraction interval can be affected in advance when the number of cameras accessing the system changes and at different times, without waiting for the occupancy rate of the processor to be affected by these factors and then adjusting through feedback. On the one hand, the problem of lag in response can be solved, and on the other hand, the fluctuation of the overall GPU occupancy rate can also be reduced.

[0038] In some embodiments, the computing power coefficient is obtained based on the GPU occupancy rate and the basic computing power coefficient. The computing power coefficient is specifically:

[0039] a = a0 * f1(P)

[0040] Wherein: a0 is the computing power base coefficient, P is a sequence composed of the GPU occupancy rates at the most recent consecutive m sampling moments, f1(·) is the first mapping relationship, and f1(P) is the tuning coefficient corresponding to the sequence P.

[0041] In this way, a certain mapping relationship is formed between the GPU occupancy rate and the tuning coefficient. Correspondingly, the GPU occupancy rate can affect the setting of the first frame extraction interval. In some embodiments, the tuning coefficient can be determined according to the mean value of the sequence P. In this way, a fitting method can be used to determine it. Specifically, through experiments, when the number N of active cameras, the average ambient light intensity, and the pre-configured basic single-frame processing time do not change, the optimal tuning coefficients corresponding to the mean values of the continuous GPU occupancy rates in different situations are fitted. Among them, the optimal tuning coefficient is the one with the best smoothness of the subsequent GPU occupancy rate among the selected multiple tuning coefficients. The smoothness can be the normalized weighted sum of the variance and the range.

[0042] Of course, in Embodiment 1, a method of fully mining the information contained in the sequence P is adopted. For example, the first mapping relationship is obtained from the first model, and the determination process of the tuning coefficient includes:

[0043] Calculate the mean value, variance, range, and increase / decrease trend of the sequence P;

[0044] If the absolute value of the increase / decrease trend is less than the pre-configured first threshold, set the adjustment coefficient to 1. Otherwise, input the mean value, variance, and range of the sequence P into the first model to obtain the adjustment coefficient α;

[0045] Update the tuning coefficient based on the adjustment coefficient: f1(P) = f3(P) * α, where f3(·) is the third mapping relationship.

[0046] In this way, the computing power coefficient is based on the sequence composed of the GPU occupancy rates at the most recent consecutive m sampling moments. By calculating the mean value, variance, range, and increase / decrease trend of the sequence P, when there is an obvious increase / decrease trend, the adjustment coefficient is enlarged or reduced, thereby affecting the computing power coefficient, and it can respond earlier when the computing power fluctuates, improving the stability of the system and maintaining the GPU in a stable load range.

[0047] In Embodiment 1, the third mapping relationship is obtained by fitting.

[0048] In Embodiment 1, the specific single-frame processing time is:

[0049] T = T0 * f2(cd)

[0050] Wherein: T0 is the basic single-frame processing time, cd is the average ambient light intensity, and f2(·) is the second mapping relationship.

[0051] Similarly, in most embodiments, the second mapping relationship is obtained by fitting.

[0052] The technical solution of this application is technically verified by setting up the following test environment:

[0053] An outdoor monitoring system for a certain substation was set up, with a total of 6 cameras. The video collected by each camera was distributed to the servers of each embodiment and comparative example through a video splitter. By disconnecting the video splitter, the transmission of data from the corresponding camera could be interrupted, making the camera become an inactive camera. The average ambient light intensity was collected at a fixed point and then transmitted to each server.

[0054] Embodiment 1: The setting method described above was adopted, that is, the single-frame processing time was specifically T = T0 * f2(cd), the tuning coefficient was f1(P) = f3(P) * α, and both the second mapping relationship and the third mapping relationship were obtained by fitting. Finally, the first frame extraction interval t1 = N * T / (G * a), and the total hardware computing power was expressed in floating-point operations per second.

[0055] Embodiment 2: The difference from Embodiment 1 was that the tuning coefficient was determined according to the mean value of the sequence P, and the specific determination method was fitting.

[0056] Comparative Example 1: The method in Chinese Patent CN 112506653A was adopted, that is: t1 = g(p, T1), where T1 was the measured single-frame processing time and p was the GPU occupancy rate at the current moment.

[0057] Comparative Example 2: The difference from Comparative Example 1 was that the number of active cameras was considered, that is: t1 = N * g(p, T1).

[0058] Embodiment 1, Embodiment 2, Comparative Example 1, and Comparative Example 2 ran on servers with the same configuration and were connected to different LAN ports of the same-level gateway. Tests were conducted at three time periods on September 21, 2024. Each test lasted for 30 minutes. In each test, the number of connected cameras was adjusted every 1 minute. The three test time periods were from 10:00 to 10:30 am (denoted as Test Period 1), from 8:00 to 8:30 am (denoted as Test Period 2), and from 5:00 to 5:30 pm (denoted as Test Period 3). The stability of the GPU occupancy rate of Embodiment 1, Embodiment 2, Comparative Example 1, and Comparative Example 2 was recorded in each test. The stability calculation formula was:

[0059] RB = 0.6 * y1(r 2 ) + 0.4 * y2(R)

[0060] where: RB was the stability, r 2σ is the variance of GPU occupancy, R is the range of GPU occupancy, and y1(·) and y2(·) are normalization functions.

[0061] The results are as follows:

[0062] Test period Example 1 Example 2 Comparative example 1 Comparative example 2 1 0.105 0.358 1 0.785 2 0.106 0.365 1 0.895 3 0.105 0.685 1 0.876

[0063] It can be seen that Example 1 achieved the best results, while the effect of Comparative Example 1 was extremely poor and completely unable to meet the requirements of stable operation under the condition of dynamic access of the camera. Comparative Example 2 was improved compared to Comparative Example 1, but the effect was still not good. The main reason was the existence of hysteresis and the limitation of the occupancy rate at a single moment, and the impact of unknown influencing factors on fluctuations was not reflected.

[0064] If the above functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

Claims

1. A multi-threaded video processing system and its method for dynamically adjusting the frame extraction interval, characterized in that Including: Obtain the number N of active cameras accessing the system; Obtain the average ambient light intensity, and determine the single-frame processing time T based on the ambient light intensity in combination with the pre-configured single-frame processing basic time; Calculate and determine the first frame extraction interval t1 = N*T / (G*a), where G is the total hardware computing power and a is the computing power coefficient.

2. A multi-threaded video processing system and its method for dynamically adjusting the frame extraction interval according to claim 1, characterized in that, The computing power coefficient is obtained based on the GPU occupancy rate and the computing power basic coefficient.

3. A multi-thread based video processing system and its method for dynamically adjusting the frame extraction interval according to claim 2, characterized in that, Specifically, the computing power coefficient is: a = a0*f1(P) where: a0 is the computing power basic coefficient, P is the sequence composed of the GPU occupancy rates at the most recent consecutive m sampling moments, f1(·) is the first mapping relationship, and f1(P) is the tuning coefficient corresponding to the sequence P.

4. A multi-thread based video processing system and its method for dynamically adjusting frame extraction interval according to claim 2, characterized in that, The first mapping relationship is obtained by the first model.

5. A video processing system based on multi-threading and its method for dynamically adjusting the frame extraction interval according to claim 4, characterized in that, The determination process of the tuning coefficient includes: Calculate the mean, variance, range and increasing or decreasing trend of the sequence P; If the absolute value of the increasing or decreasing trend is less than the pre-configured first threshold, set the adjustment coefficient to 1, otherwise, input the mean, variance and range of the sequence P into the first model to obtain the adjustment coefficient α; Update the tuning coefficient based on the adjustment coefficient: f1(P) = f3(P)*α, where f3(·) is the third mapping relationship.

6. A video processing system based on multi-threading and its method for dynamically adjusting the frame extraction interval according to claim 5, characterized in that The third mapping relationship is obtained by fitting.

7. A multi-threaded video processing system and its method for dynamically adjusting the frame extraction interval according to claim 1, characterized in that Specifically, the single-frame processing time is: T = T0*f2(cd) where: T0 is the single-frame processing basic time, cd is the average ambient light intensity, and f2(·) is the second mapping relationship.

8. A video processing system based on multi-threading and its method for dynamically adjusting the frame extraction interval according to claim 6, characterized in that, The second mapping relationship is obtained by fitting.

9. A multi-threaded video processing system and its frame extraction interval dynamic adjustment device, including a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1-8.

10. A storage medium, on which a program is stored, characterized in that, When the program is executed, it implements the method described in any one of claims 1-8.

Images