Video anti-shake acceleration method for multi-scene perception and thread scheduling, medium and equipment

Abstract

The invention discloses a video anti-shake acceleration method for multi-scene perception and thread scheduling, a medium and equipment. The method comprises the following steps: initializing system resources; multiple paths of video stream data input are received, each path of video stream data is preprocessed, and the video stream data comprise video stream images; and monitoring and analyzing the scene complexity and the performance index of the current video stream image of each path, determining whether system resources need to be redistributed based on an analysis result, if so, determining a resource distribution adjustment strategy based on the analysis result, and readjusting the system resources of the video stream data of each path according to the resource distribution adjustment strategy. According to the scheme, by analyzing the resolution and the frame rate of the video stream, the number and configuration of the GPU threads are adjusted in real time, and the task which has the greatest influence on the video stability is preferentially processed, so that the efficiency and the stability of video anti-shake processing are improved.

Description

Technical Field

[0001] The present application relates to the field of video stream data processing, and specifically to a video anti-shake acceleration method, medium and device with multi-scene perception and thread scheduling. Background Art

[0002] With the continuous development of video technology and multimedia applications, the shooting and sharing of video content has become more and more widespread. Whether it is a home video shot by an individual user or a film and television work produced by a professional team, they all need to overcome a common problem during the shooting process - video jitter. Video jitter can cause blur and distortion of the picture, seriously affecting the viewing experience. Therefore, the demand for video anti-shake technology is becoming more and more urgent.

[0003] Traditional video stabilization technology mainly relies on mechanical stabilization devices and CPU-based software algorithms. However, mechanical stabilization devices are expensive and bulky, making them unsuitable for daily use; although CPU-based software algorithms can alleviate video tremors to a certain extent, they are difficult to meet the real-time processing requirements of high-definition and high-frame-rate videos due to limited computing power.

[0004] In recent years, with the development of graphics processing unit (GPU) technology, GPU has been widely used in image processing and computationally intensive tasks due to its powerful parallel computing capabilities. Using GPU for video stabilization can not only significantly improve the processing speed, but also ensure the real-time and efficient processing. However, in practical applications, how to efficiently configure and manage GPU resources still faces many challenges.

[0005] Specifically, the resolution and frame rate of the video stream directly affect the required computing resources. For example, high-resolution video and high-frame-rate video require more computing resources to maintain real-time processing performance. In addition, dynamic changes in video content (such as fast motion and scene switching) also place higher demands on processing performance. Therefore, how to dynamically adjust the allocation of GPU resources according to the characteristics of the video stream and optimize the real-time performance and stability of the video stabilization algorithm has become an urgent problem to be solved. Summary of the invention

[0006] In view of the above problems, the present application provides a video stabilization acceleration method, medium and device with multi-scene perception and thread scheduling, which are used to improve the efficiency and stability of video stabilization.

[0007] To achieve the above objectives, in a first aspect, the present application provides a video anti-shake acceleration method with multi-scene perception and thread scheduling, the method comprising the following steps:

[0008] Initialize system resources;

[0009] Receiving multiple video stream data inputs and preprocessing each of the video stream data, wherein the video stream data includes a video stream image;

[0010] Monitor and analyze the scene complexity and performance indicators of each current video stream image, and decide whether system resources need to be reallocated based on the analysis results. If so, determine a resource allocation adjustment strategy based on the analysis results, and readjust the system resources of each video stream data according to the resource allocation adjustment strategy.

[0011] Further, the initializing system resources includes:

[0012] Obtaining the computing power resource parameters of the GPU, wherein the computing power resource parameters include the number of GPU cores, the size of shared memory, and the utilization rate of the current GPU;

[0013] Allocate the computing power resources of the GPU to each channel of video stream data.

[0014] Further, preprocessing the video stream data of each channel includes:

[0015] Read basic parameter information of multiple video stream data, wherein the basic parameter information includes frame rate and / or resolution;

[0016] Adaptively adjusting the resolution of each channel of video stream data according to the basic parameter information of the video stream data and the system resources allocated to each channel of video stream data;

[0017] And the video stream images of each video stream data are subjected to motion estimation and smoothing processing, image transformation and resampling are performed according to the motion trajectory after smoothing processing, and the resampled video stream images are output.

[0018] Further, monitoring and analyzing the scene complexity and performance indicators of each current video stream image, and determining whether to reallocate system resources based on the analysis results include:

[0019] Monitor the performance indicators of video stabilization processing of each video stream data in real time, and write the collected performance indicators into the Redis cache;

[0020] The data in the Redis cache is input into the trained neural network model, and an analysis result is output, where the analysis result includes any one of maintaining the status quo, reducing computing resources, or increasing computing resources.

[0021] Furthermore, the relevant performance indicators include the motion intensity M of the scene jitter. motion，i , Processing delay D delay，i , GPU utilization G gpu_usage_total 、Memory usage M memory-total，i , the video quality evaluation index Q of the previous and next framesquality，i .

[0022] Furthermore, the method comprises:

[0023] If the analysis result of the neural network model is that system resources need to be reallocated, and the determined resource allocation adjustment strategy does not meet the preset constraints, the benchmark greedy algorithm is used to determine the feasible scheduling parameters for each video stream data, and a new resource allocation adjustment strategy is generated based on the scheduling parameters.

[0024] Furthermore, the method comprises:

[0025] If it is determined based on the analysis results that system resources need to be reallocated, the publish or subscribe function of the Redis cache is used to trigger an event and broadcast the feedback result that system resources need to be reallocated;

[0026] The re-adjusting the system resources of each channel of video stream data according to the resource allocation adjustment strategy comprises:

[0027] Each video stream monitors the event notification of the Redis cache, and once the event notification is received, the system resources corresponding to the video stream data of the own channel are obtained according to the resource allocation adjustment strategy.

[0028] Furthermore, the method further comprises:

[0029] After each new resource allocation adjustment strategy is determined, the current timestamp information and status information of each video stream data are recorded, and a status report is generated and uploaded.

[0030] In a second aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video stabilization acceleration method with multi-scene perception and thread scheduling as described in the first aspect of the present application.

[0031] In a third aspect, the present application provides an electronic device including a processor and a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by the processor, the video stabilization acceleration method with multi-scene perception and thread scheduling as described in the first aspect of the present application is implemented.

[0032] Different from the prior art, the video stabilization acceleration method, medium and device with multi-scene perception and thread scheduling described in the above technical solution include: initializing system resources; receiving multiple video stream data inputs and preprocessing each video stream data, the video stream data including video stream images; monitoring and analyzing the scene complexity and performance indicators of each current video stream image, and determining whether to reallocate system resources based on the analysis results, and if so, determining the resource allocation adjustment strategy based on the analysis results, and readjusting the system resources of each video stream data according to the resource allocation adjustment strategy. The above solution analyzes the resolution and frame rate of the video stream, adjusts the number and configuration of GPU threads in real time, and gives priority to processing tasks that have the greatest impact on video stability, thereby improving the efficiency and stability of video stabilization processing.

[0033] The above-mentioned records related to the invention content are only an overview of the technical solution of the present application. In order to enable ordinary technicians in the field to more clearly understand the technical solution of the present application, and then implement it according to the text of the specification and the contents recorded in the drawings, and to make the above-mentioned purpose and other purposes, features and advantages of the present application easier to understand, the following is an explanation in combination with the specific implementation mode and drawings of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings are only used to illustrate the principles, implementation methods, applications, characteristics and effects of the specific embodiments of the present invention and other related contents, and shall not be considered as limiting the present application.

[0035] In the drawings of the specification:

[0036] Figure 1 This is a flow chart of a video stabilization acceleration method for multi-scene perception and thread scheduling involved in the first exemplary embodiment of the present application;

[0037] Figure 2 This is a flow chart of a video stabilization acceleration method for multi-scene perception and thread scheduling involved in a second exemplary embodiment of the present application;

[0038] Figure 3 This is a flow chart of a video stabilization acceleration method for multi-scene perception and thread scheduling involved in the third exemplary embodiment of the present application;

[0039] Figure 4 This is a flowchart of a video stabilization acceleration method for multi-scene perception and thread scheduling involved in a fourth exemplary embodiment of the present application;

[0040] Figure 5 This is an architecture diagram of a video anti-shake acceleration method for multi-scene perception and thread scheduling involved in an exemplary embodiment of the present application;

[0041] Figure 6This is an architecture diagram of a neural network model calculation involved in an exemplary embodiment of the present application;

[0042] Figure 7 This is a schematic diagram of a module of an electronic device according to an exemplary embodiment of the present application;

[0043] Reference numerals:

[0044] 10. Electronic equipment;

[0045] 101. Processor;

[0046] 102. Storage medium. DETAILED DESCRIPTION

[0047] In order to explain in detail the possible application scenarios, technical principles, specific schemes that can be implemented, and the purposes and effects that can be achieved, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0048] Reference to "embodiment" herein means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The term "embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or association with other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the various technical features mentioned in the embodiments can be combined in any way to form a corresponding implementable technical solution.

[0049] Unless otherwise defined, the technical terms used in this document have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms in this document is only for describing specific embodiments and is not intended to limit this application.

[0050] In the first aspect, Figure 1 As shown, the present application provides a video anti-shake acceleration method with multi-scene perception and thread scheduling, and the method comprises the following steps:

[0051] First, enter step S101 to initialize system resources;

[0052] Then, the process proceeds to step S102 to receive multiple video stream data inputs and pre-process each of the video stream data, wherein the video stream data includes a video stream image;

[0053] Then, step S103 is entered to monitor and analyze the scene complexity and performance index of each current video stream image, and determine whether system resources need to be reallocated based on the analysis result;

[0054] If the judgment result of step S103 is yes, the process proceeds to step S104 to determine a resource allocation adjustment strategy based on the analysis result, and readjust the system resources of each channel of video stream data according to the resource allocation adjustment strategy.

[0055] In this embodiment, during the process of implementing step S101, as shown in FIG. Figure 2 As shown, the initialization system resources include:

[0056] First, go to step S201 to obtain the computing power resource parameters of the GPU;

[0057] Then, the process proceeds to step S202 to evenly distribute the computing resources of the GPU to each channel of video stream data.

[0058] In step S201, the computing resource parameters include the number of GPU cores, the size of shared memory and the utilization rate of the current GPU.

[0059] In step S202, for example, if the number of cores of the GPU of Jetson TX2 is 256, then for dual-channel video streams, 128 CUDA cores can be allocated to each video stream, and if there are 4-channel video stream data, 64 CUDA cores can be allocated to each video stream. Resource allocation preferably follows the principle of multiples of 8: 16, 32, 64, 96, 128, etc.

[0060] In this embodiment, during the process of implementing step S102, Figure 3 As shown, preprocessing the video stream data of each channel includes:

[0061] S301: Read basic parameter information of multiple video stream data, where the basic parameter information includes frame rate and / or resolution;

[0062] S302: Adaptively adjusting the resolution of each channel of video stream data according to the basic parameter information of the video stream data and the system resources allocated to each channel of video stream data; and

[0063] S303: Perform motion estimation and smoothing processing on the video stream images of each channel of video stream data, perform image transformation and resampling according to the motion trajectory after the smoothing processing, and output the resampled video stream images.

[0064] In step S302, resolution grading processing may be performed according to currently allocated resources, and the resolution of the video stream may be divided into three levels: high, medium and low, as follows:

[0065] When sufficient system resources are allocated to a certain video stream data, the original high resolution (such as 1080p) of the video stream data is maintained;

[0066] When the system resources allocated to a certain video stream data are limited, the video stream data is reduced to a medium resolution (such as 720p);

[0067] When the system resources allocated to a certain channel of video stream data are insufficient, the video stream data is reduced to a low resolution (such as 480p) again.

[0068] In step S303, the motion estimation and smoothing of the video stream image are specifically implemented in the following manner:

[0069] Use the CUDA-optimized ORB or FAST algorithm to extract feature points from video stream images, use the shared memory-based KLT optical flow algorithm to track feature points, and then estimate camera motion based on the feature point tracking results, and use the Kalman filter to smooth the motion trajectory.

[0070] In step S303, the image is resampled using a bilinear interpolation algorithm optimized by CUDA, and a video stream image after stabilization is output.

[0071] In this embodiment, during the process of implementing step S104, as shown in FIG. Figure 4 As shown, monitoring and analyzing the scene complexity and performance indicators of the current video stream images of each channel, and determining whether to reallocate system resources based on the analysis results include:

[0072] First, enter step S401 to monitor the relevant performance indicators of video anti-shake processing of each video stream data in real time, and write the collected relevant performance indicators into the Redis cache;

[0073] Then, the process proceeds to step S402 to input the data in the Redis cache into the trained neural network model and output the analysis result.

[0074] In step S401, the relevant performance indicators include the motion intensity M of the scene jitter. motion，i , Processing delay D delay，i , GPU utilization G gpu_usage_total 、Memory usage M memory-total，i , the video quality evaluation index Q of the previous and next frames quality，i , the specific calculation methods of various relevant performance indicators are as follows:

[0075] (1) Motion intensity M of scene jitter motion，i

[0076] Using the intermediate results of feature point tracking in the anti-shake module, calculate the normalized average feature point displacement:

[0077]

[0078] Among them, (xt，j ,y t，j ) and x t-1，j ,y t-1，j ) are the coordinates of the jth feature point in the current frame and the previous frame, respectively, N tracked is the number of successfully tracked feature points.

[0079] (2) Processing delay D delay，i

[0080] D delay，i =t out，i -t in，i ;

[0081] Among them, t out，i is the anti-shake algorithm output timestamp of the i-th video, t in，i Is the input timestamp.

[0082] (3) GPU utilization G gpu_usage_total

[0083] GPU parallel usage rate of a single camera G gpu_usage_my，i The calculation formula is as follows:

[0084]

[0085] The GPU parallel usage rate of all road cameras G gpu_usage_total The calculation formula is as follows:

[0086]

[0087] (4) Memory usage M memory-total，i

[0088] Memory usage of a single camera M memory-my，i The calculation formula is as follows:

[0089]

[0090] Memory usage of all road cameras M memory-total，i The calculation formula is as follows:

[0091]

[0092] (5) Video quality evaluation index Q of the previous and next frames after anti-shake quality，i

[0093] Use the simplified structural similarity index (SSIM) to evaluate the similarity between the previous frame after anti-shake and the current frame after anti-shake:

[0094] in:

[0095] μ1 and μ2 are the average brightness of the previous frame after anti-shake and the current frame after anti-shake, respectively;

[0096] σ1 and σ2 are the standard deviations of the previous frame after anti-shake and the current frame after anti-shake, respectively;

[0097] σ 12 is the covariance between the two frames;

[0098] C1 and C2 are small constants used to avoid zero denominators.

[0099] In step S402, the analysis result includes any one of maintaining the status quo, reducing computing resources, or increasing computing resources.

[0100] like Figure 5 As shown, the anti-shake program synchronously reports the above data to the distributed state storage management system (such as redis).

[0101] First, proceed to step S1: obtaining status data of multi-channel video anti-shake.

[0102] Specifically, the state data of multi-channel video stabilization is obtained from the distributed state storage management system as the known quantities of the constraint problem, and these known quantities can be used as input data of the neural network model.

[0103] Then proceed to step S2: solving the constraint problem to optimize the scheduling.

[0104] Specifically, the scheduling parameters for anti-shake of each video are calculated based on the scene complexity and performance indicators. The calculation method can give priority to using neural network prediction. If the output of the neural network does not meet the constraints, the benchmark greedy algorithm is used to output feasible scheduling parameters.

[0105] In short, if the analysis result of the neural network model is that system resources need to be reallocated, and the determined resource allocation adjustment strategy does not meet the preset constraints, the benchmark greedy algorithm is used to determine the feasible scheduling parameters for each video stream data, and a new resource allocation adjustment strategy is generated based on the scheduling parameters.

[0106] Then proceed to step S3: upload the solution of the constraint problem to the distributed state storage management system.

[0107] Specifically, the solution results of the constraint problem are uploaded to the distributed state storage management system, and the publish / subscribe function of Redis is used to trigger events, broadcast the scheduling results calculation, and notify other modules.

[0108] Then proceed to step S4: obtaining the current anti-shake resource allocation parameters.

[0109] The solution results of the constraint problem are uploaded to the distributed state storage management system, and the publish / subscribe function of Redis is used to trigger events, broadcast the completion of the scheduling result calculation, and notify other modules.

[0110] Monitor the event that the resource scheduling result calculation is completed. Once the event notification is received, obtain the scheduling parameters for anti-shake of this route from the distributed state storage management system.

[0111] In short, if it is determined based on the analysis results that system resources need to be reallocated, the publish or subscribe function of the Redis cache is used to trigger an event, and the feedback result that the system resources need to be reallocated is broadcast; the system resources of each video stream data are readjusted according to the resource allocation adjustment strategy, including: each video stream listens to the event notification of the Redis cache, and once the event notification is received, the system resources corresponding to its own video stream data are obtained according to the resource allocation adjustment strategy.

[0112] Then proceed to step S5: resource reallocation decision.

[0113] Determine the three states of maintaining the status quo, reducing computing resources, and increasing computing resources. Based on the anti-shake scheduling parameters and the current resource allocation situation, decide whether resources need to be reallocated. If reallocation is required, allocate resources according to the new resource allocation plan and report the processing results.

[0114] Then enter step S6: feedback loop.

[0115] Trigger a new round of processing and status monitoring.

[0116] In this embodiment, the design of the maximized overall performance index S of dynamic scheduling is as follows:

[0117] This indicator takes into account video quality and processing delay, using the aforementioned performance indicators and resource usage evaluation indicators as optimization parameters.

[0118] The optimization problem is stated as follows:

[0119]

[0120] In addition, gpu_cores，i and computing power requirements C computility-equired，i The constraint is positively correlated, that is, for any two video streams i and j (i≠j):

[0121] If C computility-equired，i ≤C computility-equired，j , then P gpu_cores，i ≤P gpu_cores，， .

[0122] Among them, C computility-equired，i The computing power requirement is calculated as follows:

[0123] (The specific method will be given later);

[0124] Among them, α importance，i is the importance weight of each video channel, and n is the number of video channels. quality and w delay If the application is more concerned with video quality, increase w quality If real-time performance is more important, increase w delay . f(D delay，i ) is a nonlinear penalty function for latency, calculated as follows:

[0125]

[0126] The constraints are:

[0127] (The total number of GPU parallelism does not exceed the number of available cores, for example, NVIDIA JetsonTX2 uses NVIDIA Pascal TM There are 256 CUDA cores, value P total_cores =256)

[0128] D delay，i ≤D max_delay (The processing delay of each video channel does not exceed the maximum allowed delay, for example, a value between 30 and 50 milliseconds);

[0129] G gpu_usage，i ≤G max_gpu_usage (The GPU utilization of each video channel does not exceed the maximum allowed value);

[0130] M memory，i ≤M max_memory (The memory usage of each video channel does not exceed the maximum allowed value);

[0131] R resolution，i ∈{480p, 720p, 1080p} (resolution can only take specified values);

[0132] P gpu_cores，i ∈{8, 16, 32, 64, 96, 128, 256} (the number of GPU parallel cores can only take the specified value);

[0133] The solution to this optimization problem will give the optimal resolution R for each video channel. resolution，i and the number of GPU parallel cores P gpu_cores，i , in order to maximize the overall performance index S.

[0134] The variables and parameters are described as follows:

[0135] Q quality，i : De-jitter quality index of the i-th video;

[0136] D delay，i : Processing delay of the i-th video (milliseconds);

[0137] G gpu_usage，i : GPU utilization of the i-th video channel;

[0138] M memory，i : Memory usage of the i-th video;

[0139] R resolution，i : The resolution of the i-th video;

[0140] P gpu_cores，i : The number of GPU parallel cores allocated to the i-th video channel;

[0141] P total_cores : Total number of available GPU parallel cores;

[0142] D max_delay : Maximum allowed processing delay;

[0143] G max_gpu_usage : Maximum allowed GPU utilization;

[0144] M max_memory : Maximum allowed memory usage;

[0145] This optimization problem now uses more descriptive and clear variable names, making the problem formulation easier to understand and implement.

[0146] About the quality evaluation index Q of the front and back frames of the anti-shake video quality，i , the description is as follows:

[0147] Use the simplified structural similarity index (SSIM) to evaluate the similarity between the previous frame after anti-shake and the current frame after anti-shake:

[0148]

[0149] in:

[0150] μ1 and μ2 are the average brightness of the previous frame after anti-shake and the current frame after anti-shake, respectively;

[0151] σ1 and σ2 are the standard deviations of the previous frame after anti-shake and the current frame after anti-shake, respectively;

[0152] σ 12 is the covariance between the two frames;

[0153] C1 and C2 are small constants used to avoid the denominator being zero, for example, C1 = C2 = 0.01;

[0154] α importance，i , w quality and w delay Practical suggestions for parameter selection:

[0155] Initialization: Initialize all weights using empirical values ​​or a uniform distribution.

[0156] Offline optimization: Use historical data and simulation scenarios to optimize weights through grid search, genetic algorithms, and other methods.

[0157] Online adaptation: During system operation, the weights are continuously fine-tuned based on actual performance and feedback.

[0158] A / B testing: Compare the effects of different weight combinations in a real environment.

[0159] Expert knowledge: Adjust weights based on the advice of domain experts, especially for specific application scenarios.

[0160] Periodic evaluation: Regularly evaluate system performance and adjust weights based on new requirements or hardware updates.

[0161] This optimization problem achieves global optimization of multi-channel video anti-shake acceleration by balancing multiple key factors (video quality, processing delay, resource constraints). Its design takes into account both theoretical completeness and operability in practical applications. By adjusting various weights and parameters, this framework can adapt to different application scenarios and requirements, providing a solid theoretical foundation for the implementation and optimization of practical systems.

[0162] The adjustable parameters of the anti-shake module are the input resolution and the number of GPU parallelism used.

[0163] Specifically, the input resolution is explained as follows: for example, 1080p color is converted to 720p black and white, or 480p black and white. This is used to estimate how to de-jitter in the end.

[0164] The explanation of the number of GPU parallelism used is as follows: the number of GPUs allocated to each camera is different, for example, less for static images and more for intense motion. But the value is 8 or 16, 32, 64, 96, 128, 256, etc., in order to make full use of GPU parallelism.

[0165] C computility-equired，i The method of obtaining the computing power requirement (checking the resource allocation rules) is as follows:

[0166] Given the motion intensity M of the scene jitter of the input multi-channel video motion，i and resolution R resolution，i In this case, computing power demand is positively correlated with the two.

[0167] That is, video streams with high motion intensity need to obtain more GPU core threads. Video streams with low motion intensity obtain fewer GPU core threads. When the motion intensity is almost 0, that is, when there is no shaking, the anti-shake algorithm is not started by default to release related computing resources.

[0168] Videos with higher resolutions require more GPU core threads, while video streams with lower resolutions require fewer GPU core threads.

[0169] Usually the computing power required is C computility-equired，i With P gpu_cores，i Positively correlated:

[0170] If C computility-equired，i ≤C computility-equired，j , then P gpu_cores，i ≤P gpu_cores，j .

[0171] Computing power requirement C computility-equired，i The method is as follows:

[0172] Usually considering the required computing power requirements C computility-equied，i ~M motion，i ·R resolution，i , considering two factors: motion intensity and resolution. It is simple and intuitive, easy to understand and implement. However, it assumes that the impact of motion intensity and resolution on computational requirements is linear, which is not accurate enough and does not take into account the possible basic computational overhead.

[0173] Taking these factors into consideration, we propose a new expression for computing power requirements:

[0174]

[0175] in:

[0176] a, b, and c are constants.

[0177] a represents the influence coefficient of exercise intensity on computing requirements.

[0178] b represents the basic computational overhead.

[0179] c represents the exponential effect of resolution on computational requirements (usually 1≤c≤2).

[0180] In practice, computational requirements are often not strictly linear with resolution. For example, when the resolution doubles, the amount of computation may increase by a factor of 4 (if it is a two-dimensional operation). Therefore, a power function is used. More suitable.

[0181] Basic overhead: Even in almost static scenes, the stabilization algorithm requires some basic computational resources. This is represented by the constant b.

[0182] Effect of exercise intensity: The factor a allows us to adjust the degree to which exercise intensity affects computational requirements.

[0183] Flexibility: By adjusting the values ​​of a, b, and c, this model can be adapted to different algorithms and hardware characteristics.

[0184] Image processing complexity: In many image processing tasks, the computational complexity is often superlinear with the number of pixels (i.e. resolution). For example, the complexity of some filtering operations may be O(n log n), where n is the number of pixels.

[0185] Experimental observation: In actual video processing systems, we often observe that when the resolution increases, the required computing resources do not simply increase linearly.

[0186] Algorithm characteristics: Different anti-shake algorithms may have different sensitivities to motion intensity and resolution. Introducing coefficients and exponents can better model these differences.

[0187] Hardware considerations: GPU processing typically involves parallel computations, whose efficiency may scale nonlinearly with problem size (here dominated by resolution).

[0188] The proposed expression provides greater flexibility and precision, and can better reflect the complex relationship between motion intensity, resolution, and required computational resources in real systems. By properly adjusting the parameters, it can guide resource allocation decisions more accurately.

[0189] Preferably, the values ​​are as follows: a=1.5, b=50, c=1.5.

[0190] When a = 1.5, this value represents the effect of exercise intensity on computational requirements. It is set to be greater than 1 because an increase in exercise intensity usually leads to a superlinear increase in computational requirements. 1.5 is a moderate value, indicating that a 50% increase in exercise intensity may increase computational requirements by about 75%.

[0191] When b = 50, this value represents the basic computational overhead, and the unit can be regarded as the equivalent number of GPU cores. Even in static scenes, the anti-shake algorithm requires a certain amount of computing resources for operations such as inter-frame comparison and feature extraction. 50 is a relatively conservative estimate and can be adjusted according to the complexity of the actual algorithm.

[0192] When c = 1.5, this exponent describes the impact of increased resolution on computational requirements. It is set to 1.5 because: if c = 1, it indicates a linear relationship, which usually underestimates the impact of increased resolution. If c = 2, it may overestimate the impact (unless it is a pure pixel-level operation). 1.5 is a compromise value, indicating that when the resolution doubles, the computational requirements increase by about 2.8 times (2^1.5≈2.8). Applying these values, our expression becomes:

[0193]

[0194] In practical applications, we can also refer to the following methods for optimization:

[0195] Initialization: Use these reference values ​​as a starting point.

[0196] Performance testing: Run videos of different motion intensities and resolutions on an actual system and record the GPU resources required.

[0197] Parameter tuning: Use the collected data to adjust the values ​​of a, b, and c through regression analysis or other optimization methods.

[0198] Validation: Run new tests with the adjusted parameters to ensure that resource allocation is more accurate.

[0199] Regular updates: As algorithms improve or hardware is upgraded, these parameters may need to be re-evaluated.

[0200] Consider dynamic parameters: In more complex systems, these parameters can even be adjusted dynamically based on real-time performance feedback.

[0201] In this embodiment, the scheduling of system resources related to each video stream data can be realized through the central coordination program module. The central coordination program module needs to ensure that resources are released before allocation, which is crucial to the stability of the system and the effectiveness of resource management.

[0202] The resource adjustment process is as follows:

[0203] 1. Situations where resources need to be reduced

[0204] For programs that need to reduce resources, the new lower resource configuration can be written to Redis. And a unique "reduce resources" event (such as event: reduce_resource: videol) is published for each program that needs to reduce resources.

[0205] Each anti-shake program that needs to reduce resources receives its own "reduced resources" event. These programs execute resource release operations. After each program completes resource release, a confirmation flag is set in Redis (such as status: resource_reduced: videol).

[0206] The confirmation phase includes the following steps:

[0207] a. The central scheduler monitors the confirmation flags of all programs that need to reduce resources.

[0208] b. Use Redis's transaction mechanism to ensure atomicity and check all confirmation flags.

[0209] c. When all programs confirm that the resource release is complete, the resource increase phase begins.

[0210] For the resource addition phase, the operation process is as follows:

[0211] a. The central dispatcher clears the previous confirmation flag.

[0212] b. For programs that require additional resources:

[0213] First write the new higher resource configuration to Redis, and then publish an "increase resource" event (such as event: increase resource).

[0214] c. The anti-shake program that needs to increase resources receives the "increase resources" event.

[0215] d. These programs perform resource addition operations.

[0216] e. After each program completes the resource increase, a confirmation flag is set in Redis (such as status:resourceincreased:video2).

[0217] Final confirmation stage:

[0218] a. The central dispatcher checks the confirmation flags of all programs again.

[0219] b. When all adjustments are confirmed, the entire resource adjustment process is completed.

[0220] Timeout and error handling:

[0221] a. Set a timeout for each stage.

[0222] b. If a program does not confirm completion within the specified time:

[0223] Error logs are recorded, with the option to force restart the program or roll back the entire adjustment process.

[0224] An example of Redis key and event design is as follows:

[0225] config: videol: gpu_cores: number of GPU cores allocated to video 1;

[0226] event: reduce_resource: video1: event of reducing resources for video 1;

[0227] status:resource_reduced:videol:Video 1 resource reduction confirmation flag;

[0228] event: increase_resource: global event of resource increase;

[0229] status:resource_increased:video2:Video 2 resource increase confirmation flag.

[0230] In implementing the above steps, you may also consider the following details:

[0231] Use Redis transactions (MULTI / EXEC) to ensure atomicity when checking multiple confirmation flags.

[0232] Use Redis's key expiration function (EXPIRE) to set a lifetime for the confirmation mark to avoid stale status.

[0233] Consider using Redis's Lua scripting capabilities to encapsulate complex checks and status update logic to improve efficiency and consistency.

[0234] This optimized method ensures the safe release and allocation of resources, and greatly reduces the risk of resource conflicts through clear phase division and confirmation mechanism. At the same time, the introduced timeout and error handling mechanism improves the robustness of the system and can better cope with various abnormal situations. This design not only meets the needs of asynchronous programs, but also provides a reliable resource management framework suitable for embedded system environments such as Jetson TX2.

[0235] Regarding the benchmark algorithm, the Enhanced Multi-stage Constrained Optimization Resource Allocation Algorithm, our ultimate goal is to design a reinforcement learning neural network that can solve the attached objective function and constraints. However, at the beginning, there is no training corpus, and the output of this neural network is not smart enough. We need to first use traditional methods to design a solution algorithm output as P gpu_cores，i , no resolution information is output, because the input resolution (e.g. 1080p) is used.

[0236] Specifically, the basic constraints of current multi-dimensional resource allocation satisfy the following constraints:

[0237]

[0238] In addition, P gpu_cores，i and computing power requirements C computility-equired，i Positive correlation constraint, that is, for any two video streams i and j (i≠j):

[0239] If C computility-equired，i ≤C computility-equired，j , then P gpu_cores，i ≤P gpu_cores，j .

[0240] Among them, M motion，i is the scene jitter motion intensity of the i-th video stream; P gpu_cores，i is the number of GPU core threads allocated to the i-th video stream; the calculation formula is as follows:

[0241]

[0242] Enhanced Multi-stage ConstrainedOptimization Resource Allocation Algorithm, this optimized version will pay more attention to ensuring that each video stream can get the minimum GPU resource allocation while maintaining the original optimization goals and constraints. This algorithm aims to allocate GPU resources to multiple video streams to achieve video stabilization. It uses a multi-stage optimization process to allocate appropriate GPU resources to each video stream as much as possible while satisfying various constraints.

[0243] The input parameters of the algorithm are as follows:

[0244] n: number of video streams;

[0245] M motion，i: The scene jitter motion intensity of each video stream;

[0246] R resolution，i : The resolution of each video stream;

[0247] P total_cores : Total number of available GPU cores;

[0248] G max_gpu_usage : Maximum allowed GPU usage;

[0249] M max_memory : Maximum allowed memory usage;

[0250] The output parameters of the algorithm are as follows:

[0251] P gpu_cores，i : The number of GPU cores allocated to each video stream.

[0252] The algorithm process is as follows:

[0253] Initialization and priority calculation phase:

[0254] a. Calculate the computational requirements for each video stream i:

[0255]

[0256] b. Initialize the minimum guaranteed resource P min_cores =8;

[0257] c. Create a priority queue Q, according to C computility-required，i Sort the video stream in descending order;

[0258] Minimum resource guarantee allocation phase:

[0259] a. Set P remaining =P total_cores ;

[0260] b. For each video stream i:

[0261] If P remaining ≥P min_cores , set P gpu_cores，i =P min_cores , P remaining =P remaining -P min_cores ;

[0262] Otherwise: Set P gpu_cores，i =0.

[0263] Add i to the unassigned list U.

[0264] Dynamic constraint adjustment phase:

[0265] a. If there are video streams with unallocated resources:

[0266] Dynamic Adjustment G max_gpu_ussage and M max_memory , making it slightly looser;

[0267] b. Apply the relaxed constraints to the video streams that have not been allocated resources and try to allocate them again;

[0268] Priority-based resource allocation phase:

[0269] For each video stream i in the priority queue Q:

[0270] a. Calculate the maximum additional resources that can be allocated:

[0271] P max_extra，i =min(P remaining , 256-P gpu_cores，i );

[0272] b. Estimate ideal additional resources:

[0273]

[0274] c. Apply the packing strategy and select the one closest to P ideal_extra，i Allowed values:

[0275] P extra，i =max({x∈{0, 8, 16, 32, 64, 96, 128, 256}: x≤min(P ideal_extra，i , P max_extra，i )})-P gpu_cores，i ;

[0276] d. Update allocation:

[0277] P gpu_cores，i =P gpu_cores，i +P extra，i ;

[0278] e. Update remaining resources:

[0279] P remaining =P remaining -P extra，i ;

[0280] Constraint satisfaction verification and adjustment phase:

[0281] For each video stream i:

[0282] a. Check GPU usage and memory constraints:

[0283] If G gpu_usage，i >G max_gpu_usage or M memory，i >Mmax_memory :

[0284] Use binary search to find the maximum allowed P that satisfies the constraints gpu_cores，i value, and update P remaining ;

[0285] b. Ensure P gpu_cores，i ≥P min_cores , unless the total resources are insufficient.

[0286] Resource utilization maximization stage:

[0287] When P remaining > 0 and can continue to allocate:

[0288] a. Traverse the priority queue Q:

[0289] Try to put P gpu_cores，i Raise to the next allowed value, and if all constraints are satisfied after the promotion, update P gpu_cores，i , otherwise, keep the original value.

[0290] b. If no further allocations can be made, exit the loop.

[0291] Final verification phase:

[0292] a. Ensure

[0293] b. Verify all P gpu_cores，i are all allowed values;

[0294] c. Check C computility-required，i and P gpu_cores，i positive correlation;

[0295] Output: Returns the final P of each video stream. gpu_cores，i value.

[0296] The above algorithm has the following characteristics:

[0297] Minimum resource guarantee: Ensure that each video stream receives the minimum GPU resource allocation possible.

[0298] Dynamic constraint adjustment: When resources are tight, constraints are appropriately relaxed to accommodate more video streams.

[0299] Priority-based allocation: Allocate resources based on the priority of computing needs.

[0300] Constraint satisfaction: Strictly adhere to the constraints of GPU utilization and memory usage.

[0301] Maximize resource utilization: Make full use of remaining resources through iterative optimization.

[0302] Positive correlation guarantee: Ensure that resource allocation remains positively correlated with computing demand.

[0303] This optimized algorithm not only maintains the original optimization objectives and constraint satisfaction characteristics, but also adds a guarantee for minimum resource allocation, improving the applicability and fairness of the algorithm in practical applications. It provides each video stream with the opportunity to obtain GPU resources for anti-shake processing, and only in extreme cases will resources not be allocated.

[0304] In some embodiments, the output of the neural network can be adjusted to the resolution of the input video stream data and the number of GPU parallelism used.

[0305] like Figure 6 As shown, the neural network architecture is described in detail as follows:

[0306] The input processing module includes a time series feature extraction submodule and a random number processing submodule. The time series feature extraction submodule specifically includes a layer type (such as a 3D convolutional network), an input dimension (F, n, K), and an output dimension (F', n', 256). The layer type of the random number processing submodule is a fully connected layer, the input dimension is 1, and the output dimension is 256.

[0307] The layer types of the feature fusion module include attention mechanism, the input is temporal features and random features, and the output dimension is 256.

[0308] The prediction module includes resource allocation prediction, whose layer type is a fully connected layer and whose output dimension is 2x n. The layer type of performance indicator prediction is a fully connected layer and whose output dimension is 1 (predicted performance indicator S).

[0309] The layer type of the output processing module is a fully connected layer plus Gumbel-Softmax, and the output dimension is 2x n.

[0310] The neural network model designed according to the above scheme has the following characteristics:

[0311] Modular design: Each module has a clear function, which makes it easy to optimize and replace independently.

[0312] 3D Convolutional Network: Effectively process spatiotemporal data and capture temporal and spatial features in historical data.

[0313] Attention mechanism: Fusion of temporal features and random features allows the model to focus on the most relevant information.

[0314] Prediction module: predicts resource allocation and performance metrics simultaneously, enabling the model to learn the relationship between the two.

[0315] Gumbel-Softmax: Generates discrete outputs while keeping the network differentiable.

[0316] Constraint satisfaction module: ensures that the output meets basic constraints and regenerates random numbers when necessary.

[0317] In actual application, the following operations can be performed to meet the actual operation requirements:

[0318] Input processing: Design an input processing module that conforms to the K xnx F+1 dimension.

[0319] Randomness: Through random number processing submodules and feature fusion, the ability to generate different results under the same input is achieved.

[0320] Prediction capability: The prediction module predicts resource allocation and performance indicators simultaneously, implicitly predicting the performance for the next 5 minutes.

[0321] Output format: The output processing module ensures that the output dimension is 2xn, which conforms to the required format.

[0322] The training strategy is as follows: use the basic algorithm output plus random values ​​as the initial training data, take the performance indicator S after 1 minute as the training target, and gradually increase the prediction time to eventually reach 5 minutes.

[0323] Performance maximization: By predicting resource allocation and performance metrics simultaneously, the model is able to learn how to maximize the performance metric S.

[0324] This design combines historical data analysis, randomness introduction, and performance prediction to create a system that can handle complex inputs, meet multi-dimensional constraints, and has predictive capabilities. Its modular structure allows for flexible adjustments and optimizations based on specific needs in the future. Through the prediction module, the system can learn to make decisions that maximize the overall performance indicator S after 5 minutes.

[0325] like Figure 5 As shown, the submodules included in the distributed autonomous adjustment module and the functions of each submodule are as follows:

[0326] 1. Resource configuration listener:

[0327] Continuously monitor resource scheduling signals from the distributed state storage management system.

[0328] When new resource configuration information is received, the dynamic resource adjuster is triggered immediately.

[0329] 2. Dynamic Resource Adjuster:

[0330] This is the core part of the module, responsible for making real-time adjustments based on the received resource configuration signals. It contains the following subcomponents:

[0331] a. Processing resolution adjustment: Dynamically adjust the processing resolution of the input video based on the allocated resources. Possible adjustments include lowering the resolution to reduce the computational burden, or increasing the resolution to improve the output quality.

[0332] b. GPU core usage adjustment: Adjust the number and method of GPU core usage based on the allocated GPU resources. This may involve adjusting parameters such as CUDA thread block size and grid size.

[0333] c. Memory usage optimization:

[0334] Dynamically adjust memory usage strategy based on allocated memory limits.

[0335] This may include adjusting buffer sizes, optimizing memory allocation and deallocation strategies, etc.

[0336] 3. Anti-shake module:

[0337] Receives resized input video.

[0338] Execute the video stabilization algorithm based on the current resource configuration.

[0339] Output the stabilized video.

[0340] 4. Resource Usage Reporter:

[0341] Regularly collect actual resource usage, including GPU usage, memory usage, current processing resolution, etc.

[0342] 5. Report the collected information to the distributed state storage management system.

[0343] 6. Resource configuration verification:

[0344] Before applying a new resource configuration, verify the legality and feasibility of the configuration.

[0345] Ensure that the new configuration does not cause system instability or excessive resource utilization.

[0346] 7. Smooth transition mechanism:

[0347] When making major resource adjustments, implement incremental changes to avoid sudden fluctuations in system performance. This may involve multiple small steps of adjustment rather than a large one-time change.

[0348] 8. Exception handling and recovery:

[0349] Monitor system operation status and detect abnormal situations.

[0350] When problems occur (such as resource allocation failure, performance degradation, etc.), implement the predetermined recovery strategy.

[0351] 9. Resource release confirmation:

[0352] When it is time to reduce resource usage, ensure that resources are released properly.

[0353] Send a confirmation message to the distributed state storage management system, indicating that the resources have been successfully released.

[0354] 10. Performance self-optimization:

[0355] Continue to optimize the parameters and strategies of the anti-shake algorithm within the allocated resources.

[0356] According to real-time performance indicators (such as processing frame rate, anti-shake quality, etc.), fine-tune internal parameters to achieve the best results.

[0357] The above design allows each stabilization algorithm instance to autonomously adjust its behavior based on the allocated resources while maintaining synchronization with the central coordinator. In this way, the system can strike a balance between global resource management and local performance optimization to achieve efficient and flexible multi-channel video stabilization processing.

[0358] In some embodiments, the method further includes: after each determination of a new resource allocation adjustment strategy, recording current timestamp information and status information of each channel of video stream data, generating a status report and uploading it.

[0359] In a second aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for intelligently pushing lecture information as described in the first aspect of the present invention is implemented.

[0360] The computer-readable storage medium may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories.

[0361] The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD ROM); the magnetic surface memory can be a disk memory or a tape memory.

[0362] The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM). The computer-readable storage medium described in the embodiments of the present invention is intended to include these and any other suitable types of memory.

[0363] like Figure 7 As shown, in the sixth aspect, the present invention provides an electronic device 10, including a processor 101 and a storage medium 102, on which a computer program is stored, and when the computer program is executed by the processor, the method for intelligently pushing lecture information as described in the first aspect of the present invention is implemented.

[0364] In some embodiments, the processor can be implemented by software, hardware, firmware or a combination thereof, and can use circuits, single or multiple application-specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), central processing units (CPU), controllers, microcontrollers, and at least one of microprocessors, so that the processor can execute some or all of the steps in the message push method described in the various embodiments of the present application, or any combination of the steps therein.

[0365] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of this application, this does not limit the scope of patent protection of this application. All technical solutions generated by replacing or modifying equivalent structures or equivalent processes based on the essential concept of this application using the contents recorded in the specification and drawings of this application, as well as directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, are included in the scope of patent protection of this application.

Claims

1. A video anti-shake acceleration method with multi-scene perception and thread scheduling, characterized in that: The method comprises the following steps: Initialize system resources; Receiving multiple video stream data inputs and preprocessing each of the video stream data, wherein the video stream data includes a video stream image; Monitor and analyze the scene complexity and performance indicators of each current video stream image, and decide whether system resources need to be reallocated based on the analysis results. If so, determine a resource allocation adjustment strategy based on the analysis results, and readjust the system resources of each video stream data according to the resource allocation adjustment strategy.

2. The video anti-shake acceleration method of multi-scene perception and thread scheduling as claimed in claim 1, characterized in that: The initialization system resources include: Obtaining the computing power resource parameters of the GPU, wherein the computing power resource parameters include the number of GPU cores, the size of shared memory, and the utilization rate of the current GPU; Allocate the computing power resources of the GPU to each channel of video stream data.

3. The video anti-shake acceleration method of multi-scene perception and thread scheduling as claimed in claim 1 or 2, characterized in that: Preprocessing the video stream data of each channel includes: Read basic parameter information of multiple video stream data, wherein the basic parameter information includes frame rate and / or resolution; Adaptively adjusting the resolution of each channel of video stream data according to the basic parameter information of the video stream data and the system resources allocated to each channel of video stream data; And the video stream images of each video stream data are subjected to motion estimation and smoothing processing, image transformation and resampling are performed according to the motion trajectory after smoothing processing, and the resampled video stream images are output.

4. The video anti-shake acceleration method of multi-scene perception and thread scheduling as claimed in claim 1, characterized in that: Monitoring and analyzing the scene complexity and performance indicators of the current video stream images of each channel, and determining whether to reallocate system resources based on the analysis results include: Monitor the performance indicators of video stabilization processing of each video stream data in real time, and write the collected performance indicators into the Redis cache; The data in the Redis cache is input into the trained neural network model, and an analysis result is output, where the analysis result includes any one of maintaining the status quo, reducing computing resources, or increasing computing resources.

5. The video anti-shake acceleration method of multi-scene perception and thread scheduling as claimed in claim 4, characterized in that: The relevant performance indicators include the motion intensity M of the scene jitter motion，i , Processing delay D delay，i , GPU utilization G gpu_usage_total 、Memory usage M memory-total，i , video quality assessment index Q for previous and next frames quality，i .

6. The video anti-shake acceleration method of multi-scene perception and thread scheduling as claimed in claim 4, characterized in that: The method comprises: If the analysis result of the neural network model is that system resources need to be reallocated, and the determined resource allocation adjustment strategy does not meet the preset constraints, the benchmark greedy algorithm is used to determine the feasible scheduling parameters for each video stream data, and a new resource allocation adjustment strategy is generated based on the scheduling parameters.

7. The video anti-shake acceleration method of multi-scene perception and thread scheduling as claimed in claim 4, characterized in that: The method comprises: If it is determined based on the analysis results that system resources need to be reallocated, the publish or subscribe function of the Redis cache is used to trigger an event and broadcast the feedback result that system resources need to be reallocated; The re-adjusting the system resources of each channel of video stream data according to the resource allocation adjustment strategy comprises: Each video stream monitors the event notification of the Redis cache, and once the event notification is received, the system resources corresponding to the video stream data of the own channel are obtained according to the resource allocation adjustment strategy.

8. The video anti-shake acceleration method of multi-scene perception and thread scheduling as claimed in claim 1, characterized in that: The method further comprises: After each new resource allocation adjustment strategy is determined, the current timestamp information and status information of each video stream data are recorded, and a status report is generated and uploaded.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video stabilization acceleration method of multi-scene perception and thread scheduling as described in any one of claims 1 to 8 is implemented.

10. An electronic device, characterized in that: It includes a processor and a storage medium, wherein a computer program is stored on the storage medium, and when the computer program is executed by the processor, the video stabilization acceleration method with multi-scene perception and thread scheduling as described in any one of claims 1 to 8 is implemented.

Images