Adaptive frame rate real-time video processing method and system

Abstract

The application provides a real-time video processing method and system with adaptive frame rate, which comprises the following steps: extracting multi-dimensional space-time domain feature parameters of a current frame; calculating marginal reservation utility value through a frame reservation effect function, marking the frame as a discarded frame when the marginal reservation utility value is lower than a dynamic utility threshold, otherwise marking the frame as a core key frame or a low-priority frame; constructing a multi-priority queue structure, collecting actual operation time consumption of each frame in each operation unit in real time, using an improved parameter exponential smoothing model to predict computing power and generating a computing power prediction value; using the computing power prediction value to perform logical operation on the queue according to frame type labels, constructing an operation unit set, and using the operation unit set and the computing power prediction value to perform logical operation on the queue structure according to the frame type labels to obtain frame processing data, and sorting the frame processing data to obtain corresponding ordered frames. The application can complete simplified operation using idle computing power without increasing the backlog of the main queue.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a real-time video processing method and system with adaptive frame rate. Background Technology

[0002] With the rapid development of digital image processing technology, real-time video analytics has been widely applied in scenarios such as classroom behavior analysis, facial recognition attendance, security monitoring, and video conferencing. In these applications, systems typically need to perform multi-dimensional analysis and computation on real-time video images (such as person recognition, behavior analysis, and attention assessment), and then overlay the results onto the image for display. However, practical applications generally face the following technical challenges: The raw video image needs to undergo multiple processing and computation steps sequentially. Due to limitations in computational complexity and hardware conditions, the processing time of intermediate steps is often much lower than the input speed of the image, resulting in an asymmetry between input and output. This leads to a gradual accumulation of latency during operation, severely degrading real-time performance. Existing technologies typically improve output performance by collecting image feature information to build a model and selecting optimization strategies, or by dynamically reducing the input frame rate to match the output capability. However, the former relies on a large amount of data for training and model iteration, has stringent deployment conditions, and poor transfer performance; the latter uses a stepped threshold adjustment, which has a significant lag from the start of adjustment to the effect taking effect. Furthermore, when the input frame rate is much higher than the output frame rate, a large number of images to be processed accumulate in the cache, resulting in severe latency. Summary of the Invention

[0003] Therefore, the purpose of this invention is to provide a real-time video processing method and system with adaptive frame rate, so as to at least solve the shortcomings of the above-mentioned technology.

[0004] This invention proposes a real-time video processing method with adaptive frame rate, comprising: Acquire video frame data stream, decode each frame of the video frame data stream to obtain a frame image, and extract multi-dimensional spatiotemporal domain feature parameters of the current frame, wherein the multi-dimensional spatiotemporal domain feature parameters include temporal domain features, spatial domain features and frequency domain features; Construct a frame retention utility function, and use the frame retention utility function and the spatiotemporal domain feature parameters to calculate the marginal retention utility value of the current frame. When the marginal retention utility value is lower than the dynamic utility threshold, mark the current frame as a discard frame; otherwise, mark the current frame as a core key frame or a low priority frame. Construct a multi-priority queue structure, in which each frame carries a frame type label, the frame type label including the core key frame and the low priority frame; The actual computation time of each frame in each computing unit is collected in real time. An improved parametric exponential smoothing model is used to predict computing power, and feedforward correction is performed in combination with the real-time backlog length of the queue to be computed to generate a computing power prediction value. A set of computing units is constructed, and logical operations are performed on the queue structure according to the frame type label using the set of computing units and the predicted computing power value to obtain frame processing data. The frame processing data is then sorted to obtain the corresponding ordered frames.

[0005] Furthermore, the steps of acquiring the video frame data stream, decoding each frame of the video frame data stream to obtain a frame image, and extracting multi-dimensional spatiotemporal feature parameters of the current frame include: The raw data stream is obtained from the video source and decoded to obtain independent frame images; The frame image is divided into multiple non-overlapping macroblocks, a reference frame is defined, and within the search range of the reference frame, the most similar matching block is found for each macroblock in the current frame. The spatial displacement between the matching block and the current macroblock is calculated to obtain the corresponding temporal features. A three-dimensional joint entropy is constructed, and the three-dimensional joint entropy is used to perform color space feature statistics on the frame image to obtain the corresponding spatial domain features; The frame image is transformed from the spatial domain to the frequency domain using the discrete cosine transform algorithm, and the proportion of high-frequency energy is calculated to obtain the corresponding frequency domain features.

[0006] Furthermore, the formula for calculating the frame retention utility function is as follows:

[0007] In the formula, This indicates the number of frames currently backed up in the queue to be processed. This represents the ideal backlog in the queue to be processed. This indicates the average time required to complete the processing of one frame. This represents the computation time per frame under no-load conditions. Indicates the penalty curvature coefficient; The formula for calculating the marginal retention utility value is as follows:

[0008] In the formula, Indicates the current frame The three-dimensional joint entropy, This represents the theoretical maximum value of the three-dimensional joint entropy. Indicates the current frame The time-domain characteristics, This represents the normalized reference maximum value of the time-domain features. The weighting coefficients representing the returns of the three-dimensional joint entropy. The return weight coefficients representing time-domain characteristics, This represents the penalty weighting coefficient; The formula for calculating the dynamic utility threshold is:

[0009] In the formula, This represents the preset basic utility threshold. Indicates the load sensitivity coefficient. This represents the total number of processing units used to process video frame data streams. This represents the random disturbance coefficient. Indicates recent The standard deviation of the marginal retention utility value of a frame.

[0010] Furthermore, the steps for constructing a multi-priority queue structure include: The upper and lower backlog thresholds of the operation queue are calculated respectively, and the corresponding multi-priority strategy is constructed by combining the upper and lower backlog thresholds with the real-time backlog length of the queue to be operated. Verification rules are generated based on the multi-priority strategy and the frame type label, and a multi-priority queue structure is obtained using the verification rules.

[0011] Furthermore, the steps for generating the predicted computing power value include: real-time acquisition of the actual computation time of each frame in each computing unit, using an improved parametric exponential smoothing model for computing power prediction, and combining this with the real-time backlog length of the queue to be computed for feedforward correction. Regarding the first Frame, calculate its position in the whole The original maximum time consumed in each processing unit:

[0012] In the formula, Indicates the first Frame in The actual computation time in each arithmetic unit; If the current frame If the frame type label is a low-priority frame, then its correction factor is calculated. :

[0013] In the formula, Indicates recent The weighted average of the maximum processing time of all core keyframes in each processing unit. Indicates recent The weighted average of the maximum processing time of all low-priority frames in each processing unit. Indicates the strengthening coefficient. This is an indicator function that takes the value 1 when the condition is true and 0 otherwise. Indicates recent The arithmetic mean of the original maximum time taken for each frame. Indicates recent Standard deviation of the original maximum frame time; Generate the corresponding effective update time based on the frame type label. :

[0014] In the formula, Indicates the current frame Frame type label; Get prediction step size For the first Frames, using their effective update time As observed values, the corresponding horizontal components, trend components, and periodic components are calculated. The prediction step size is calculated based on the horizontal component, the trend component, and the periodic component. The forward prediction base value is obtained, and feedforward correction is performed based on the forward prediction base value and the real-time backlog length of the queue to be processed to generate the computing power prediction value.

[0015] This invention also proposes an adaptive frame rate real-time video processing system, comprising: The data acquisition module is used to acquire video frame data stream, decode each frame of the video frame data stream to obtain a frame image, and extract multi-dimensional spatiotemporal domain feature parameters of the current frame, wherein the multi-dimensional spatiotemporal domain feature parameters include temporal domain features, spatial domain features and frequency domain features. The data comparison module is used to construct a frame retention utility function and calculate the marginal retention utility value of the current frame using the frame retention utility function and the spatiotemporal domain feature parameters. When the marginal retention utility value is lower than the dynamic utility threshold, the current frame is marked as a discarded frame; otherwise, the current frame is marked as a core key frame or a low priority frame. The structure building module is used to build a multi-priority queue structure, in which each frame carries a frame type label, the frame type label including the core key frame and the low priority frame; The computing power prediction module is used to collect the actual computing time of each frame in each computing unit in real time, and uses an improved parametric exponential smoothing model to predict computing power. It also combines the real-time backlog length of the queue to be computed for feedforward correction to generate the computing power prediction value. The data processing module is used to construct a set of computing units, and to perform logical operations on the queue structure according to the frame type label using the set of computing units and the computing power prediction value to obtain frame processing data, and to sort the frame processing data to obtain the corresponding ordered frames.

[0016] Furthermore, the data acquisition module is specifically used for: The raw data stream is obtained from the video source and decoded to obtain independent frame images; The frame image is divided into multiple non-overlapping macroblocks, a reference frame is defined, and within the search range of the reference frame, the most similar matching block is found for each macroblock in the current frame. The spatial displacement between the matching block and the current macroblock is calculated to obtain the corresponding temporal features. A three-dimensional joint entropy is constructed, and the three-dimensional joint entropy is used to perform color space feature statistics on the frame image to obtain the corresponding spatial domain features; The frame image is transformed from the spatial domain to the frequency domain using the discrete cosine transform algorithm, and the proportion of high-frequency energy is calculated to obtain the corresponding frequency domain features.

[0017] Furthermore, the structure building module is specifically used for: The upper and lower backlog thresholds of the operation queue are calculated respectively, and the corresponding multi-priority strategy is constructed by combining the upper and lower backlog thresholds with the real-time backlog length of the queue to be operated. Verification rules are generated based on the multi-priority strategy and the frame type label, and a multi-priority queue structure is obtained using the verification rules.

[0018] Furthermore, the computing power prediction module is specifically used for: Regarding the first Frame, calculate its position in the whole The original maximum time consumed in each processing unit:

[0019] In the formula, Indicates the first Frame in The actual computation time in each arithmetic unit; If the current frame If the frame type label is a low-priority frame, then its correction factor is calculated. :

[0020] In the formula, Indicates recent The weighted average of the maximum processing time of all core keyframes in each processing unit. Indicates recent The weighted average of the maximum processing time of all low-priority frames in each processing unit. Indicates the strengthening coefficient. This is an indicator function that takes the value 1 when the condition is true and 0 otherwise. Indicates recent The arithmetic mean of the original maximum time taken for each frame. Indicates recent Standard deviation of the original maximum frame time; Generate the corresponding effective update time based on the frame type label. :

[0021] In the formula, Indicates the current frame Frame type label; Get prediction step size For the first Frames, using their effective update time As observed values, the corresponding horizontal components, trend components, and periodic components are calculated. The prediction step size is calculated based on the horizontal component, the trend component, and the periodic component. The forward prediction base value is obtained, and feedforward correction is performed based on the forward prediction base value and the real-time backlog length of the queue to be processed to generate the computing power prediction value.

[0022] The present invention also proposes a storage medium on which a computer program is stored, which, when executed by a processor, implements the above-described adaptive frame rate real-time video processing method.

[0023] The present invention also proposes a computer, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described adaptive frame rate real-time video processing method.

[0024] Compared with existing technologies, the adaptive frame rate real-time video processing method and system of this invention quantifies the retention value of each frame by constructing a frame retention utility function. When the load is high, only frames with a utility value exceeding a dynamic threshold are retained, thereby controlling the number of frames entering the computation pipeline at the source and effectively avoiding latency accumulation. By setting a dynamic utility threshold, it can automatically adjust according to the real-time load status. When the load increases, the threshold automatically increases, and only frames of extremely high importance can be enqueued. When the load decreases, the threshold automatically decreases, allowing more frames to be retained to make full use of idle computing power. By setting low-priority frames to not occupy the main queue storage space, but instead directly injecting them into the input queue of the current idle computation unit, simplified computation is completed using idle computing power without aggravating the backlog in the main queue. Attached Figure Description

[0025] Figure 1 This is a flowchart of the real-time video processing method with adaptive frame rate in the first embodiment of the present invention; Figure 2 This is a structural block diagram of the adaptive frame rate real-time video processing system according to the second embodiment of the present invention; Figure 3 This is a structural block diagram of the computer in the third embodiment of the present invention.

[0026] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0027] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0029] Example 1 Please see Figure 1 The figure shows a real-time video processing method with adaptive frame rate according to the first embodiment of the present invention, the method specifically including steps S101 to S105: S101, acquire video frame data stream, decode each frame of the video frame data stream to obtain frame image, and extract multi-dimensional spatiotemporal domain feature parameters of the current frame, wherein the multi-dimensional spatiotemporal domain feature parameters include temporal domain features, spatial domain features and frequency domain features. Furthermore, step S101 specifically includes steps S1011 to S1014: S1011, Obtain the raw data stream from the video source and decode the raw data stream to obtain independent frame images; S1012, the frame image is divided into multiple non-overlapping macroblocks, a reference frame is defined, and within the search range of the reference frame, the most similar matching block is found for each macroblock in the current frame, and the spatial displacement between the matching block and the current macroblock is calculated to obtain the corresponding temporal features. S1013, construct a three-dimensional joint entropy, and use the three-dimensional joint entropy to perform color space feature statistics on the frame image to obtain the corresponding spatial domain features; S1014, the frame image is converted from the spatial domain to the frequency domain by the discrete cosine transform algorithm, and the high-frequency energy ratio is calculated to obtain the corresponding frequency domain features.

[0030] In practice, the raw data stream is obtained from the video source (such as a live webcam stream, a local webcam, or a video file). This raw data stream can be data extracted from scenarios such as classroom behavior analysis, facial recognition attendance, security monitoring, and video conferencing. The data stream is then decoded to obtain independent frame images. ,in, This indicates the time of the frame in the time series. Basic analysis of the image of this frame is performed to facilitate subsequent feature extraction.

[0031] Furthermore, the decoded frame image is analyzed from three dimensions. Quantitative analysis is conducted to comprehensively characterize its properties. Among these: 1. Temporal features are used to describe the displacement information of objects in an image between adjacent frames. Specifically: The current frame image The macroblock is divided into multiple non-overlapping macroblocks. Within the search range of the reference frame (in this embodiment, the reference frame is set to the previous keyframe), the most similar matching block is found for each macroblock in the current frame. The spatial displacement between the matching block and the current macroblock is calculated, which is the motion vector of the macroblock.

[0032] 2. Spatial domain features are used to capture the structural information of the image. In this implementation, three-dimensional joint entropy is used. For quantization, the 3D joint entropy considers the spatial relationship between a pixel and its neighboring pixels. The specific calculation formula is as follows:

[0033] In the formula, These represent triplets consisting of a pixel in the current frame and the gray values ​​of its right and bottom adjacent pixels (ranging from 0 to 255). This indicates the grayscale triplet. The joint probability of occurrence throughout the entire frame image. It represents a very small positive number.

[0034] Furthermore, frequency domain features are extracted using the Discrete Cosine Transform (DCT) algorithm. The DCT algorithm transforms the image from the spatial domain to the frequency domain, concentrating image energy on a few low-frequency coefficients. Specifically: For frame images The image is divided into blocks (8x8 pixel blocks in this embodiment), and DCT is performed on each block to obtain DCT coefficients. ,in, These are frequency domain coordinates. Calculate the proportion of high-frequency energy:

[0035] In the formula, Representing the high-frequency region, defined as satisfying All frequency domain coordinates The set, This indicates the cutoff frequency threshold that distinguishes between high and low frequencies. It represents the entire frequency range, that is, the set of all frequency domain coordinates. It represents a very small positive number.

[0036] After extracting the features from the three dimensions obtained above, these features are synthesized to integrate the motion vector information of all macroblocks:

[0037] In the formula, This indicates the total number of macroblocks in the current frame. Indicates the first The magnitude of the motion vector within a macroblock, i.e., the magnitude of the displacement. Indicates the first The variance of the motion vectors within a macroblock, This represents a preset variance reference value; Using frequency domain features as a correction factor, Weighting is performed to obtain the corresponding frequency domain features:

[0038] In the formula, This represents the high-frequency correction coefficient.

[0039] S102, construct a frame retention utility function, and use the frame retention utility function and the spatiotemporal domain feature parameters to calculate the marginal retention utility value of the current frame. When the marginal retention utility value is lower than the dynamic utility threshold, mark the current frame as a discarded frame; otherwise, mark the current frame as a core key frame or a low priority frame. Furthermore, the formula for calculating the frame retention utility function is as follows:

[0040] In the formula, This indicates the number of frames currently backed up in the queue to be processed. This represents the ideal backlog in the queue to be processed. This indicates the average time required to complete the processing of one frame. This represents the computation time per frame under no-load conditions. Indicates the penalty curvature coefficient; The formula for calculating the marginal retention utility value is as follows:

[0041] In the formula, Indicates the current frame The three-dimensional joint entropy, This represents the theoretical maximum value of the three-dimensional joint entropy. Indicates the current frame The time-domain characteristics, This represents the normalized reference maximum value of the time-domain features. The weighting coefficients representing the returns of the three-dimensional joint entropy. The return weight coefficients representing time-domain characteristics, This represents the penalty weighting coefficient; The formula for calculating the dynamic utility threshold is:

[0042] In the formula, This represents the preset basic utility threshold. Indicates the load sensitivity coefficient. This represents the total number of processing units used to process video frame data streams. This represents the random disturbance coefficient. Indicates recent The standard deviation of the marginal retention utility value of a frame.

[0043] Furthermore, the marginal retention utility value of the current frame is... With dynamic utility threshold Comparisons were made, and a secondary core offset coefficient was introduced. ( To distinguish between "core keyframes" and "low-priority frames": Scenario 1 (Discard): If If the frame is marked as a discarded frame, it means that the content of the frame is of low importance and the current load is high, and the "benefit" of retaining the frame is not enough to offset the "latency cost" it brings. In this case, the frame is directly marked as a discarded frame and is not included in the queue for processing, thus completely releasing the computing resources occupied by the frame. Case 2 (low priority frame): If If the frame is marked as low priority (e.g., a slight change in the image), it means that the frame has some value for preservation but has not yet reached the level of "requiring full and detailed processing". In this case, the frame is marked as a low priority frame and will be sent to the queue for processing.

[0044] Case 3 (Core Keyframe): If This means that the content of the frame is extremely important, and its utility value far exceeds the current entry threshold. In this case, the frame is marked as a core key frame, and it will also be sent to the queue for processing.

[0045] Understandable, secondary core offset coefficient The introduction of this factor creates a "hysteresis interval" at the decision boundary. This effectively prevents frame types from frequently changing in critical regions.

[0046] S103, construct a multi-priority queue structure, in which each frame carries a frame type label, the frame type label including the core key frame and the low priority frame; Furthermore, step S103 specifically includes steps S1031 to S1032: S1031, calculate the upper limit threshold and lower limit threshold of the backlog of the operation queue respectively, and use the upper limit threshold and the lower limit threshold in combination with the real-time backlog length of the queue to be operated to construct the corresponding multi-priority strategy; S1032, Generate verification rules based on the multi-priority strategy and the frame type label, and use the verification rules to obtain a multi-priority queue structure.

[0047] In practical implementation, the upper and lower backlog thresholds of the operation queue are calculated separately. The formula for calculating the upper backlog threshold is as follows:

[0048] In the formula, This represents the upper limit adjustment coefficient; The formula for calculating the backlog lower limit threshold is:

[0049] In the formula, This represents the lower limit adjustment coefficient; Furthermore, based on the current real-time backlog length of the queue to be processed. Based on the relative relationship between the two thresholds mentioned above, a three-tiered differentiated admission strategy is implemented: (1) Idle state ( ) This indicates that there is sufficient idle computing power in the downstream processing units, and both core key frames and low-priority frames are allowed to enter the main queue area of ​​the waiting queue for processing normally. In this state, the goal is to "maximize computing power utilization" and make full use of idle resources to process more frames (including low-priority frames).

[0050] (2) Load state ( ) This indicates that downstream computing units are in a normal load range with a small amount of margin for maneuver, but frame priorities must be considered. In this state, frames are allowed to enter the main queue of the processing queue and wait for complete computation. Low-priority frames do not enter the main queue; instead, they are written directly to the tail of the input queue of currently idle computing units without occupying main queue storage space. If there are no idle computing units, the low-priority frame is temporarily stored in a very small bypass buffer (depth set to 1-2), waiting to be injected immediately after the next computing unit completes its current task. This method ensures that low-priority frames can complete simplified computations using idle computing power without exacerbating main queue backlog.

[0051] (3) Crowded state ( ) This indicates that the downstream processing units are severely overloaded and the main queue is congested to a dangerous level. In this state, all frame enqueue operations are suspended (both core critical frames and low-priority frames are blocked), and the queue congestion needs to be cleared.

[0052] In this embodiment, when the queue is crowded, in order to avoid the delay from growing indefinitely, all low-priority frames in the main queue area of ​​the queue to be processed are scanned, the marginal retention utility value of each frame is extracted, and the frames are sorted in ascending order according to the value from smallest to largest. Starting from the low-priority frame with the smallest utility value, the frames are removed from the main queue in turn and the image memory they occupy is released. The operation is repeated until there are no low-priority frames to be removed from the queue or the queue enters the idle state.

[0053] S104 collects the actual computation time of each frame in each computing unit in real time, uses an improved parametric exponential smoothing model to predict computing power, and combines the real-time backlog length of the queue to be computed to perform feedforward correction to generate a computing power prediction value. Furthermore, step S104 specifically includes steps S1041 to S1045: S1041, targeting the first Frame, calculate its position in the whole The original maximum time consumed in each processing unit:

[0054] In the formula, Indicates the first Frame in The actual computation time in each arithmetic unit; S1042, if the current frame If the frame type label is a low-priority frame, then its correction factor is calculated. :

[0055] In the formula, Indicates recent The weighted average of the maximum processing time of all core keyframes in each processing unit. Indicates recent The weighted average of the maximum processing time of all low-priority frames in each processing unit. Indicates the strengthening coefficient. This is an indicator function that takes the value 1 when the condition is true and 0 otherwise. Indicates recent The arithmetic mean of the original maximum time taken for each frame. Indicates recent Standard deviation of the original maximum frame time; S1043, Generate the corresponding effective update time based on the frame type label. :

[0056] In the formula, Indicates the current frame Frame type label; S1044, Obtain the prediction step size For the first Frames, using their effective update time As observed values, the corresponding horizontal components, trend components, and periodic components are calculated. S1045, calculate the prediction step size based on the horizontal component, the trend component, and the periodic component. The forward prediction base value is obtained, and feedforward correction is performed based on the forward prediction base value and the real-time backlog length of the queue to be processed to generate the computing power prediction value.

[0057] In this embodiment, an improved parametric exponential smoothing model is used for prediction, which includes three dimensions: horizontal component, trend component, and periodic component. This model can simultaneously track the long-term baseline, trend, and periodic fluctuations of computing power.

[0058] Regarding the first Frames, using their effective update time As observed values, the corresponding horizontal component, trend component, and periodic component are calculated. The formula for calculating the horizontal component is as follows:

[0059] In the formula, Indicates the first The horizontal component of time (unit: milliseconds). Indicates the horizontal smoothing coefficient. Indicates lag Periodic component values ​​(unit: milliseconds) for each cycle. The period length (unit: frames) represents the frame interval in which the computation time fluctuates periodically. Indicates the previous moment The horizontal component (unit: milliseconds). This represents the trend component from the previous moment (unit: milliseconds / frame). The formula for calculating the trend component is:

[0060] In the formula, Indicates the first Trend components at any given time (unit: milliseconds / frame). Indicates the trend smoothing coefficient; The formula for calculating the periodic component is:

[0061] In the formula, Indicates the first Periodic components of time (unit: milliseconds). This represents the periodic smoothing coefficient.

[0062] Based on the updated three parameters, first calculate Forward prediction base value (Unit: milliseconds):

[0063] In the formula, Represents the periodic component value (in milliseconds) corresponding to the historical phase point. In the predicted base value Two feedforward correction terms are superimposed to generate the final computing power prediction value. (Unit: milliseconds):

[0064] In the formula, This represents the feedforward adjustment weight (unit: milliseconds / frame). This represents the excess queue backlog length (unit: frames), when the queue length exceeds the total number of processing units. This indicates that there is additional queuing delay that is not immediately processed by the pipeline; this excess is related to the feedforward adjustment weights. It is added to the predicted value as a penalty term. Represents the differential feedforward coefficients (unit: milliseconds) 2 / frame), The first-order time derivative (unit: frames / milliseconds) represents the length of the queue to be processed.

[0065] S105, construct a set of computing units, and use the set of computing units and the computing power prediction value to perform logical operations on the queue structure according to the frame type label to obtain frame processing data, and sort the frame processing data to obtain the corresponding ordered frames.

[0066] In specific implementation, The computational units form a set of computational units, and the queue structure described above is logically operated on according to frame type using this set of computational units and the obtained computing power prediction value. Specifically, for frame type labels that are core key frames, full-precision ESPCN super-resolution computation is performed. The structure of ESPCN (Efficient Sub-Pixel Convolutional Neural Network) is as follows: The input is a low-resolution image of three-channel YCbCr, but considering visual sensitivity and computational efficiency, only the luminance channel (Y channel) is actually fed into the ESPCN network (size: [insert size here]). The remaining two channels (Cb and Cr) are merged at the output through nearest neighbor interpolation.

[0067] First convolutional layer (feature extraction): The convolution kernel size is 5x5, the number of output channels is 64, the activation function is ReLU, and the output feature map size is: ; Second convolutional layer (non-linear mapping): The convolution kernel size is 3x3, the number of output channels is 32, the activation function is ReLU, and the output feature map size is: ; Third convolutional layer (upsampling preparation): The convolution kernel size is 3x3, and the number of output channels is Activation function: ReLU function, output feature map size: ,in, This is the super-resolution upsampling rate.

[0068] The size is The feature map is periodically rearranged pixel by pixel, and the rearrangement rule is as follows: Located in the same spatial position on each channel The pixels, unfolded into Position 4 adjacent sub-pixels. After sub-pixel convolution, the output feature map size becomes 10 ... That is, it is completed Super-resolution upsampling by a factor of 1.

[0069] Finally, the high resolution output of ESPCN will be... The channel is merged with the Cb and Cr channels, which have been magnified by nearest-neighbor interpolation, to obtain the final high-resolution color image with a size of [size missing]. .

[0070] For frames with low-priority frame type labels, the full ESPCN is no longer executed. Instead, a super-resolution processing bypass is performed based on sub-pixel convolutional cascade degradation. This mechanism dynamically reduces the number of output channels in the third convolutional layer, thereby skipping a large number of high-frequency reconstruction filters and significantly reducing computational load.

[0071] Formula for calculating the number of dynamic output channels:

[0072] In the formula, This represents the actual number of output channels of the third convolutional layer after cascading degradation. Indicates the super-resolution upsampling rate. This represents the floor function. This represents the natural exponential function. This represents the degraded curvature coefficient. This indicates the minimum number of channels to reserve.

[0073] The third convolutional layer originally had There are 1 filter, each responsible for generating the output feature map of one channel. When the number of output channels is compressed to 1... At that time, the logic operation layer performs the following operations: 1. From the original In a set of filters, the top filters are retained according to frequency importance. One low-frequency reconstruction filter is responsible for the basic structure and brightness information of the image, while the other high-frequency filters responsible for edge sharpening and texture details are skipped. 2. The size of the output feature map of the third convolutional layer after cropping is... ; 3. Perform standard subpixel convolutional reconstruction on the cropped feature map, and... The channels are rearranged to size .because The rearranged high-resolution image will have unfilled "holes" in space. At this time, fast neighborhood filling is used to fill the gaps by taking the average of the nearest effective sub-pixels to form the final low-precision complete high-resolution output. All the processed frame data are sorted to obtain the corresponding ordered frames, and the ordered frames are displayed.

[0074] In summary, the adaptive frame rate real-time video processing method in the above embodiments of the present invention constructs a frame retention utility function to quantify the retention value of each frame. When the load is high, only frames with a utility value exceeding a dynamic threshold are retained, thereby controlling the number of frames entering the computation pipeline at the source and effectively avoiding latency accumulation. By setting a dynamic utility threshold, it can automatically adjust according to the real-time load status. When the load increases, the threshold automatically increases, and only frames of extremely high importance can be enqueued. When the load decreases, the threshold automatically decreases, allowing more frames to be retained to make full use of idle computing power. By setting low-priority frames to not occupy the main queue storage space but directly injecting them into the input queue of the current idle computation unit, simplified computation is completed using idle computing power without aggravating the backlog in the main queue.

[0075] Example 2 In another aspect, this invention also proposes a real-time video processing system with adaptive frame rate; please refer to [link / reference]. Figure 2 The figure shows a real-time video processing system with adaptive frame rate according to a second embodiment of the present invention. The system includes: The data acquisition module 11 is used to acquire video frame data stream, decode each frame of the video frame data stream to obtain a frame image, and extract multi-dimensional spatiotemporal domain feature parameters of the current frame, wherein the multi-dimensional spatiotemporal domain feature parameters include temporal domain features, spatial domain features and frequency domain features. The data comparison module 12 is used to construct a frame retention utility function and calculate the marginal retention utility value of the current frame using the frame retention utility function and the spatiotemporal domain feature parameters. When the marginal retention utility value is lower than the dynamic utility threshold, the current frame is marked as a discarded frame; otherwise, the current frame is marked as a core key frame or a low priority frame. The structure building module 13 is used to build a multi-priority queue structure, in which each frame carries a frame type label, the frame type label including the core key frame and the low priority frame; The computing power prediction module 14 is used to collect the actual computing time of each frame in each computing unit in real time, use an improved parametric exponential smoothing model to predict computing power, and combine the real-time backlog length of the queue to be computed to perform feedforward correction to generate computing power prediction value. The data processing module 15 is used to construct a set of computing units, and to perform logical operations on the queue structure according to the frame type label using the set of computing units and the computing power prediction value to obtain frame processing data, and to sort the frame processing data to obtain the corresponding ordered frames.

[0076] Furthermore, the data acquisition module 11 is specifically used for: The raw data stream is obtained from the video source and decoded to obtain independent frame images; The frame image is divided into multiple non-overlapping macroblocks, a reference frame is defined, and within the search range of the reference frame, the most similar matching block is found for each macroblock in the current frame. The spatial displacement between the matching block and the current macroblock is calculated to obtain the corresponding temporal features. A three-dimensional joint entropy is constructed, and the three-dimensional joint entropy is used to perform color space feature statistics on the frame image to obtain the corresponding spatial domain features; The frame image is transformed from the spatial domain to the frequency domain using the discrete cosine transform algorithm, and the proportion of high-frequency energy is calculated to obtain the corresponding frequency domain features.

[0077] Furthermore, the structure building module 13 is specifically used for: The upper and lower backlog thresholds of the operation queue are calculated respectively, and the corresponding multi-priority strategy is constructed by combining the upper and lower backlog thresholds with the real-time backlog length of the queue to be operated. Verification rules are generated based on the multi-priority strategy and the frame type label, and a multi-priority queue structure is obtained using the verification rules.

[0078] Furthermore, the computing power prediction module 14 is specifically used for: Regarding the first Frame, calculate its position in the whole The original maximum time consumed in each processing unit:

[0079] In the formula, Indicates the first Frame in The actual computation time in each arithmetic unit; If the current frame If the frame type label is a low-priority frame, then its correction factor is calculated. :

[0080] In the formula, Indicates recent The weighted average of the maximum processing time of all core keyframes in each processing unit. Indicates recent The weighted average of the maximum processing time of all low-priority frames in each processing unit. Indicates the strengthening coefficient. This is an indicator function that takes the value 1 when the condition is true and 0 otherwise. Indicates recent The arithmetic mean of the original maximum time taken for each frame. Indicates recent Standard deviation of the original maximum frame time; Generate the corresponding effective update time based on the frame type label. :

[0081] In the formula, Indicates the current frame Frame type label; Get prediction step size For the first Frames, using their effective update time As observed values, the corresponding horizontal components, trend components, and periodic components are calculated. The prediction step size is calculated based on the horizontal component, the trend component, and the periodic component. The forward prediction base value is obtained, and feedforward correction is performed based on the forward prediction base value and the real-time backlog length of the queue to be processed to generate the computing power prediction value.

[0082] The functions or operation steps implemented by the above modules and units are largely the same as those in the above method embodiments, and will not be repeated here.

[0083] The adaptive frame rate real-time video processing system provided in this embodiment of the invention has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the system embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0084] Example 3 This invention also proposes a computer, please refer to [link / reference]. Figure 3 The computer shown in the third embodiment of the present invention includes a memory 10, a processor 20, and a computer program 30 stored in the memory 10 and executable on the processor 20. When the processor 20 executes the computer program 30, it implements the above-described adaptive frame rate real-time video processing method.

[0085] The memory 10 includes at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10 can be an internal storage unit of a computer, such as the computer's hard disk. In other embodiments, the memory 10 can be an external storage device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Furthermore, the memory 10 can include both internal and external storage units of the computer. The memory 10 can be used not only to store application software and various types of data installed on the computer, but also to temporarily store data that has been output or will be output.

[0086] In some embodiments, the processor 20 may be an electronic control unit (ECU), a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip, used to run program code stored in the memory 10 or process data, such as executing access restriction programs.

[0087] It should be pointed out that, Figure 3 The structure shown does not constitute a limitation on the computer. In other embodiments, the computer may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0088] This invention also proposes a storage medium storing a computer program that, when executed by a processor, implements the real-time video processing method with adaptive frame rate as described above.

[0089] Those skilled in the art will understand that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0090] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0091] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0092] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A real-time video processing method with adaptive frame rate, characterized in that, include: Acquire video frame data stream, decode each frame of the video frame data stream to obtain a frame image, and extract multi-dimensional spatiotemporal domain feature parameters of the current frame, wherein the multi-dimensional spatiotemporal domain feature parameters include temporal domain features, spatial domain features and frequency domain features; Construct a frame retention utility function, and use the frame retention utility function and the spatiotemporal domain feature parameters to calculate the marginal retention utility value of the current frame. When the marginal retention utility value is lower than the dynamic utility threshold, mark the current frame as a discard frame; otherwise, mark the current frame as a core key frame or a low priority frame. Construct a multi-priority queue structure, in which each frame carries a frame type label, the frame type label including the core key frame and the low priority frame; The actual computation time of each frame in each computing unit is collected in real time. An improved parametric exponential smoothing model is used to predict computing power, and feedforward correction is performed in combination with the real-time backlog length of the queue to be computed to generate a computing power prediction value. A set of computing units is constructed, and logical operations are performed on the queue structure according to the frame type label using the set of computing units and the predicted computing power value to obtain frame processing data. The frame processing data is then sorted to obtain the corresponding ordered frames.

2. The real-time video processing method with adaptive frame rate according to claim 1, characterized in that, The steps of acquiring a video frame data stream, decoding each frame of the video frame data stream to obtain a frame image, and extracting multi-dimensional spatiotemporal feature parameters of the current frame include: The raw data stream is obtained from the video source and decoded to obtain independent frame images; The frame image is divided into multiple non-overlapping macroblocks, a reference frame is defined, and within the search range of the reference frame, the most similar matching block is found for each macroblock in the current frame. The spatial displacement between the matching block and the current macroblock is calculated to obtain the corresponding temporal features. A three-dimensional joint entropy is constructed, and the three-dimensional joint entropy is used to perform color space feature statistics on the frame image to obtain the corresponding spatial domain features; The frame image is transformed from the spatial domain to the frequency domain using the discrete cosine transform algorithm, and the proportion of high-frequency energy is calculated to obtain the corresponding frequency domain features.

3. The real-time video processing method with adaptive frame rate according to claim 1, characterized in that, The formula for calculating the frame retention utility function is as follows: In the formula, This indicates the number of frames currently backed up in the queue to be processed. This represents the ideal backlog in the queue to be processed. This indicates the average time required to complete the processing of one frame. This represents the computation time per frame under no-load conditions. Indicates the penalty curvature coefficient; The formula for calculating the marginal retention utility value is as follows: In the formula, Indicates the current frame The three-dimensional joint entropy, This represents the theoretical maximum value of the three-dimensional joint entropy. Indicates the current frame The time-domain characteristics, This represents the normalized reference maximum value of the time-domain features. The weighting coefficients representing the returns of the three-dimensional joint entropy. The return weight coefficients representing time-domain characteristics, This represents the penalty weighting coefficient; The formula for calculating the dynamic utility threshold is: In the formula, This represents the preset basic utility threshold. Indicates the load sensitivity coefficient. This represents the total number of processing units used to process video frame data streams. This represents the random disturbance coefficient. Indicates recent The standard deviation of the marginal retention utility value of a frame.

4. The real-time video processing method with adaptive frame rate according to claim 1, characterized in that, The steps to construct a multi-priority queue structure include: The upper and lower backlog thresholds of the operation queue are calculated respectively, and the corresponding multi-priority strategy is constructed by combining the upper and lower backlog thresholds with the real-time backlog length of the queue to be operated. Verification rules are generated based on the multi-priority strategy and the frame type label, and a multi-priority queue structure is obtained using the verification rules.

5. The real-time video processing method with adaptive frame rate according to claim 1, characterized in that, The steps for generating the predicted computing power value include: real-time acquisition of the actual computation time of each frame in each computing unit, using an improved parametric exponential smoothing model for computing power prediction, and combining the real-time backlog length of the queue to be computed for feedforward correction. Regarding the first Frame, calculate its position in the whole The original maximum time consumed in each processing unit: In the formula, Indicates the first Frame in The actual computation time in each arithmetic unit; If the current frame If the frame type label is a low-priority frame, then its correction factor is calculated. : In the formula, Indicates recent The weighted average of the maximum processing time of all core keyframes in each processing unit. Indicates recent The weighted average of the maximum processing time of all low-priority frames in each processing unit. Indicates the strengthening coefficient. This is an indicator function that takes the value 1 when the condition is true and 0 otherwise. Indicates recent The arithmetic mean of the original maximum time taken for each frame. Indicates recent Standard deviation of the original maximum frame time; Generate the corresponding effective update time based on the frame type label. : In the formula, Indicates the current frame Frame type label; Get prediction step size For the first Frames, using their effective update time As observed values, the corresponding horizontal components, trend components, and periodic components are calculated. The prediction step size is calculated based on the horizontal component, the trend component, and the periodic component. The forward prediction base value is obtained, and feedforward correction is performed based on the forward prediction base value and the real-time backlog length of the queue to be processed to generate the computing power prediction value.

6. A real-time video processing system with adaptive frame rate, characterized in that, include: The data acquisition module is used to acquire video frame data stream, decode each frame of the video frame data stream to obtain a frame image, and extract multi-dimensional spatiotemporal domain feature parameters of the current frame, wherein the multi-dimensional spatiotemporal domain feature parameters include temporal domain features, spatial domain features and frequency domain features. The data comparison module is used to construct a frame retention utility function and calculate the marginal retention utility value of the current frame using the frame retention utility function and the spatiotemporal domain feature parameters. When the marginal retention utility value is lower than the dynamic utility threshold, the current frame is marked as a discarded frame; otherwise, the current frame is marked as a core key frame or a low priority frame. The structure building module is used to build a multi-priority queue structure, in which each frame carries a frame type label, the frame type label including the core key frame and the low priority frame; The computing power prediction module is used to collect the actual computing time of each frame in each computing unit in real time, and uses an improved parametric exponential smoothing model to predict computing power. It also combines the real-time backlog length of the queue to be computed for feedforward correction to generate the computing power prediction value. The data processing module is used to construct a set of computing units, and to perform logical operations on the queue structure according to the frame type label using the set of computing units and the computing power prediction value to obtain frame processing data, and to sort the frame processing data to obtain the corresponding ordered frames.

7. The real-time video processing system with adaptive frame rate according to claim 6, characterized in that, The data acquisition module is specifically used for: The raw data stream is obtained from the video source and decoded to obtain independent frame images; The frame image is divided into multiple non-overlapping macroblocks, a reference frame is defined, and within the search range of the reference frame, the most similar matching block is found for each macroblock in the current frame. The spatial displacement between the matching block and the current macroblock is calculated to obtain the corresponding temporal features. A three-dimensional joint entropy is constructed, and the three-dimensional joint entropy is used to perform color space feature statistics on the frame image to obtain the corresponding spatial domain features; The frame image is transformed from the spatial domain to the frequency domain using the discrete cosine transform algorithm, and the proportion of high-frequency energy is calculated to obtain the corresponding frequency domain features.

8. The real-time video processing system with adaptive frame rate according to claim 6, characterized in that, The structure building module is specifically used for: The upper and lower backlog thresholds of the operation queue are calculated respectively, and the corresponding multi-priority strategy is constructed by combining the upper and lower backlog thresholds with the real-time backlog length of the queue to be operated. Verification rules are generated based on the multi-priority strategy and the frame type label, and a multi-priority queue structure is obtained using the verification rules.

9. A readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the real-time video processing method with adaptive frame rate as described in any one of claims 1 to 5.

10. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the real-time video processing method with adaptive frame rate as described in any one of claims 1 to 5.

Images