Video adaptive following electronic amplification method, system and device based on macro block coding characteristics and medium

Abstract

The invention discloses a video self-adaptive following electronic amplification method, system and device based on macro block coding characteristics and a medium. The method comprises the following steps: analyzing macro block division information of each frame in a video decoding process, constructing a macro block division graph, and establishing a mapping function from macro block size to detail weight; dividing a video frame into grid areas, calculating detail density values of grid units based on detail weights, and generating a macro block density thermodynamic diagram; determining a visual focus coordinate in the video frame based on the macro block density thermodynamic diagram; performing dynamic amplification according to the visual focus coordinates and a self-adaptive amplification strategy, and calculating an amplification proportion and an electronic amplification window range; and evaluating whether a magnification exit condition is satisfied, and if the magnification exit condition is satisfied, exiting the magnification state. According to the invention, the problems of tedious manual operation, large calculation overhead of an AI scheme, poor real-time performance, high privacy risk, poor compatibility and the like in the existing video amplification technology are solved.

Description

Technical Field

[0001] This invention relates to the field of digital video processing technology, specifically to a video adaptive following electronic amplification method, system, device, and medium based on macroblock coding features. Background Technology

[0002] With the popularization of video technology, the rapid development of video streaming, and the widespread application of ultra-high-definition video, users have placed higher demands on the video viewing experience. In practical applications, viewers often need to focus on specific details in the video, such as moving targets in surveillance footage. Traditional players' electronic zoom functions rely on manual user operation, which presents the following problems: (1) The operation is cumbersome, requiring users to operate frequently and manually adjust the magnified area and ratio, which affects the viewing experience.

[0003] (2) It can only zoom in on a fixed area, making it difficult to continuously track the area of ​​interest in dynamic scenes and easily causing the loss of important details. Existing technologies include AI-based intelligent magnification solutions, typically employing deep learning models (such as CNN and YOLO) for object detection and region recognition. However, these methods have the following limitations: (1) It is easy to miss the key points: In dynamic videos, important content (such as moving targets, faces, and text) appears quickly. Relying on the efficiency of AI recognition, it is difficult to locate and magnify in time.

[0004] (2) High computational overhead: Deep learning models require high-performance GPUs and are difficult to run smoothly on ordinary devices or browsers.

[0005] (3) Strong scene dependence: Model training requires a large amount of labeled data, and the generalization ability is limited by the training dataset.

[0006] Video coding technologies (such as H.264 / AVC, H.265 / HEVC, and AV1) employ a block-based hierarchical coding structure, adapting to local image features through macroblock partitioning. During compression, the macroblock size is adaptively adjusted based on the local complexity of the image. The encoder dynamically selects the macroblock size according to the region's complexity; for example, small macroblocks (4×4, 8×8) are used in areas rich in detail, such as edges, textures, and moving objects, while large macroblocks (16×16, 32×32) are used in flat areas. This macroblock partitioning pattern naturally forms a spatial distribution map of image information density, and the size and distribution of these macroblocks naturally reflect the image's "information density."

[0007] Although modern video coding standards widely adopt this adaptive macroblock partitioning technique, existing playback systems have not yet effectively utilized this rich coding layer information to enhance the viewing experience. Summary of the Invention

[0008] The purpose of this invention is to provide a video adaptive following electronic amplification method, system, device and medium based on macroblock coding features, so as to solve the problems of cumbersome manual operation, large computational overhead of AI solutions, poor real-time performance, high privacy risks and poor compatibility in existing video amplification technologies.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: A video adaptive following electronic amplification method based on macroblock coding features includes the following steps: S1: During the video decoding process, the macroblock partitioning information of each frame is analyzed in real time, a macroblock partitioning map is constructed, the spatial coordinates, size and type of macroblocks are recorded, and a mapping function from macroblock size to detail weight is established. S2: Divide the video frame into grid regions, calculate the detail density value of each grid unit based on the detail weight, and perform Gaussian filtering to generate a macroblock density heatmap. S3: Based on macroblock density heatmap, visual focus coordinates in video frames are determined through candidate region selection, region scoring, focus coordinate calculation and temporal smoothing; S4: Based on the visual focus coordinates and the adaptive magnification strategy, dynamically magnify the visual focus and calculate the magnification ratio and the range of the electronic magnification window. S5: Evaluate whether the amplification exit condition is met. If the amplification exit condition is met, exit the amplification state.

[0010] To optimize the above technical solution, the specific limitations also include: In step S1, the spatial coordinates of the macroblock are precise coordinates in pixels, and the macroblock types include intra-frame coded macroblocks and inter-frame predictive macroblocks.

[0011] Further, in step S2, the formula for calculating the detail density value is:

[0012] in, Here are the coordinates of the current grid cell, and K is the number of macroblocks within the current grid cell. For the first k The size of a macroblock For the first k The detail weights corresponding to each macroblock size For the first k The area percentage of each macroblock in the current grid cell The area of ​​the grid cell; The Gaussian filtering process uses a 3×3 Gaussian kernel for smoothing, and the formula for calculating the detail density value after filtering is as follows:

[0013] Where G is the Gaussian kernel. ; m This represents the row direction offset within the neighborhood of the current grid cell. n This represents the column offset within the neighborhood of the current grid cell.

[0014] Further, in step S3, the candidate region is selected as a connected region in the heatmap with a detail density higher than the dynamic threshold; the region score is defined as:

[0015] in, The average detail density of the region. For the region with the highest detail density, To achieve the highest detail density across the entire frame, For the area, For the full frame area, The percentage of moving macroblocks within the region. The maximum percentage of motion across the entire frame; Preferably, the focal coordinates are calculated by weighted center coordinates for regions with a score higher than 60.

[0016] in, , For the weighted center coordinates of the region, P D( represents the total number of pixels in the region) p (pixel) p Detail density value at that location , For pixels p The coordinates of the location.

[0017] Preferably, in step S4, the adaptive amplification strategy requires the following three conditions to be met simultaneously to trigger the amplification function: The maximum detail density value is greater than a set detail density threshold, wherein the maximum detail density value is the maximum value in the density heatmap; The proportion of the area of ​​the detail density region exceeding the set detail density threshold is greater than 15%; The video frames maintain a detail density greater than the set detail density threshold for three or more consecutive frames.

[0018] The formula for calculating the set detail density threshold is defined as follows:

[0019] in, The average density of the frame. is the standard deviation of frame density.

[0020] Furthermore, the amplification exit condition is considered met when any of the following conditions are satisfied: The maximum detail density value is greater than the set detail density threshold; The area of ​​detail density regions exceeding the set detail density threshold is less than 10%; The user must manually intervene to exit the zoomed-out state.

[0021] This invention also proposes a video adaptive following electronic amplification system based on macroblock coding features, comprising: The feature analysis module is used to parse the macroblock partitioning information of each frame in real time during the video decoding process, construct a macroblock partitioning map, record the spatial coordinates, size and type of macroblocks, and establish a mapping function from macroblock size to detail weights. The density heatmap generation module is used to divide video frames into grid regions, calculate the detail density value of each grid unit based on the detail weight obtained by the feature analysis module, and perform Gaussian filtering to generate a macroblock density heatmap. The localization module is used to determine the visual focus coordinates in a video frame based on the macroblock density heatmap generated by the density heatmap generation module, through candidate region selection, region scoring, focus coordinate calculation and temporal smoothing. The magnification control module is used to dynamically magnify the visual focus based on the coordinates of the visual focus obtained from the positioning module and the adaptive magnification strategy, and to calculate the magnification ratio and the range of the electronic magnification window.

[0022] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the video adaptive following electronic amplification method based on macroblock coding features as described above.

[0023] The present invention also proposes a computer-readable storage medium storing a computer program that causes a computer to execute the video adaptive following electronic amplification method based on macroblock coding features as described above.

[0024] Compared with the prior art, the beneficial effects of the present invention are: This invention provides a video adaptive tracking electronic magnification method based on macroblock coding features. By real-time analysis of the macroblock partitioning map of each frame, it achieves instantaneous response to dynamic scenes. Through a mapping function from macroblock size to weights, the coded features are transformed into detail density values, accurately reflecting the visual importance and detail richness of image regions. Secondly, a multi-stage strategy for visual focus localization is employed, integrating density, area, and motion features to achieve robust focus tracking, avoiding jitter and enhancing the user viewing experience. Furthermore, adaptive magnification control, based on dynamic thresholds and ratio calculations, ensures magnification is triggered only when detail areas are significant, and optimizes the view centered on the focus point, balancing detail enhancement and image stability. Finally, a video magnification exit mechanism automatically manages resources through intelligent conditions, improving system energy efficiency.

[0025] Furthermore, macroblock partitioning information is an additional output of the video decoding process and can be obtained without additional computation. Compared to AI technology solutions that require running deep learning models, this invention saves GPU resources by parsing existing encoded data and can run on ordinary CPUs or embedded devices. Moreover, by processing only structural information such as macroblock size and type, without parsing the semantic content of the video, it avoids the risk of privacy leaks and complies with data protection regulations.

[0026] Gaussian filtering is used to smooth the detail density values, eliminate abrupt changes caused by uneven macroblock partitioning or coding noise, improve the stability of the heatmap, and provide reliable input for focus localization.

[0027] A four-stage strategy is adopted to determine the visual focus, which includes candidate region selection, region scoring, focus coordinate calculation and temporal smoothing. It comprehensively considers density, area and motion characteristics to avoid misjudgment and achieves smooth and accurate visual focus tracking. Attached Figure Description

[0028] Figure 1 : A schematic diagram of the video adaptive following electronic amplification method based on macroblock coding features of the present invention. Detailed Implementation The present invention will be further described in detail below through specific embodiments, but it should not be construed as limiting the scope of the subject matter of the present invention to the following embodiments. All technologies implemented based on the above content of the present invention fall within the scope of the present invention.

[0029] The following is an explanation of some of the terms used in this invention: Macroblock: A basic processing unit used for image segmentation and encoding in video coding, such as H.264 / AVC and H.265 / HEVC. It is typically a 16×16 pixel block. Macroblocks in different regions may be divided in different ways, such as 16×16, 8×8, 4×4, etc., and their size and distribution reflect the complexity of the image.

[0030] Macroblock size: refers to the size of macroblocks or sub-macroblocks dynamically selected during the encoding process to adapt to the local complexity of the image. Regions with rich detail or dramatic changes tend to use smaller macroblocks (such as 4×4), while flat regions use larger macroblocks, such as 16×16.

[0031] The technical solution of the present invention will be further described in detail below with reference to specific embodiments: In one embodiment of the present invention, a video adaptive following electronic amplification method based on macroblock coding features is proposed, the flowchart of which is shown below. Figure 1 As shown, the entire method includes the following steps: S1: During the video decoding process, the macroblock partitioning information of each frame is analyzed in real time, a macroblock partitioning map is constructed, the spatial coordinates, size and type of macroblocks are recorded, and a mapping function from macroblock size to detail weight is established. S2: Divide the video frame into grid regions, calculate the detail density value of each grid unit based on detail weights and perform Gaussian filtering to generate a macroblock density heatmap. S3: Based on macroblock density heatmap, visual focus coordinates in video frames are determined through candidate region selection, region scoring, focus coordinate calculation and temporal smoothing; S4: Based on the visual focus coordinates and the adaptive magnification strategy, dynamically magnify the visual focus and calculate the magnification ratio and the range of the electronic magnification window. S5: Evaluate whether the amplification exit condition is met. If the amplification exit condition is met, exit the amplification state.

[0032] In step S1, the spatial coordinates of the macroblock are precise coordinates (x, y) in pixels, based on a 3840×2160 4K video, with x ranging from [0, 3839] and y ranging from [0, 2159].

[0033] Macroblock size is used to record the actual partition size, including but not limited to 64×64 (CTU in H.265 / HEVC), 32×32, 16×16 (basic macroblock unit), 8×8 and 4×4 (smallest partition unit).

[0034] Macroblock types include intra-frame coding and inter-frame predictive coding, where intra-frame coding includes I_16x16, I_8x8 and I_4x4; inter-frame predictive coding includes P_16x16, P_16x8, P_8x16, P_8x8, P_8x4, P_4x8 and P_4x4.

[0035] The mapping function from macroblock size to detail weights is:

[0036] Where d represents the macroblock size, and w(d) represents the detail weight value corresponding to different macroblock sizes. This weight reflects the amount of detail information contained in macroblocks of different sizes and is used to calculate macroblock density.

[0037] In step S2, the strategy for dividing the video frame into grid regions is as follows: For 1080p video (1920×1080): a 24×18 grid is used, with each grid cell containing 80×60 pixels; For 720p video (1280×720): use a 16×12 grid, with each grid cell being 80×60 pixels; For 4K video (3840×2160): a 48×36 grid is used, with each grid cell containing 80×60 pixels.

[0038] For each mesh cell (i,j), calculate its detail density value:

[0039] in, Here are the coordinates of the current grid cell, and K is the number of macroblocks within the current grid cell. For the first k The size of a macroblock For the first k The detail weights corresponding to each macroblock size For the first k The area percentage of each macroblock in the current grid cell The area of ​​the grid cell; Then, the density values ​​are smoothed using a Gaussian filter with a 3×3 Gaussian kernel:

[0040] The formula for calculating the detail density value after filtering is:

[0041] Where G is the Gaussian kernel. ; m This represents the row direction offset within the neighborhood of the current grid cell. nThis represents the column offset within the neighborhood of the current grid cell.

[0042] In step S3, a four-stage strategy is used to determine the visual focus based on the density heatmap: Candidate region selection identifies connected regions in a heatmap whose density values ​​are higher than a dynamic threshold. :

[0043] in The average density of the entire frame. This represents the maximum density across the entire frame.

[0044] Regional scoring calculates a comprehensive score for each candidate region, with a maximum score of 100.

[0045] in, The average detail density of the region. For the region with the highest detail density, To achieve the highest detail density across the entire frame, For the area, For the full frame area, The percentage of moving macroblocks within the region. This represents the maximum percentage of motion across the entire frame.

[0046] For regions with a score higher than 60, the weighted center coordinates are calculated.

[0047] in, , For the weighted center coordinates of the region, P D( represents the total number of pixels in the region) p (pixel) p Detail density value at (range 0.0-4.0), , For pixels p The coordinates of the location.

[0048] Time-domain smoothing is performed using an exponential moving average to smooth the focal coordinates over time.

[0049] The smoothing coefficient is 0.3, which strikes a balance between response speed and stability. The current smoothing result is 30% from the current actual measurement value and 70% from the previous smoothing result. This represents the current actual observation value at time t. This represents the smoothed (or predicted) value at time t-1, which is the result after smoothing from the previous time step. This represents the new smoothed value at time t.

[0050] The final output coordinates transition gradually and smoothly to the new target position along a smooth trajectory. When there is a moving object, the zoom box will smoothly follow it, instead of locking onto the object's center frame by frame, which would cause jitter.

[0051] In step S4, the adaptive amplification strategy triggers the amplification function when the following three conditions are met simultaneously: (1) Maximum density value > ;

[0052] in, The maximum value found in the density heatmap represents the small region with the richest details and densest information in this frame. To set a detail density threshold; Frame average density represents the overall average level of detail in the video content; , where is the frame density standard deviation, representing the range of fluctuations in the dynamic changes or complexity of the video content; the weight coefficients are 0.6 and 0.4, respectively, which define whether the algorithm relies more on the average basis or the dynamic changes. A higher weight indicates that the system places more trust in the overall, average level of detail. The weights are relatively small, but they provide important adaptive adjustment capabilities. When the salience of a certain region ( ), exceeding the overall average level ( ), while also taking into account the current fluctuations ( It reached a reasonable significance threshold. When it comes to this, we believe it deserves our attention.

[0053] (2) The proportion of the area of ​​the detail density region that exceeds the set detail density threshold is greater than 15%; (3) The video frame maintains a detail density state greater than the set detail density threshold for more than 3 consecutive frames.

[0054] In step S4, the magnification ratio is calculated using a piecewise function, and the calculation formula is as follows:

[0055] The maximum allowable magnification factor is set to 3.0 to avoid excessive magnification.

[0056] The electronic magnification window is controlled by calculating the magnification range centered on the visual focus:

[0057]

[0058] in, and The width and height of the original frame. and To adjust the width and height of the window; at the same time, it is necessary to ensure that the boundaries of the video window do not exceed the frame range.

[0059] In step S5, evaluate whether the amplification exit condition is met. If any of the following conditions are met, exit the amplification state: (1) The maximum detail density value is greater than the set detail density threshold; (2) The area of ​​the detail density region that meets the set detail density threshold is less than 10%; (3) The user manually intervenes to exit the magnification state.

[0060] This invention also proposes a video adaptive following electronic amplification system based on macroblock coding features, comprising: The feature analysis module is used to parse the macroblock partitioning information of each frame in real time during the video decoding process, construct a macroblock partitioning map, record the spatial coordinates, size and type of macroblocks, and establish a mapping function from macroblock size to detail weights. The density heatmap generation module is used to divide video frames into grid regions, calculate the detail density value of each grid unit based on the detail weight obtained by the feature analysis module, and perform Gaussian filtering to generate a macroblock density heatmap. The localization module is used to determine the visual focus coordinates in a video frame based on the macroblock density heatmap generated by the density heatmap generation module, through candidate region selection, region scoring, focus coordinate calculation and temporal smoothing. The magnification control module is used to dynamically magnify the visual focus based on the coordinates of the visual focus obtained from the positioning module and the adaptive magnification strategy, and to calculate the magnification ratio and the range of the electronic magnification window.

[0061] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the above-described video adaptive following electronic amplification method based on macroblock coding features.

[0062] The present invention also proposes a computer-readable storage medium storing a computer program that enables a computer to execute the above-described video adaptive following electronic amplification method based on macroblock coding features.

[0063] This invention provides a lightweight method for detecting high-detail regions. By analyzing the size and spatial distribution of macroblocks in video frames, it automatically identifies high-detail regions and dynamically adjusts the magnification center to achieve content-following playback. It innovatively uses macroblock information from the coding layer for visual saliency analysis, taking macroblock segmentation information generated during video encoding as a visual importance indicator, thus avoiding complex AI image content analysis. Furthermore, based on a macroblock density-weighted heatmap generation method, it proposes a density calculation method based on macroblock size weighting, accurately reflecting the detail richness of image regions. In addition, by combining spatial density distribution and temporal continuity, it achieves smooth and accurate visual focus tracking.

[0064] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0065] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent substitutions, and improvements made by those skilled in the art to the above embodiments without departing from the scope of the technical solution of the present invention, based on the technical essence of the present invention, shall still fall within the protection scope of the technical solution of the present invention.

Claims

1. A video adaptive following electronic amplification method based on macroblock coding features, characterized in that, Includes the following steps: S1: During the video decoding process, the macroblock partitioning information of each frame is analyzed in real time, a macroblock partitioning map is constructed, the spatial coordinates, size and type of macroblocks are recorded, and a mapping function from macroblock size to detail weight is established. S2: Divide the video frame into grid regions, calculate the detail density value of each grid unit based on the detail weight, and perform Gaussian filtering to generate a macroblock density heatmap. S3: Based on macroblock density heatmap, visual focus coordinates in video frames are determined through candidate region selection, region scoring, focus coordinate calculation and temporal smoothing; S4: Based on the visual focus coordinates and the adaptive magnification strategy, dynamically magnify the visual focus and calculate the magnification ratio and the range of the electronic magnification window. S5: Evaluate whether the amplification exit condition is met. If the amplification exit condition is met, exit the amplification state.

2. The video adaptive following electronic amplification method based on macroblock coding features according to claim 1, characterized in that: In step S1, the spatial coordinates of the macroblock are precise coordinates in pixels, and the macroblock types include intra-frame coded macroblocks and inter-frame predictive macroblocks.

3. The video adaptive following electronic amplification method based on macroblock coding features according to claim 1, characterized in that: In step S2, the formula for calculating the detail density value is: in, Here are the coordinates of the current grid cell, and K is the number of macroblocks within the current grid cell. For the first k The size of a macroblock For the first k The detail weights corresponding to each macroblock size For the first k The area percentage of each macroblock in the current grid cell The area of ​​the grid cell; The Gaussian filtering process uses a 3×3 Gaussian kernel for smoothing, and the formula for calculating the detail density value after filtering is as follows: Where G is the Gaussian kernel. ; m This represents the row direction offset within the neighborhood of the current grid cell. n This represents the column offset within the neighborhood of the current grid cell.

4. The video adaptive following electronic amplification method based on macroblock coding features according to claim 1, characterized in that: In step S3, the candidate region is selected as a connected region in the heatmap with a detail density higher than the dynamic threshold; the region score is defined as: in, The average detail density of the region. For the region with the highest detail density, To achieve the highest detail density across the entire frame, For the area, For the full frame area, The percentage of moving macroblocks within the region. The maximum percentage of motion across the entire frame; The focal coordinate calculation is performed by calculating the weighted center coordinates for regions with a score higher than 60. in, , For the weighted center coordinates of the region, P D( represents the total number of pixels in the region) p (pixel) p Detail density value at that location , For pixels p The coordinates of the location.

5. The video adaptive following electronic amplification method based on macroblock coding features according to claim 1, characterized in that: In step S4, the adaptive amplification strategy requires the following three conditions to be met simultaneously to trigger the amplification function: The maximum detail density value is greater than a set detail density threshold, wherein the maximum detail density value is the maximum value in the density heatmap; The proportion of the area of ​​the detail density region exceeding the set detail density threshold is greater than 15%; The video frames maintain a detail density greater than the set detail density threshold for three or more consecutive frames.

6. The video adaptive following electronic amplification method based on macroblock coding features according to claim 1, characterized in that: The formula for calculating the set detail density threshold is defined as follows: in, The average density of the frame. is the standard deviation of frame density.

7. The video adaptive following electronic amplification method based on macroblock coding features according to claim 1, characterized in that: The amplification exit condition is considered met when any of the following conditions are satisfied: The maximum detail density value is greater than the set detail density threshold; The area of ​​detail density regions exceeding the set detail density threshold is less than 10%; The user must manually intervene to exit the zoomed-out state.

8. A video adaptive following electronic amplification system based on macroblock coding features, characterized in that, include: The feature analysis module is used to parse the macroblock partitioning information of each frame in real time during the video decoding process, construct a macroblock partitioning map, record the spatial coordinates, size and type of macroblocks, and establish a mapping function from macroblock size to detail weights. The density heatmap generation module is used to divide video frames into grid regions, calculate the detail density value of each grid unit based on the detail weight obtained by the feature analysis module, and perform Gaussian filtering to generate a macroblock density heatmap. The localization module is used to determine the visual focus coordinates in a video frame based on the macroblock density heatmap generated by the density heatmap generation module, through candidate region selection, region scoring, focus coordinate calculation and temporal smoothing. The magnification control module is used to dynamically magnify the visual focus based on the coordinates of the visual focus obtained from the positioning module and the adaptive magnification strategy, and to calculate the magnification ratio and the range of the electronic magnification window.

9. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the video adaptive following electronic amplification method based on macroblock coding features as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program causes the computer to execute the video adaptive following electronic amplification method based on macroblock coding features as described in any one of claims 1-7.

Images