Video anti-shake processing method, medium and device

Abstract

The present invention relates to the field of video processing technology, and specifically discloses a video anti-shake processing method, medium, and device. The method comprises: extracting feature points by downsampling a first video frame, calculating the homography transformation matrix between adjacent frames, and generating a global transformation matrix to achieve frame alignment; combining adaptive detail grid division and uniform grid division to generate a comprehensive grid block set covering key areas and the global area, and marking the type of each grid block; performing pixel-level fusion based on distance weight and type weight coefficient through a weighted fusion and recombination method, and outputting a stable video frame. The method combines global motion compensation with local grid optimization, and can process video frame images in violent shaking scenes, restore the picture content, obtain stable video frame images, avoid missing monitoring images, and ensure the safety of port operations.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a video anti-shake processing method, medium and device. Background Art

[0002] In the field of port container transportation and logistics operations, cameras installed on container trolleys can monitor and record the entire process of container loading, unloading, stacking, and transshipment in real time, which is of great significance for ensuring operational safety and improving operational efficiency. However, compared with ordinary monitoring scenarios, port container monitoring scenarios have significant special features:

[0003] First, scenes are fixed and periodically repetitive. Most surveillance cameras in port container yards are fixed installations, monitoring the same fixed area over long periods of time. The images are highly repetitive and predictable. This characteristic enables the system to memorize and learn from historical scenes and achieve scene prediction.

[0004] Second, there are unique shaking patterns. During port operations, heavy machinery such as cranes and gantry cranes can cause sudden, large-scale shaking, typically lasting less than a second. Unlike the more common, continuous, mild vibrations, this type of shaking is instantaneous and intense, causing the surveillance footage to become severely blurred for a moment, significantly impacting monitoring effectiveness.

[0005] Third, real-time monitoring requirements are stringent. Given the stringent safety and efficiency requirements of port container operations, monitoring systems must process video streams in real time. Traditional offline processing methods have high latency solutions and cannot meet the actual needs of port operations.

[0006] like Figure 1 As shown, the blue part shows the shaking pattern targeted by the usual anti-shake algorithm, while the red part shows the sudden and large-scale shaking in the port container operation scenario that this application focuses on. There are significant differences between the two in shaking characteristics and scene performance, which makes the traditional video anti-shake algorithm unable to be applied to the processing of video images in the port container monitoring scenario, which easily leads to the loss of monitoring images and leaves safety hazards. Summary of the Invention

[0007] In view of the above problems, the present invention provides a video stabilization processing method, medium and device to solve the technical problems that the existing video stabilization algorithm is not applicable to the stabilization processing of video images with severe shaking, cannot meet the application of port container monitoring scenarios, and is prone to missing monitoring images, affecting safe operations.

[0008] To achieve the above objectives, in a first aspect, the present application provides a video stabilization processing method, the method comprising the following steps:

[0009] S1: Receive a first video frame, downsample the first video frame, extract feature points from the downsampled frame, calculate a homography transformation matrix between adjacent frames based on the feature points, accumulate and generate a global transformation matrix relative to a reference frame, and perform an alignment operation on the first video frame according to the global transformation matrix to obtain a second video frame;

[0010] S2: using an adaptive detail grid division method and generating a first grid block set based on a corner detection algorithm, where the first grid block set covers a key area on the second video frame;

[0011] S3: Generate a second set of grid blocks with global coverage using a uniform grid division method;

[0012] S4: Taking the union of the first grid block set and the second grid block set to generate a comprehensive grid block set, adding type tags to all grid blocks in the comprehensive grid block set, and outputting a grid block information list corresponding to all grid blocks in the comprehensive grid block set;

[0013] S5: According to the grid block information list, a weighted fusion and recombination method is applied to perform pixel-level fusion based on a distance weight function and a type tag weight coefficient, and a stable video frame is output.

[0014] Furthermore, step S1 includes:

[0015] The first video frame Downsample to , downsampling ratio , ;

[0016] Get downsampled consecutive frames and , extract feature points and establish correspondence between pixels ;

[0017] Use the RANSAC algorithm to estimate the homography transformation matrix , so that the pixels of consecutive frames after downsampling satisfy: ,in, represents the geometric transformation from frame t to frame t+1;

[0018] Select a reference frame , for the downsampled frame t, calculate its cumulative transformation matrix relative to the reference frame , the calculation formula is as follows: ;

[0019] The accumulated transformation matrix at the downsampled resolution Transformation matrix converted to original resolution , for the first video frame at original resolution , apply the transformation matrix Align and get the second video frame , the calculation formula is as follows: .

[0020] Furthermore, the image size of the second video frame is recorded as , the grid block size is , the allowed overlap size between grid blocks is A, then the minimum distance between feature points is calculated , step S2 includes:

[0021] Converting the second video frame into a grayscale image;

[0022] Use Shi-Tomasi corner detection algorithm to detect grayscale images Medium detection feature points, and the feature point set is expressed as , where each point , any two points and The Euclidean distance between ;

[0023] For each feature point , generate a Central The grid blocks are of size, and the grid blocks cover an area of: ;

[0024] Only the grid blocks that are completely within the grayscale image boundary are retained, and the center point of the grid block The following conditions are met: and , output the coordinate list of all valid grid block center points , and obtain the first grid block set.

[0025] Furthermore, step S3 includes:

[0026] Place the second video frame in a two-dimensional space coordinate system and calculate the center of the X-axis grid block: until , and calculate the Y-axis grid block center: until , ensuring that: and , so that the edge grid blocks can completely cover the edge of the second video frame, and obtain the second grid block set: ;

[0027] For each pixel in the image , calculate the number of grid blocks covering the pixel, create a list for each pixel, record all grid block indices covering the pixel, and obtain the pixel to grid block coverage mapping map;

[0028] The second set of grid blocks and the pixel-to-grid block coverage mapping map are output.

[0029] Furthermore, the size A of the overlapping area of ​​adjacent grid blocks is dynamically adjusted according to the intensity of the video frame shake. The adjustment formula is as follows:

[0030] ;

[0031] in, is the base overlap size; is the sloshing intensity factor, is the adjustment coefficient.

[0032] Furthermore, step S4 includes:

[0033] Receive the center coordinate list of the first grid block set output in step S2 And the center coordinate list of the second grid block set output in step S3 ,right and Taking the union to obtain a comprehensive grid block set, and adding type tags to all grid blocks in the comprehensive grid block set, wherein the type tags of the grid blocks originally belonging to the first grid block set are adaptive grid type tags, and the type tags of the grid blocks originally belonging to the second grid block set and not belonging to the first grid block set are uniform grid type tags;

[0034] Outputting a merged grid block information list, the grid block information list including the center coordinates of the grid block, at least two edge coordinates of the grid block, and a type tag corresponding to the grid block.

[0035] Furthermore, step S5 includes:

[0036] Each pixel position , calculate according to the following formula and output a stable video frame:

[0037] ;

[0038] in, Is the position on the stable video frame The pixel value of It is grid blocks at position The pixel value of is the weight function value based on distance, is the weight adjustment coefficient of the grid block type, Is the coverage location The total number of grid blocks;

[0039] in, , d is the pixel The Euclidean distance to the center of the grid block, and K is the half-width of the grid block overlap area A.

[0040] Furthermore, the weight adjustment coefficient of the grid block marked with the uniform grid type is , the weight adjustment coefficient of the grid block marked by the adaptive grid type .

[0041] In a second aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video stabilization processing method as described in the first aspect of the present application.

[0042] In a third aspect, the present application provides an electronic device on which a computer program is stored, including a processor and a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by the processor, it implements the video stabilization processing method as described in the first aspect of the present application.

[0043] Different from the existing technology, the above technical solution provides a video stabilization processing method, medium and equipment. The method includes: extracting feature points by downsampling the first video frame, calculating the homography transformation matrix between adjacent frames, and generating a global transformation matrix to achieve frame alignment; combining adaptive detail grid division and uniform grid division to generate a comprehensive grid block set covering key areas and the global area, and marking the type of each grid block; using a weighted fusion and recombination method, pixel-level fusion is performed based on distance weight and type weight coefficient to output a stable video frame. By combining global motion compensation with local grid optimization, this method can process video frame images in violent shaking scenes, restore the picture content, obtain stable video frame images, avoid the loss of monitoring images, and ensure the safety of port operations.

[0044] The above-mentioned description of the invention content is only an overview of the technical solution of the present invention. In order to enable ordinary technicians in this field to more clearly understand the technical solution of the present invention, and then implement it according to the text of the specification and the contents recorded in the drawings, and to make the above-mentioned objects and other objects, features and advantages of the present invention easier to understand, the following is an explanation in combination with the specific implementation methods and drawings of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are only used to illustrate the principles, implementations, applications, features, and effects of the specific embodiments of the present invention and other related contents, and are not to be considered as limiting the present invention.

[0046] In the drawings of the specification:

[0047] Figure 1 Schematic diagram comparing the shaking patterns corresponding to a common anti-shake algorithm and the anti-shake algorithm involved in this application;

[0048] Figure 2 is a first flow chart of a video stabilization processing method according to a specific embodiment;

[0049] Figure 3 is a second flow chart of the video stabilization processing method involved in a specific embodiment;

[0050] Figure 4 Schematic diagram of the distribution of feature points detected by the corner detection algorithm involved in the specific implementation method;

[0051] Figure 5 is a schematic diagram of the distribution of grid blocks in a first grid block set involved in a specific embodiment;

[0052] Figure 6 This is a flowchart of the specific implementation steps of step S2 involved in the specific implementation method;

[0053] Figure 7 A schematic diagram of the distribution of central coordinate points of grid blocks involved in a specific implementation manner;

[0054] Figure 8 is a schematic diagram of the distribution of grid blocks in the second grid block set involved in a specific embodiment;

[0055] Figure 9 This is a flowchart of the specific implementation steps of step S3 involved in the specific implementation method;

[0056] Figure 10 This is a flowchart of the specific implementation steps of step S4 involved in the specific implementation method;

[0057] Figure 11 Schematic diagram of the distribution of the center points of the uniform grid blocks and the adaptive grid blocks involved in the specific implementation manner after fusion;

[0058] Figure 12 A schematic diagram of the distribution of grid blocks in a comprehensive grid block set involved in a specific embodiment;

[0059] Figure 13 This is a flowchart of specific implementation steps of step S5 involved in a specific implementation method;

[0060] Figure 14A schematic diagram of a module of an electronic device according to a specific embodiment;

[0061] The reference numerals in the above drawings are described as follows:

[0062] 10. Electronic equipment;

[0063] 101. Processor;

[0064] 102. Storage medium. DETAILED DESCRIPTION

[0065] In order to explain in detail the possible application scenarios, technical principles, specific solutions that can be implemented, and the purpose and effects of the present invention, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present invention and are therefore only examples and are not intended to limit the scope of protection of the present invention.

[0066] References to "embodiments" herein mean that the specific features, structures, or characteristics described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the term "embodiment" in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or relevance to other embodiments. In principle, in the present invention, as long as there are no technical contradictions or conflicts, the various technical features mentioned in the embodiments can be combined in any manner to form a corresponding implementable technical solution.

[0067] Unless otherwise defined, the technical terms used herein have the same meanings as those generally understood by those skilled in the art to which the present invention belongs. The use of relevant terms herein is only for describing specific embodiments and is not intended to limit the present invention.

[0068] In the description of the present invention, the term "and / or" is used to describe a logical relationship between objects, indicating that three possible relationships exist. For example, A and / or B means: A exists, B exists, and both A and B exist. Furthermore, the character " / " generally indicates that the objects are in a logical "or" relationship.

[0069] In the present invention, terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship of quantity, priority or sequence between these entities or operations.

[0070] Without further restrictions, in the present invention, the words "include", "comprise", "have" or other similar expressions used in sentences are intended to cover non-exclusive inclusion. These expressions do not exclude the presence of additional elements in the process, method or product including the elements, so that the process, method or product including a series of elements may include not only those limited elements, but also other elements not explicitly listed, or also include elements inherent to such process, method or product.

[0071] In the present invention, expressions such as "greater than," "less than," and "exceed" are understood to exclude the number itself; expressions such as "above," "below," and "within" are understood to include the number itself. Furthermore, in the description of the embodiments of the present invention, "multiple" means two or more (including two), and similar expressions related to "multiple," such as "multiple groups" and "multiple times," are also understood in this manner, unless otherwise specifically defined.

[0072] In the description of the embodiments of the present invention, the space-related expressions used, such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "vertical", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicate the orientation or position relationship based on the orientation or position relationship shown in the specific embodiments or drawings, and are only for the convenience of describing the specific embodiments of the present invention or facilitating the reader's understanding, and do not indicate or imply that the device or component referred to must have a specific position, a specific orientation, or be constructed or operated in a specific orientation, and therefore cannot be understood as limiting the embodiments of the present invention.

[0073] Unless otherwise expressly specified or limited, in the description of the embodiments of the present invention, the terms "installed", "connected", "connected", "fixed", "set", etc. used should be understood in a broad sense. For example, the "connection" can be a fixed connection, a detachable connection, or an integrated setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection between two elements or the interaction relationship between two elements. For those skilled in the art of the technology to which the present invention belongs, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific circumstances.

[0074] like Figure 2 As shown, in a first aspect, the present application provides a video stabilization processing method, comprising the following steps:

[0075] S1: Receive a first video frame, downsample the first video frame, extract feature points from the downsampled frame, calculate a homography transformation matrix between adjacent frames based on the feature points, accumulate and generate a global transformation matrix relative to a reference frame, and perform an alignment operation on the first video frame according to the global transformation matrix to obtain a second video frame;

[0076] S2: using an adaptive detail grid division method and generating a first grid block set based on a corner detection algorithm, where the first grid block set covers a key area on the second video frame;

[0077] S3: Generate a second set of grid blocks with global coverage using a uniform grid division method;

[0078] S4: Taking the union of the first grid block set and the second grid block set to generate a comprehensive grid block set, adding type tags to all grid blocks in the comprehensive grid block set, and outputting a grid block information list corresponding to all grid blocks in the comprehensive grid block set;

[0079] S5: According to the grid block information list, a weighted fusion and recombination method is applied to perform pixel-level fusion based on a distance weight function and a type tag weight coefficient, and a stable video frame is output.

[0080] The above solution combines global motion compensation with local grid optimization to process video frame images in violent shaking scenes, restore the picture content, obtain stable video frame images, avoid missing monitoring images, and ensure the safety of port operations.

[0081] like Figure 3-Figure 13 As shown, in some embodiments, step S1 includes:

[0082] The first video frame Downsample to , downsampling ratio , ;

[0083] Get downsampled consecutive frames and , extract feature points and establish correspondence between pixels ;

[0084] Use the RANSAC algorithm to estimate the homography transformation matrix , so that the pixels of consecutive frames after downsampling satisfy: ,in, represents the geometric transformation from frame t to frame t+1;

[0085] Select a reference frame , for the downsampled frame t, calculate its cumulative transformation matrix relative to the reference frame , the calculation formula is as follows: ;

[0086] The accumulated transformation matrix at the downsampled resolution Transformation matrix converted to original resolution , for the first video frame at original resolution , apply the transformation matrix Align and get the second video frame , the calculation formula is as follows: .

[0087] The above steps can be achieved through a fast de-shake preprocessing module, which is specifically designed to address the problem of instantaneous and severe shake in port container surveillance videos (usually lasting less than 1 second). This module uses image downsampling and transformation matrix estimation techniques to achieve efficient preliminary global alignment, laying the foundation for subsequent fine-grid processing. Its key features include: using a downsampling strategy to significantly reduce computational complexity; accurately estimating the geometric transformation relationship between consecutive frames; accumulating transformation matrices to represent the total displacement relative to the stable state; and applying a scaled transformation matrix to align the original resolution frames. Unlike traditional video stabilization algorithms, this method does not require complex matrix path smoothing processing because the shake that occurs in port containers during surveillance is short-lived, and the system will naturally return to a stable state within a short period of time.

[0088] The size of the original port monitoring video frame (i.e. the first video frame) is (e.g. 1920×1080) as an example, the downsampled video frame size can be set to , that is, keeping the aspect ratio of the original video frame unchanged, and reducing the overall length and width of the original video frame.

[0089] The RANSAC algorithm estimates the model parameters by randomly sampling a subset of data and optimizing the model using inliers (data points that conform to the model) and outliers (data points that do not conform to the model), thereby accurately estimating the homography transformation matrix in the presence of noise and outliers. In this embodiment, Preferably matrix.

[0090] In this embodiment, the reference frame For the stable frame before shaking, for the frame recovered after shaking, the cumulative matrix will naturally approach the unit transformation matrix, indicating that the system has returned to a stable state. Transformation matrix converted to original resolution This can be achieved by following these steps:

[0091] For the translation component of the perspective transformation matrix, perform the following transformation, leaving the other matrix elements unchanged:

[0092] ;

[0093] .

[0094] In this embodiment, when applying the transformation matrix After alignment, bilinear or bicubic interpolation can be used to ensure the quality of the transformed image.

[0095] After the above steps, an aligned original resolution frame sequence can be generated while maintaining the original image details. These aligned frames (i.e., the second video frames) can be used as input for the subsequent grid block division and processing module.

[0096] The rapid de-shaking pre-processing module is used to process the instantaneous and violent shaking images in the port container surveillance video, which has the following advantages:

[0097] (1) Computational efficiency: The downsampling strategy can reduce the computational complexity by about times; for 1080P input, the compression amount is times; significantly reducing the computational burden of feature extraction, matching, and matrix estimation.

[0098] (2) Rapid response capability: It can adapt to the characteristics of short-term violent shaking (usually time < 1 second) in the port environment; no complex path smoothing algorithm is required, which simplifies the calculation process; matrix accumulation naturally captures the total displacement relative to the stable state.

[0099] (3) Guarantee of spatiotemporal coherence: Global alignment ensures that the subsequently divided grid blocks remain coherent in the temporal dimension, so that the grid blocks can contain meaningful spatiotemporal information of the previous and next frames.

[0100] (4) Resolution adaptability: The system can handle various port monitoring resolutions from 720P to 4K; the downsampling ratio can be flexibly adjusted according to actual computing resources and real-time requirements.

[0101] In some embodiments, as Figure 4-Figure 6 As shown, the image size of the second video frame is recorded as , the grid block size is , the allowed overlap size between grid blocks is A, then the minimum distance between feature points is calculated , step S2 includes:

[0102] Converting the second video frame into a grayscale image;

[0103] Use Shi-Tomasi corner detection algorithm to detect grayscale images Medium detection feature points, and the feature point set is expressed as , where each point , any two points and The Euclidean distance between ,This can ensure that the distribution of feature points in the port environment is reasonable;

[0104] For each feature point , generate a Central The grid blocks are of size, and the grid blocks cover an area of: ;

[0105] Only the grid blocks that are completely within the grayscale image boundary are retained, and the center point of the grid block The following conditions are met: and , output the coordinate list of all valid grid block center points , and obtain the first grid block set.

[0106] Adaptive detail meshing is particularly suitable for key areas of the de-shaking algorithm for port container surveillance videos (such as container identification, operating equipment, etc.). This method is based on a feature point detection algorithm to identify points with significant features in the video. For example, the maximum number of feature points can be set to 200, the quality threshold can be set to 0.01 (the lower the value, the more feature points are detected), and the minimum distance between feature points can be set to 0. , the distribution of feature points is obtained as Figure 4 As shown, fixed-size grid blocks are generated with these feature points as the center to achieve accurate coverage of the key areas of the port scene. The generated grid blocks are distributed as follows: Figure 5 shown.

[0107] When performing boundary checking on the port container surveillance video image, only the grid blocks that are completely within the boundary of the port surveillance image are retained. Specifically, for each detected feature point , calculate the boundary coordinates of the grid block, that is, the coordinates of the upper left corner and the lower right corner coordinates , compare the two boundary coordinates of the grid block with the boundary coordinates of the video image to determine whether the current grid block is completely within the boundary of the port container monitoring video image. If it is within the boundary, Add to the grid block center coordinate list, and then return a list containing the coordinates of all valid grid block center points The entire implementation process of step S2 is referenced Figure 6 shown.

[0108] In some embodiments, step S3 includes:

[0109] Place the second video frame in a two-dimensional space coordinate system and calculate the center of the X-axis grid block: until , and calculate the Y-axis grid block center: until , ensuring that: and , so that the edge grid blocks can completely cover the edge of the second video frame, and obtain the second grid block set: ;

[0110] For each pixel in the image , calculate the number of grid blocks covering the pixel, create a list for each pixel, record all grid block indices covering the pixel, and obtain the pixel to grid block coverage mapping map;

[0111] The second set of grid blocks and the pixel-to-grid block coverage mapping map are output.

[0112] In this example, uniform gridding is a systematic image processing technique used to ensure complete coverage of port surveillance video. Unlike adaptive segmentation, uniform gridding emphasizes global balance, ensuring that every pixel in the surveillance image is covered by an appropriate number of grid blocks. This makes it particularly suitable for processing large areas and backgrounds in port container operations.

[0113] In this embodiment, the grid block size parameter It can be set to 64 or 128 pixels. The distribution of the midpoints of the grid blocks obtained by the above method is as follows Figure 7 The corresponding grid distribution diagram is shown as Figure 8 As shown. By processing the boundaries, it is possible to ensure that the grid blocks at the edge of the grid can completely cover the boundary of the port container monitoring video image. If the edge of the last grid block does not reach the image boundary, additional grid block centers are added to ensure complete coverage. The entire implementation process of step S2 is referenced Figure 9 shown.

[0114] In some embodiments, the size A of the overlapping area of ​​adjacent grid blocks is dynamically adjusted according to the intensity of the video frame jitter, and the adjustment formula is as follows:

[0115] ;

[0116] in, is the base overlap size; is the sloshing intensity factor, is the adjustment coefficient.

[0117] When performing anti-shake processing on port container surveillance video images, parameter selection has a significant impact on system performance. The following are the design principles for key parameters:

[0118] (1) Adaptive grid block size: The basic size is Pixels (preferably or or ), the specific size of S needs to be adapted to the corresponding de-shaking algorithm.

[0119] (2) Intelligent overlapping mechanism: The size of the overlapping area of ​​adjacent grid blocks A (preferably the basic value A = 16 or 32) can be dynamically adjusted according to the shaking intensity. The shaking intensity factor M is preferably in the range of [0, 1], and the adjustment coefficient Preferably it is 0.5.

[0120] Taking different shaking conditions commonly seen in port container monitoring as an example, when setting Pixels, The specific adjustment calculation method of the grid block overlapping area size A is as follows:

[0121] Slight shaking ( ) : Pixels;

[0122] Medium shake ( ) : Pixels;

[0123] Severe shaking ( ) : Pixels;

[0124] Shake intensity By analyzing the size of the optical flow field of adjacent frames, the system automatically increases the overlapping area to improve the jitter compensation effect for severe shaking caused by gantry cranes and cranes in port operations.

[0125] Taking a port container surveillance video image with a resolution of 1080P (1920×1080 pixels) as an example, when S = 64 (block size), A = 16 (overlap area size), and effective step size = SA = 48 pixels, the following results can be obtained when the image is grid-blocked:

[0126] The number of blocks per row in the horizontal direction = 1920 / 48 = 40 blocks;

[0127] The number of blocks per column in the vertical direction = 1080 / 48 = 23 blocks;

[0128] Total number of blocks = 40 × 23 = 920 blocks.

[0129] When S=128 (i.e., larger blocks) and A=32 (i.e., larger overlapping areas), the effective step size is 96 pixels. When the image is divided into grid blocks, the following results can be obtained:

[0130] Total number of blocks = (1920 / 96) x (1080 / 96) = 20 × 12 = 240 blocks.

[0131] In some embodiments, step S4 includes:

[0132] Receive the center coordinate list of the first grid block set output in step S2 And the center coordinate list of the second grid block set output in step S3 ,right and Taking the union to obtain a comprehensive grid block set, and adding type tags to all grid blocks in the comprehensive grid block set, wherein the type tags of the grid blocks originally belonging to the first grid block set are adaptive grid type tags, and the type tags of the grid blocks originally belonging to the second grid block set and not belonging to the first grid block set are uniform grid type tags;

[0133] Outputting a merged grid block information list, the grid block information list including the center coordinates of the grid block, at least two edge coordinates of the grid block, and a type tag corresponding to the grid block.

[0134] In this embodiment, the adaptive and uniform grid block merging module directly combines the results of the two grid division strategies, eliminating the need for complex redundancy removal. Adaptive grid blocks correspond to feature-rich areas (such as container markings and operating equipment) in port container surveillance video images, while uniform grid blocks ensure complete coverage of the surveillance image. By simply merging these two types of grid blocks, the system can provide more refined processing of key areas while maintaining global coverage, thereby improving overall anti-shake effectiveness.

[0135] In actual port surveillance scenes, the number of adaptive grid blocks (based on feature points) is typically far smaller than that of uniform grid blocks, typically accounting for only 20%-30% of the total number of blocks. This moderate redundancy does not significantly increase the computational burden. For critical areas in port surveillance video images (such as container markings and operating equipment), the repeated coverage of multiple grid blocks actually helps improve the robustness and accuracy of anti-shake processing. Removing redundant detection steps significantly simplifies the algorithm flow, reduces system complexity, and improves processing efficiency and maintainability.

[0136] In the weighted fusion and reconstruction stage, this application designs a weight mechanism specifically for handling multi-grid block coverage, which can naturally cope with redundant coverage situations.

[0137] Assume that the grid block center coordinate set generated by adaptive grid division is , the grid block center coordinate set generated by uniform grid division is , the merge operation is simplified to:

[0138] Comprehensive grid block center coordinate set ;

[0139] The total number of merged grid blocks , and add a type tag to each grid block, where the type tag of the adaptive grid block is 1 and the type tag of the uniform grid block is 0.

[0140] Taking a 1080P port container surveillance video image (resolution of 1920×1080 pixels) as an example, assuming a grid block size of S = 128 pixels and an overlapping area size of A = 32 pixels, adaptive grid division can yield 150 grid blocks concentrated around port containers and equipment (i.e., covering key areas), and 240 uniform grid blocks covering the entire surveillance image. By merging these two types of grid blocks, a total of 390 grid blocks can be obtained.

[0141] The complete implementation process of step S4 is as follows Figure 10 As shown. Figure 11 and Figure 12 As shown in the figure, through this simple and effective direct merging strategy, the system fully utilizes the precise feature capture capability of the adaptive grid blocks and the comprehensive coverage characteristics of the uniform grid blocks, providing the optimal grid division basis for subsequent anti-shake processing. In actual port monitoring applications, the efficiency and robustness of this method can effectively cope with various complex shaking scenarios. Figure 12 It is not difficult to see that the average coverage of the key area of ​​the image is 4-6 grid blocks / pixel, indicating that the area is covered by both adaptive grid blocks and uniform grid blocks, while the average coverage of the non-key area (i.e., background area) is 2-3 grid blocks / pixel, indicating that the area is mainly covered by uniform grid blocks.

[0142] In some embodiments, step S5 includes:

[0143] Each pixel position , calculate according to the following formula and output a stable video frame:

[0144] ;

[0145] in, Is the position on the stable video frame The pixel value of It is grid blocks at position The pixel value of is the weight function value based on distance, is the weight adjustment coefficient of the grid block type, Is the coverage location The total number of grid blocks;

[0146] in, , d is the pixel The Euclidean distance to the center of the grid block, and K is the half-width of the grid block overlap area A.

[0147] Weighted fusion reconstruction is an advanced image reconstruction method used to merge overlapping, independently de-shaded grid blocks from port container surveillance video into a seamless, high-quality overall image. The algorithm specifically considers the characteristics of two different types of grid blocks: adaptive and uniform. Through intelligent weighting, it ensures optimal fusion and eliminates blur caused by severe shaking in port operation videos.

[0148] Weight adjustment coefficient of the grid block marked by the uniform grid type , the weight adjustment coefficient of the grid block marked by the adaptive grid type Adaptive grid blocks are given higher weights because they usually cover areas with more details and textures in the port container surveillance video images. By assigning higher weights, these details and textures can be better preserved.

[0149] Specifically, you can Set the weight to the maximum , at the edge of the grid block Set the weight to the minimum , with smooth transition in the middle area, ensuring seamless fusion of each grid block in the port container monitoring video image.

[0150] In this embodiment, for each pixel position in the port container monitoring video image, , you can get a list of all grid block indices covering the pixel, and then for each grid block covering the pixel , calculate the distance from a pixel to the center of the grid block: , at the same time, calculate the distance weight , and determine the grid block type weight adjustment coefficient The adaptive grid blocks are given higher weights, and the calculation process is as follows:

[0151] Accumulate weighted pixel values: ;

[0152] Cumulative weight: ;

[0153] Calculate the final pixel value: ;

[0154] For each frame image in the port container monitoring video sequence, the final stable frame image is obtained by updating the position information of the adaptive grid block (i.e., tracking the position of the key feature points) and recalculating the mapping relationship between pixels and grid blocks.

[0155] For example, the pixel position in the port container monitoring video image , assuming that the pixel is covered by two adaptive grid blocks and two uniform grid blocks, as follows:

[0156] Adaptive grid block 1 (this grid block covers the container logo): the center coordinates are ,distance , , , pixel value ;

[0157] Adaptive grid block 2 (this gateway block covers the feature area): the center coordinates are ,distance , , , pixel value ;

[0158] Uniform grid block 1 (this grid block covers the background area): the center coordinates are ,distance , , , pixel value ;

[0159] Uniform grid block 2 (this grid block covers the background area): the center coordinates are ,distance , , , pixel value ;

[0160] According to the above formula, the final pixel value corresponding to the pixel on the stable frame image can be calculated as: .

[0161] The specific implementation process of step S5 is as follows Figure 13As shown. This step combines distance weights and grid block type weights to more accurately control the fusion process and adaptively balance detail retention and smooth transitions in port surveillance video images. Specifically, it provides the maximum weight at the center of the grid block to ensure that key port monitoring features are retained, and smoothly decays to zero at the edges of the grid block to avoid visible splicing marks. Intelligently assign importance weights based on grid block characteristics, while maintaining overall consistency while enhancing detail-rich areas in port surveillance videos. It can handle grid block changes in mobile scenes in port container monitoring.

[0162] Adapt to the changes of feature points between different frames and maintain stable output.

[0163] The proposed method for stabilizing port container surveillance video uses adaptive grid segmentation, uniform grid coverage, and a weighted fusion algorithm to effectively address the problem of blurred surveillance footage, missing images, and difficulty tracing the source of the footage caused by the violent shaking of heavy equipment such as cranes and gantry cranes in port operating environments. This method achieves high-quality video stabilization by intelligently identifying shaking intensity, dynamically adjusting grid parameters, and combining a distance weighting function with grid block type weights.

[0164] In a second aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the video stabilization processing method as described in the first aspect of the present invention.

[0165] The computer-readable storage medium may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories.

[0166] The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface storage, an optical disc, or a compact disc read-only memory (CD ROM); the magnetic surface storage may be a magnetic disk storage or a magnetic tape storage.

[0167] The volatile memory may be a random access memory (RAM) that is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronized dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM). The computer-readable storage medium described in the embodiments of the present invention is intended to include these and any other suitable types of memory.

[0168] like Figure 14 As shown, in a third aspect, the present invention provides an electronic device 10, comprising a processor 101 and a storage medium 102, wherein a computer program is stored on the storage medium, and when the computer program is executed by the processor, the video stabilization processing method as described in the first aspect of the present invention is implemented.

[0169] In some embodiments, the processor can be implemented by software, hardware, firmware or a combination thereof, and can use a circuit, a single or multiple application-specific integrated circuits (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, or at least one of a microprocessor, so that the processor can execute some or all of the steps or any combination of the steps in the video stabilization processing method described in each embodiment of the present application.

[0170] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of this application, this does not limit the scope of patent protection of this application. All technical solutions generated by replacing or modifying equivalent structures or equivalent processes based on the essential concepts of this application using the contents recorded in the specification and drawings of this application, as well as directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, are included in the scope of patent protection of this application.

Claims

1. A video stabilization processing method, characterized in that: The following steps are involved: S1: Receive a first video frame, downsample the first video frame, extract feature points from the downsampled frame, calculate a homography transformation matrix between adjacent frames based on the feature points, accumulate and generate a global transformation matrix relative to a reference frame, and perform an alignment operation on the first video frame according to the global transformation matrix to obtain a second video frame; S2: using an adaptive detail grid division method and generating a first grid block set based on a corner detection algorithm, where the first grid block set covers a key area on the second video frame; S3: Generate a second set of grid blocks with global coverage using a uniform grid division method; S4: Taking the union of the first grid block set and the second grid block set to generate a comprehensive grid block set, adding type tags to all grid blocks in the comprehensive grid block set, and outputting a grid block information list corresponding to all grid blocks in the comprehensive grid block set; S5: According to the grid block information list, a weighted fusion and recombination method is applied to perform pixel-level fusion based on a distance weight function and a type tag weight coefficient, and a stable video frame is output.

2. The video stabilization processing method according to claim 1, wherein: Step S1 includes: The first video frame l orig (W×H) downsampled to I small (W s ×H s ), downsampling ratio Get downsampled consecutive frames and Extract feature points and establish correspondence between pixels Use the RANSAC algorithm to estimate the homography transformation matrix Make the pixels of consecutive frames after downsampling satisfy: in, represents the geometric transformation from frame t to frame t+1; Select reference frame I ref , for the downsampled frame t, calculate its cumulative transformation matrix relative to the reference frame The calculation formula is as follows: The accumulated transformation matrix at the downsampled resolution Convert to the transformation matrix H at the original resolution ref，t , for the first video frame at original resolution Apply the transformation matrix H ref,t Align and get the second video frame The calculation formula is as follows:

3. The video stabilization processing method according to claim 2, wherein: Let the image size of the second video frame be W×H, the grid block size be S×S, and the allowed overlap size between grid blocks be A. Then calculate the minimum distance D between feature points. min =SA, step S2 includes: Converting the second video frame into a grayscale image; Use Shi-Tomasi corner detection algorithm to detect N feature points in the grayscale image I. The feature point set is expressed as P = {p1, p2, ..., p n }, where each point p i =(x i ,y i ), any two points p i and p j The Euclidean distance between them is at least (SA); For each feature point p i =(x i ,y i ), generate a p i The grid block is S×S in size and centered at , and the grid block covers the area: Only the grid blocks that are completely within the grayscale image boundary are retained, and the center point p of the grid block is i =(x i ,y i ) meets the following conditions: and Output the coordinate list of all valid grid block center points L = {(x1, y1), (x2, y2), ..., (x m ,y m )}, and obtain the first grid block set.

4. The video stabilization processing method according to claim 3, wherein: Step S3 includes: Place the second video frame in a two-dimensional space coordinate system and calculate the center of the X-axis grid block: until And calculate the Y-axis grid block center: until Ensure that: and So that the edge grid blocks can completely cover the edge of the second video frame, a second grid block set is obtained: C1 = {(x, y) | x∈X, y∈Y}; For each pixel (x, y) in the image, calculate the number of grid blocks covering the pixel, create a list for each pixel, record the indices of all grid blocks covering the pixel, and obtain a coverage mapping map from pixels to grid blocks; The second set of grid blocks and the pixel-to-grid block coverage mapping map are output.

5. The video stabilization processing method according to claim 3 or 4, wherein: The size A of the overlapping area of ​​adjacent grid blocks is dynamically adjusted according to the intensity of the video frame shake. The adjustment formula is as follows: A adjusted =A base ×(1+k×M); Among them, A base is the basic overlap size; M is the shaking intensity factor, and k is the adjustment coefficient.

6. The video stabilization processing method according to claim 4, wherein: Step S4 includes: receiving the center coordinate list V of the first grid block set outputted in step S2 and the center coordinate list U of the second grid block set outputted in step S3, taking the union of V and U to obtain a comprehensive grid block set, and adding type tags to all grid blocks in the comprehensive grid block set, wherein the type tags of the grid blocks originally belonging to the first grid block set are adaptive grid type tags, and the type tags of the grid blocks originally belonging to the second grid block set and not belonging to the first grid block set are uniform grid type tags; Outputting a merged grid block information list, the grid block information list including the center coordinates of the grid block, at least two edge coordinates of the grid block, and a type tag corresponding to the grid block.

7. The video stabilization processing method according to claim 5, wherein: Step S5 includes: Each pixel position (x, y) is calculated according to the following formula to output a stable video frame: Among them, I final (x, y) is the pixel value at position (x, y) on the stable video frame, I i (x, y) is the pixel value of the i-th grid block at position (x, y), W i (x, y) is the weight function value based on the distance, α i is the weight adjustment factor for the grid block type, and n is the total number of grid blocks covering the position (x, y); in, d is the Euclidean distance from the pixel (x, y) to the center of the grid block, and K is the half-width of the grid block overlap area A.

8. The video stabilization processing method according to claim 7, wherein: Weight adjustment coefficient α for grid blocks marked with uniform grid type uniform =1.0, weight adjustment coefficient α of the grid block marked with adaptive grid type adaptive >1.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the video stabilization processing method according to any one of claims 1 to 8 is implemented.

10. An electronic device having a computer program stored thereon, characterized in that: The apparatus comprises a processor and a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by the processor, the video stabilization processing method according to any one of claims 1 to 8 is implemented.

Images