High-performance image stabilization method and device based on video surveillance equipment

Abstract

The present application discloses a high-performance image stabilization method and device based on video surveillance equipment. The method includes: obtaining the real-time camera posture of the camera and multiple two-dimensional image sequences; according to the feature matching algorithm, extracting and matching the feature points of each frame of the two-dimensional image in the multiple two-dimensional image sequences to obtain multiple matching feature points; according to the motion recovery structure algorithm, estimating the multiple matching feature points to obtain the estimated camera posture corresponding to the camera; fusing the real-time camera posture with the estimated camera posture to obtain the fused camera posture, and according to the fused camera posture, obtaining the fused motion trajectory corresponding to the camera; performing smoothing filtering on the fused motion trajectory to obtain the smoothing filtering trajectory, and according to the smoothing filtering trajectory and the fused motion trajectory, obtaining the cropping change matrix; according to the cropping change matrix, transforming the multiple two-dimensional image sequences to obtain a stabilized image.

Description

Technical Field

[0001] The present application relates to the field of video surveillance technology, and in particular to a high-performance image stabilization method and device based on video surveillance equipment. Background Art

[0002] With the advent of the digital video era, digital videos are widely used in various scenes in life. The monitoring equipment that obtains videos is often in a harsh external environment, resulting in deviations in the quality of the obtained video images. In addition, in a harsh external environment, severe weather such as wind and snow can cause mechanical shaking of the supporting structure of the monitoring equipment, resulting in blurring of the images obtained by the monitoring equipment. As a result, video stabilization technology has also emerged. Video stabilization technology can also be called video de-shaking technology. Video stabilization technology mainly uses algorithms to reduce video jitter to generate stable videos and improve video quality.

[0003] However, the traditional video stabilization method uses a single electronic image stabilization algorithm and uses image processing methods to eliminate various jitters in the video image. Its focus is only on image processing, and the disturbance caused by mechanical movement is not considered, which makes the single electronic image stabilization algorithm have low stability and the obtained image quality deviation.

[0004] Application Contents

[0005] The purpose of this application is to solve the technical problem of low stability of traditional video stabilization methods. To achieve the above purpose, this application provides a high-performance stabilization method and device based on video monitoring equipment.

[0006] The present application provides a high-performance image stabilization method based on a video surveillance device, comprising:

[0007] Obtain the real-time camera pose and multiple 2D image sequences of the camera;

[0008] Extracting and matching feature points of each two-dimensional image in the plurality of two-dimensional image sequences according to a feature matching algorithm to obtain a plurality of matching feature points;

[0009] estimating the plurality of matching feature points according to a structure-from-motion algorithm to obtain an estimated camera pose corresponding to the camera;

[0010] Performing data fusion on the real-time camera posture and the estimated camera posture to obtain a fused camera posture, and obtaining a fused motion trajectory corresponding to the camera according to the fused camera posture;

[0011] Performing smoothing filtering on the fused motion trajectory to obtain a smoothing filtering trajectory, and obtaining a clipping change matrix according to the smoothing filtering trajectory and the fused motion trajectory;

[0012] The plurality of two-dimensional image sequences are transformed according to the cropping change matrix to obtain a stable image.

[0013] In one embodiment, in the step of fusing the real-time camera posture with the estimated camera posture to obtain a fused camera posture, and obtaining a fused motion trajectory corresponding to the camera according to the fused camera posture, the real-time camera posture includes a real-time camera translation amount and a real-time camera relative rotation angle, and the estimated camera posture includes an estimated camera translation amount and an estimated camera relative rotation angle;

[0014] The step of fusing the real-time camera posture with the estimated camera posture to obtain a fused camera posture, and obtaining a fused motion trajectory corresponding to the camera according to the fused camera posture includes:

[0015] Performing data fusion on the real-time camera translation amount and the estimated camera translation amount to obtain a fused camera translation amount;

[0016] Performing data fusion of the real-time camera relative rotation angle and the estimated camera relative rotation angle to obtain a fused camera relative rotation angle;

[0017] The fusion motion trajectory is formed according to the translation amount of the fusion camera and the relative rotation angle of the fusion camera.

[0018] In one embodiment, performing smoothing filtering on the fused motion trajectory to obtain a clipping change matrix includes:

[0019] constructing an objective function according to the fused motion trajectory;

[0020] Adjusting the objective function to minimize, obtaining a smoothing filter trajectory, and obtaining the clipping change matrix according to the smoothing filter trajectory and the fused motion trajectory;

[0021] Wherein, the objective function is:

[0022]

[0023] A = B × C;

[0024] A represents the smoothing filter trajectory, B represents the fusion motion trajectory, C represents the cropping change matrix, w1, w2, w3 and w4 represent the balance factors of the energy function, |D 1 (A) |1 represents the static data item corresponding to the static shooting of the camera, |D 2 (A) |1 represents the uniform speed data item corresponding to the camera moving at a uniform speed, |D 3 (A)|1 indicates the uniform acceleration data item corresponding to the uniformly accelerated movement of the camera. represents the data item between the fused motion trajectory and the smoothing filter trajectory, and t represents the frame number.

[0025] In one embodiment, the adjusting the objective function to minimize and obtain a smooth filtering trajectory includes:

[0026] According to the L1 norm and L2 norm, an optimization function is constructed;

[0027] The objective function is optimized according to the optimization function so as to minimize the objective function and obtain the smoothing filter trajectory.

[0028] In one embodiment, the constraint conditions of the objective function are that the relative rotation angle of the camera does not exceed 3 degrees, the camera translation does not exceed 20 mm, and the camera is stationary shooting corresponding to the stationary data item |D 1 (A)|1 remains unchanged before and after optimization.

[0029] In one embodiment, obtaining the real-time camera posture of the camera includes:

[0030] Acquiring the real-time camera attitude according to the inertial measurement element;

[0031] Wherein, the inertial measurement element is arranged on the connection mechanism between the camera and the base.

[0032] In one embodiment, the present application provides a high-performance image stabilization device based on a video surveillance device, comprising:

[0033] A data acquisition module, used to acquire the real-time camera posture and multiple two-dimensional image sequences of the camera;

[0034] A matching feature point acquisition module is used to extract and match the feature points of each frame of the two-dimensional image in the plurality of two-dimensional image sequences according to a feature matching algorithm to obtain a plurality of matching feature points;

[0035] An estimated camera pose acquisition module, used to estimate the multiple matching feature points according to a motion recovery structure algorithm to obtain an estimated camera pose corresponding to the camera;

[0036] A fusion motion trajectory acquisition module is used to perform data fusion on the real-time camera posture and the estimated camera posture to obtain a fusion camera posture, and obtain a fusion motion trajectory corresponding to the camera according to the fusion camera posture;

[0037] A cropping change matrix acquisition module is used to perform smoothing filtering on the fused motion trajectory to obtain a smoothing filter trajectory, and obtain a cropping change matrix according to the smoothing filter trajectory and the fused motion trajectory;

[0038] The stable image acquisition module is used to transform the multiple two-dimensional image sequences according to the cropping change matrix to obtain a stable image.

[0039] In one embodiment, in the fusion motion trajectory acquisition module, the real-time camera posture includes the real-time camera translation amount and the real-time camera relative rotation angle, and the estimated camera posture includes the estimated camera translation amount and the estimated camera relative rotation angle;

[0040] The fusion motion trajectory acquisition module includes:

[0041] A fused camera translation acquisition module is used to perform data fusion on the real-time camera translation and the estimated camera translation to obtain a fused camera translation;

[0042] A fusion camera relative rotation angle acquisition module is used to fuse the real-time camera relative rotation angle with the estimated camera relative rotation angle to obtain the fusion camera relative rotation angle;

[0043] The trajectory generation module is used to form the fusion motion trajectory according to the translation amount of the fusion camera and the relative rotation angle of the fusion camera.

[0044] In one embodiment, the cropping change matrix acquisition module includes:

[0045] An objective function construction module, used to construct an objective function according to the fused motion trajectory;

[0046] A change matrix generation module, used for adjusting the objective function to be minimized, obtaining a smoothing filter trajectory, and obtaining the clipping change matrix according to the smoothing filter trajectory and the fused motion trajectory;

[0047] Wherein, the objective function is:

[0048]

[0049] A = B × C;

[0050] A represents the smoothing filter trajectory, B represents the fusion motion trajectory, C represents the cropping change matrix, w1, w2, w3 and w4 represent the balance factors of the energy function, |D 1 (A) |1 represents the static data item corresponding to the static shooting of the camera, |D 2 (A) |1 represents the uniform speed data item corresponding to the camera moving at a uniform speed, |D 3 (A)|1 indicates the uniform acceleration data item corresponding to the uniformly accelerated movement of the camera. represents the data item between the fused motion trajectory and the smoothing filter trajectory, and t represents the frame number.

[0051] In one embodiment, the change matrix generation module includes:

[0052] An optimization function building module is used to build an optimization function based on the L1 norm and the L2 norm;

[0053] The smoothing filter trajectory acquisition module is used to optimize the objective function according to the optimization function so as to minimize the objective function and obtain the smoothing filter trajectory.

[0054] In one embodiment, in the smoothing filter trajectory acquisition module, the constraint conditions of the objective function are that the relative rotation angle of the camera does not exceed 3 degrees, the camera translation does not exceed 20 mm, and the static data item corresponding to the static shooting of the camera |D 1 (A)|1 remains unchanged before and after optimization.

[0055] In one embodiment, the data acquisition module acquires the real-time camera posture according to an inertial measurement element;

[0056] Wherein, the inertial measurement element is arranged on the connection mechanism between the camera and the base.

[0057] The above-mentioned high-performance image stabilization method based on video surveillance equipment can fuse the real-time camera posture of the camera measured in the actual mechanical structure with the estimated camera posture estimated by the image, and perform matrix transformation on the two-dimensional image based on the fused motion trajectory formed after the fusion, so as to obtain the final stable image. Through the high-performance image stabilization method of the present application, the inter-frame offset of the image sequence caused by external factors is reduced, the image distortion caused by camera shake is corrected, and finally a stable video image is output. The high-performance image stabilization method based on video surveillance equipment of the present application solves the problem of low accuracy of single electronic image stabilization caused by shaking of the base of the monitoring equipment, improves the quality of video surveillance, and can be applied to roadside intelligent video surveillance systems under any environmental conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a schematic diagram of the overall process of the high-performance image stabilization method based on video surveillance equipment provided by this application.

[0059] Figure 2 It is a schematic diagram of the overall structure of a high-performance image stabilization method based on video surveillance equipment provided in this application. DETAILED DESCRIPTION

[0060] The technical solution of the present application is further described in detail below through the accompanying drawings and embodiments.

[0061] See also Figure 1, the present application provides a high-performance image stabilization method based on a video surveillance device, comprising:

[0062] S10, obtaining a real-time camera posture and a plurality of two-dimensional image sequences of the camera;

[0063] S20, extracting and matching feature points of each two-dimensional image in the plurality of two-dimensional image sequences according to a feature matching algorithm to obtain a plurality of matching feature points;

[0064] S30, estimating multiple matching feature points according to a structure-from-motion algorithm to obtain an estimated camera pose corresponding to the camera;

[0065] S40, fusing the real-time camera posture with the estimated camera posture to obtain a fused camera posture, and obtaining a fused motion trajectory corresponding to the camera according to the fused camera posture;

[0066] S50, performing smoothing filtering on the fused motion trajectory to obtain a smoothing filtering trajectory, and obtaining a clipping change matrix according to the smoothing filtering trajectory and the fused motion trajectory;

[0067] S60, transforming the plurality of two-dimensional image sequences according to the cropping change matrix to obtain a stable image.

[0068] In S10, each of the multiple two-dimensional image sequences is a continuous image sequence. The multiple two-dimensional image sequences are acquired by collecting with a camera.

[0069] When obtaining the real-time camera attitude of the camera, the inertial measurement element is installed on the connection mechanism between the camera and the base. According to the inertial measurement unit (IMU), the real-time camera attitude can be obtained. The inertial measurement element can include devices such as sensors, which can obtain the real-time camera attitude. The real-time camera attitude includes the real-time camera translation and the real-time camera relative rotation angle. According to the connection mechanism between the base and the camera, the inertial measurement element can be used to perform disturbance measurement on the connection mechanism to measure the real-time camera attitude and angular velocity.

[0070] In S20, the feature matching algorithm can also be understood as a feature point extraction algorithm. The feature matching algorithm includes a scale invariant feature transform algorithm (SIFT), a directional gradient histogram algorithm (HOG), a local binary pattern (LBP algorithm), etc. In one embodiment, a scale invariant feature transform algorithm can be used to extract feature points and perform feature matching on each frame of a two-dimensional image.

[0071] In S30, a structure from motion (SFM) algorithm can obtain camera parameters and perform three-dimensional reconstruction by analyzing multiple two-dimensional image sequences. The multiple two-dimensional image sequences include visual motion information. According to the structure from motion algorithm, by analyzing multiple matching feature points obtained in the multiple two-dimensional image sequences, the motion trajectory of the camera can be estimated to obtain an estimated camera posture.

[0072] In S40, the real-time camera posture of the camera is fused with the estimated camera posture estimated according to the image sequence, so that the disturbance factors caused between the camera and the base can be fused into the camera posture estimated according to the image sequence, and the fused camera posture after fusion can be obtained. According to the fused camera posture, the corresponding fused motion trajectory after data fusion can be obtained. The camera posture includes the camera translation amount and the camera relative rotation angle. According to the camera translation amount and the camera relative rotation angle, the corresponding camera motion trajectory can be obtained.

[0073] In S50, by performing smoothing filtering on the fused motion trajectory, the fused motion trajectory can be optimized. By performing smoothing filtering on the fused motion trajectory, the jitter in the video can be further removed to obtain the optimal smoothing filtering trajectory. There is a relationship A=B×C between the smoothing filtering trajectory A, the fused motion trajectory B, and the cropping change matrix C. The cropping change matrix C can be obtained according to the smoothing filtering trajectory A and the fused motion trajectory B.

[0074] In S60, scaling, rotation, affine transformation, displacement and other transformations can be achieved through the cropping transformation matrix, and multiple two-dimensional image sequences can be transformed to obtain a stable image sequence.

[0075] Through S10 to S60, the present application can fuse the real-time camera posture of the camera measured in the actual mechanical structure with the estimated camera posture estimated by the image, and perform a matrix transformation on the two-dimensional image based on the fused motion trajectory formed after the fusion to obtain the final stable image. Through the high-performance image stabilization method of the present application, the inter-frame offset of the image sequence caused by external factors is reduced, the image distortion caused by camera shake is corrected, and finally a stable video image is output. The present application is based on a high-performance image stabilization method for video surveillance equipment, which solves the problem of low accuracy of single electronic image stabilization caused by shaking of the base of the surveillance equipment, improves the quality of video surveillance, and can be applied to roadside intelligent video surveillance systems under any environmental conditions.

[0076] In one embodiment, in S20, when feature point extraction and feature matching are performed on each frame of a plurality of two-dimensional image sequences according to a feature matching algorithm, feature point extraction is performed on adjacent two-dimensional images, and feature point matching is performed. When matching feature points, the Euclidean distance ratio determination method is used for matching, which can improve the accuracy of matching. Set the distance 1 between the feature point 1 of the t-th frame image and the feature point 2 closest to the feature point 1 in the t+1-th frame image, and the distance 2 between the feature point 1 of the t-th frame image and the feature point 3 closest to the feature point 1 in the t+1-th frame image. If the ratio of distance 1 to distance 2 exceeds the set threshold, it is considered that the feature points do not match, and the unmatched feature points are deleted. If the ratio of distance 1 to distance 2 does not exceed the set threshold, it is considered that the feature points match, and the matched feature points are retained. After feature point matching screening, accurate matching feature points are obtained.

[0077] In one embodiment, in S30, multiple matching feature points are estimated according to the motion recovery structure algorithm to obtain the estimated camera pose corresponding to the camera, the camera is calibrated to solve the intrinsic parameter matrix, and the camera parameter matrix, camera rotation matrix and camera translation matrix are obtained to obtain the estimated camera pose corresponding to the camera.

[0078] In one embodiment, in S30, initial camera positions corresponding to different frame images in multiple two-dimensional image sequences are calculated by a synchronous positioning and mapping algorithm, and the precise camera position is calculated using the SFM algorithm to obtain the corresponding estimated camera pose.

[0079] In one embodiment, in S40, the real-time camera posture and the estimated camera posture are fused to obtain the fused camera posture, and in the step of obtaining the fused motion trajectory corresponding to the camera according to the fused camera posture, the real-time camera posture includes the real-time camera translation amount and the real-time camera relative rotation angle, and the estimated camera posture includes the estimated camera translation amount and the estimated camera relative rotation angle. S40 includes:

[0080] S410, fusing the real-time camera translation amount with the estimated camera translation amount to obtain a fused camera translation amount;

[0081] S420, fusing the real-time camera relative rotation angle with the estimated camera relative rotation angle to obtain a fused camera relative rotation angle;

[0082] S430: forming a fusion motion trajectory according to the translation amount of the fusion camera and the relative rotation angle of the fusion camera.

[0083] In S410, the real-time camera translation and the estimated camera translation may be the variation between two adjacent frames of two-dimensional images. Data fusion may include summing and averaging the real-time camera translation and the estimated camera translation to obtain a fused camera translation after data fusion.

[0084] The real-time camera translation T and the real-time camera relative rotation angle θ constitute the real-time camera posture [T, θ]. x , t y , t z ] represents the real-time camera translation between two adjacent 2D image frames at the current moment. θ=[θ x ,θ y ,θ z ] represents the real-time relative rotation angle of the camera between two adjacent two-dimensional image frames at the current moment.

[0085] The estimated camera translation T and the estimated camera relative rotation angle θ constitute the estimated camera pose [T', θ']. T' = [t' x , t' y , t' z ] represents the real-time camera translation between two adjacent two-dimensional images at the current moment. θ'=[θ' x ,θ' y ,θ' z ] represents the real-time relative rotation angle of the camera between two adjacent two-dimensional images at the current moment

[0086] In S420, the real-time camera relative rotation angle and the estimated camera relative rotation angle may also be the variation between two adjacent frames of two-dimensional images. Data fusion may include summing and averaging the real-time camera relative rotation angle and the estimated camera relative rotation angle to obtain the fused camera relative rotation angle after data fusion.

[0087] In S430, the fused camera posture [T", θ"] is obtained based on the fused data, the fused camera translation amount and the relative rotation angle of the fused camera. According to the fused camera posture [T", θ"], the fused motion trajectory of the camera after the data corresponding to the camera is fused can be obtained, which can also be understood as a fused motion path.

[0088] Through S410 to S430, the present application can fuse the real-time camera posture of the camera measured in the actual mechanical structure with the estimated camera posture estimated by the image to form a fused motion trajectory. The fused motion trajectory fuses the mechanical vibration information and the image estimation information, and can stabilize the image from the motion trajectories in two different directions, reduce the inter-frame offset of the image sequence caused by external factors, correct the image distortion caused by camera shake, and finally output a stable video image.

[0089] In one embodiment, S50, performing smoothing filtering on the fused motion trajectory to obtain a clipping change matrix includes:

[0090] S510, constructing an objective function according to the fused motion trajectory;

[0091] S520, adjusting the objective function to be minimized, obtaining a smoothing filter trajectory, and obtaining a clipping change matrix according to the smoothing filter trajectory and the fused motion trajectory;

[0092] Among them, the objective function is:

[0093]

[0094] A = B × C;

[0095] A represents the smoothing filter trajectory, B represents the fusion motion trajectory, C represents the cropping change matrix, w1, w2, w3 and w4 represent the balance factors of the energy function, |D 1 (A) |1 represents the static data item corresponding to the static shooting of the camera, |D 2 (A) |1 represents the uniform speed data item corresponding to the camera moving at a uniform speed, |D 3 (A)|1 indicates the uniform acceleration data item corresponding to the uniformly accelerated movement of the camera. It represents the data item between the fused motion trajectory and the smoothing filter trajectory, and t represents the number of frames.

[0096] In S510, a motion path fitting method is adopted, and the camera path is composed of three modes: the camera is stationary, the camera moves at a constant speed, and the camera moves at a constant acceleration. From the perspective of photography, the three shooting modes can bring the best viewing experience. Then, corresponding path data items are established according to the three motion modes to construct the objective function. In addition, in order to prevent the smoothing filter trajectory from being far away from the fused motion trajectory, a data item between the smoothing filter trajectory and the fused motion trajectory is established. The data items are as follows:

[0097]

[0098]

[0099]

[0100]

[0101]

[0102] Among them, C t is the fusion motion trajectory B of the tth frame t Transformed into the smoothing filter trajectory A of the tth framet The transformation matrix represents the cropping change matrix of the tth frame.

[0103] In S520, the objective function is minimized by optimizing the objective function, and finally converged to an asymptotically stable point to smooth the filter trajectory. By performing smooth filtering on the fused motion trajectory, the jitter in the video can be further removed to obtain the optimal smooth filtering trajectory. Then, the smooth filtering trajectory is combined with the fused motion trajectory to obtain a cropping change matrix. The cropping change matrix is ​​applied to the original two-dimensional image sequence to obtain a stable image sequence.

[0104] Through S510 and S520, the present application can perform smoothing filtering on the fused motion trajectory that integrates mechanical vibration information and image estimation information to obtain the optimal smoothing filtering trajectory, and further obtain the optimal cropping change matrix to obtain a stable video image.

[0105] In one embodiment, S520, adjusting the objective function to minimize and obtaining a smoothing filter trajectory includes:

[0106] S521, constructing an optimization function according to the L1 norm and the L2 norm;

[0107] S522, optimizing the objective function according to the optimization function to minimize the objective function and obtain a smoothing filtering trajectory.

[0108] In S521, an optimization function is constructed by L1 norm and L2 norm to optimize the fusion motion trajectory of the camera. On the basis of the original L1 filter idea, an L2 filter optimization term is added to enhance the similarity between motion trajectories and strengthen the constraints. It is expanded into a trajectory smoothing processing method to achieve motion smoothing.

[0109] In S522, the objective function is optimized by the optimization function to minimize the objective function and finally converge to an asymptotically stable point to obtain the optimal smoothing filter trajectory. By performing smoothing filter processing on the fused motion trajectory, the jitter in the video can be further removed to obtain the optimal smoothing filter trajectory.

[0110] In one embodiment, the constraint conditions of the objective function are that the relative rotation angle of the camera between two adjacent two-dimensional image frames does not exceed 3 degrees, and the camera translation between two adjacent two-dimensional image frames does not exceed 20 mm, and the static data item |D corresponding to the static camera shooting 1 (A)|1 remains unchanged before and after optimization.

[0111] By establishing the constraint conditions of the objective function, the content of the image can be kept consistent when the image is transformed and will not deviate from the fusion motion trajectory. This ensures the consistency of the viewpoints of the two images after the transformation and obtains the image sequence under the final stable trajectory.

[0112] In one embodiment, when applying the algorithm in the above embodiment, GPU accelerated computing technology may be added to obtain a better camera motion trajectory.

[0113] In one embodiment, the high-performance image stabilization method based on video surveillance equipment provided by the present application further includes meshing the stabilized image obtained in S60, and establishing an optimization function according to the four vertices of the mesh corresponding to the matching feature points. The optimal transformation matrix of the mesh is obtained through the mesh optimization function, and the transformation matrix is ​​applied to the corresponding meshes respectively to further obtain a stabilized image.

[0114] See also Figure 2 In one embodiment, the present application provides a high-performance image stabilization device 100 based on a video surveillance device, including a data acquisition module 10, a matching feature point acquisition module 20, an estimated camera posture acquisition module 30, a fused motion trajectory acquisition module 40, a cropping change matrix acquisition module 50, and a stabilized image acquisition module 60. The data acquisition module 10 is used to obtain the real-time camera posture of the camera and multiple two-dimensional image sequences. The matching feature point acquisition module 20 is used to extract and match features for each frame of two-dimensional images in multiple two-dimensional image sequences according to a feature matching algorithm to obtain multiple matching feature points. The estimated camera posture acquisition module 30 is used to estimate multiple matching feature points according to a motion recovery structure algorithm to obtain the estimated camera posture corresponding to the camera;

[0115] The fused motion trajectory acquisition module 40 is used to fuse the real-time camera posture with the estimated camera posture to obtain the fused camera posture, and obtain the fused motion trajectory corresponding to the camera according to the fused camera posture. The cropping change matrix acquisition module 50 is used to perform smoothing filtering on the fused motion trajectory to obtain the smoothing filter trajectory, and obtain the cropping change matrix according to the smoothing filter trajectory and the fused motion trajectory. The stable image acquisition module 60 is used to transform multiple two-dimensional image sequences according to the cropping change matrix to obtain a stable image.

[0116] In this embodiment, the relevant description of the data acquisition module 10 may refer to the relevant description of S10 in the above embodiment. The relevant description of the matching feature point acquisition module 20 may refer to the relevant description of S20 in the above embodiment. The relevant description of the estimated camera posture acquisition module 30 may refer to the relevant description of S30 in the above embodiment. The relevant description of the fused motion trajectory acquisition module 40 may refer to the relevant description of S40 in the above embodiment. The relevant description of the cropping change matrix acquisition module 50 may refer to the relevant description of S50 in the above embodiment. The relevant description of the stabilized image acquisition module 60 may refer to the relevant description of S60 in the above embodiment.

[0117] The high-performance image stabilization device 100 provided in the present application can fuse the real-time camera posture of the camera measured in the actual mechanical structure with the estimated camera posture estimated by the image, and perform a matrix transformation on the two-dimensional image based on the fused motion trajectory formed after the fusion, so as to obtain the final stabilized image. Through the high-performance image stabilization device 100 of the present application, the inter-frame offset of the image sequence caused by external factors is reduced, the image distortion caused by the camera shake is corrected, and finally a stable video image is output. The high-performance image stabilization device 100 of the present application solves the problem of low accuracy of single electronic image stabilization caused by shaking of the base of the monitoring equipment, improves the quality of video monitoring, and can be applied to roadside intelligent video monitoring systems under any environmental conditions.

[0118] In one embodiment, in the fused motion trajectory acquisition module 40 , the real-time camera posture includes the real-time camera translation and the real-time camera relative rotation angle, and the estimated camera posture includes the estimated camera translation and the estimated camera relative rotation angle.

[0119] The fusion motion trajectory acquisition module 40 includes a fusion camera translation acquisition module, a fusion camera relative rotation angle acquisition module and a trajectory generation module. The fusion camera translation acquisition module is used to fuse the real-time camera translation with the estimated camera translation to obtain the fusion camera translation. The fusion camera relative rotation angle acquisition module is used to fuse the real-time camera relative rotation angle with the estimated camera relative rotation angle to obtain the fusion camera relative rotation angle. The trajectory generation module is used to form a fusion motion trajectory according to the fusion camera translation and the fusion camera relative rotation angle.

[0120] In this embodiment, the description of the fusion camera translation acquisition module can refer to the description of S410 in the above embodiment. The description of the fusion camera relative rotation angle acquisition module can refer to the description of S420 in the above embodiment. The description of the trajectory generation module can refer to the description of S430 in the above embodiment.

[0121] In one embodiment, the cropping change matrix acquisition module 50 includes an objective function construction module and a change matrix generation module. The objective function construction module is used to construct an objective function according to the fused motion trajectory. The change matrix generation module is used to adjust the objective function to minimize, obtain a smoothing filter trajectory, and obtain a cropping change matrix according to the smoothing filter trajectory and the fused motion trajectory.

[0122] Among them, the objective function is:

[0123]

[0124] A = B × C;

[0125] A represents the smoothing filter trajectory, B represents the fusion motion trajectory, C represents the cropping change matrix, w1, w2, w3 and w4 represent the balance factors of the energy function, |D 1 (A) |1 represents the static data item corresponding to the static shooting of the camera, |D 2 (A) |1 represents the uniform speed data item corresponding to the camera moving at a uniform speed, |D 3 (A)|1 indicates the uniform acceleration data item corresponding to the uniformly accelerated movement of the camera. It represents the data item between the fused motion trajectory and the smoothing filter trajectory, and t represents the number of frames.

[0126] In this embodiment, the description of the objective function construction module can refer to the description of S510 in the above embodiment. The description of the change matrix generation module can refer to the description of S520 in the above embodiment.

[0127] In one embodiment, the change matrix generation module includes an optimization function construction module and a smoothing filter trajectory acquisition module. The optimization function construction module is used to construct an optimization function according to the L1 norm and the L2 norm. The smoothing filter trajectory acquisition module is used to optimize the objective function according to the optimization function so that the objective function is minimized and the smoothing filter trajectory is obtained.

[0128] In this embodiment, the description of the optimization function construction module can refer to the description of S521 in the above embodiment. The description of the smoothing filter trajectory acquisition module can refer to the description of S522 in the above embodiment.

[0129] In one embodiment, in the smoothing filter trajectory acquisition module, the constraint conditions of the objective function are that the relative rotation angle of the camera does not exceed 3 degrees, the camera translation does not exceed 20 mm, and the static data item corresponding to the static camera shooting |D 1 (A)|1 remains unchanged before and after optimization.

[0130] For the description of the constraint conditions of the objective function in this embodiment, reference may be made to the above embodiment.

[0131] In one embodiment, the data acquisition module 10 acquires the real-time camera posture according to the inertial measurement element, wherein the inertial measurement element is arranged in the connection mechanism between the camera and the base.

[0132] In this embodiment, the relevant description of obtaining the real-time camera posture can refer to the above embodiment.

[0133] In the above-mentioned various embodiments, the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims provide the elements of various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0134] Those skilled in the art may also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present application may be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly demonstrate the interchangeability of hardware and software, the various illustrative components, units, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present application.

[0135] The various illustrative logic blocks or units described in the embodiments of the present application can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or the design of any combination of the above functions. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any traditional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor core, or any other similar configuration to implement.

[0136] The steps of the method or algorithm described in the embodiments of the present application can be directly embedded in the software module executed by the hardware, the processor, or a combination of the two. The software module can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM or other storage media of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Alternatively, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a user terminal. Optionally, the processor and the storage medium can also be arranged in different components in the user terminal.

[0137] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A high-performance image stabilization method based on video surveillance equipment, characterized in that: include: Obtain the real-time camera pose and multiple 2D image sequences of the camera; Extracting and matching feature points of each two-dimensional image in the plurality of two-dimensional image sequences according to a feature matching algorithm to obtain a plurality of matching feature points; estimating the plurality of matching feature points according to a structure-from-motion algorithm to obtain an estimated camera pose corresponding to the camera; Performing data fusion on the real-time camera posture and the estimated camera posture to obtain a fused camera posture, and obtaining a fused motion trajectory corresponding to the camera according to the fused camera posture; Performing smoothing filtering on the fused motion trajectory to obtain a smoothing filtering trajectory, and obtaining a clipping change matrix according to the smoothing filtering trajectory and the fused motion trajectory; The plurality of two-dimensional image sequences are transformed according to the cropping change matrix to obtain a stable image.

2. The high-performance image stabilization method based on video surveillance equipment according to claim 1, characterized in that: In the step of fusing the real-time camera posture with the estimated camera posture to obtain a fused camera posture, and obtaining a fused motion trajectory corresponding to the camera according to the fused camera posture, the real-time camera posture includes a real-time camera translation amount and a real-time camera relative rotation angle, and the estimated camera posture includes an estimated camera translation amount and an estimated camera relative rotation angle; The step of fusing the real-time camera posture with the estimated camera posture to obtain a fused camera posture, and obtaining a fused motion trajectory corresponding to the camera according to the fused camera posture includes: Performing data fusion on the real-time camera translation amount and the estimated camera translation amount to obtain a fused camera translation amount; Performing data fusion of the real-time camera relative rotation angle and the estimated camera relative rotation angle to obtain a fused camera relative rotation angle; The fusion motion trajectory is formed according to the translation amount of the fusion camera and the relative rotation angle of the fusion camera.

3. The high-performance image stabilization method based on video surveillance equipment according to claim 1, characterized in that: The step of performing smoothing filtering on the fused motion trajectory to obtain a clipping change matrix includes: constructing an objective function according to the fused motion trajectory; Adjusting the objective function to minimize, obtaining a smoothing filter trajectory, and obtaining the clipping change matrix according to the smoothing filter trajectory and the fused motion trajectory; Wherein, the objective function is: A = B × C; A represents the smoothing filter trajectory, B represents the fusion motion trajectory, C represents the cropping change matrix, w1, w2, w3 and w4 represent the balance factors of the energy function, |D 1 (A) |1 represents the static data item corresponding to the static shooting of the camera, |D 2 (A) |1 represents the uniform speed data item corresponding to the camera moving at a uniform speed, |D 3 (A)|1 indicates the uniform acceleration data item corresponding to the uniformly accelerated movement of the camera. represents the data item between the fused motion trajectory and the smoothing filter trajectory, and t represents the frame number.

4. The high-performance image stabilization method based on video surveillance equipment according to claim 3, characterized in that: The step of adjusting the objective function to minimize and obtaining a smooth filtering trajectory includes: According to the L1 norm and L2 norm, an optimization function is constructed; The objective function is optimized according to the optimization function so as to minimize the objective function and obtain the smoothing filter trajectory.

5. The high-performance image stabilization method based on video surveillance equipment according to claim 4, characterized in that: The constraint conditions of the objective function are that the relative rotation angle of the camera does not exceed 3 degrees, the camera translation does not exceed 20 mm, and the static data item corresponding to the static shooting of the camera |D 1 (A)|1 remains unchanged before and after optimization.

6. The high-performance image stabilization method based on video surveillance equipment according to claim 1, characterized in that: The obtaining of the real-time camera posture of the camera comprises: Acquiring the real-time camera attitude according to the inertial measurement element; Wherein, the inertial measurement element is arranged on the connection mechanism between the camera and the base.

7. A high-performance image stabilization device based on video surveillance equipment, characterized in that: include: A data acquisition module, used to acquire the real-time camera posture and multiple two-dimensional image sequences of the camera; A matching feature point acquisition module is used to extract and match the feature points of each frame of the two-dimensional image in the plurality of two-dimensional image sequences according to a feature matching algorithm to obtain a plurality of matching feature points; An estimated camera pose acquisition module, used to estimate the multiple matching feature points according to a motion recovery structure algorithm to obtain an estimated camera pose corresponding to the camera; A fusion motion trajectory acquisition module is used to perform data fusion on the real-time camera posture and the estimated camera posture to obtain a fusion camera posture, and obtain a fusion motion trajectory corresponding to the camera according to the fusion camera posture; A cropping change matrix acquisition module is used to perform smoothing filtering on the fused motion trajectory to obtain a smoothing filter trajectory, and obtain a cropping change matrix according to the smoothing filter trajectory and the fused motion trajectory; The stable image acquisition module is used to transform the multiple two-dimensional image sequences according to the cropping change matrix to obtain a stable image.

8. The high-performance image stabilization device based on video surveillance equipment according to claim 7, characterized in that: In the fusion motion trajectory acquisition module, the real-time camera posture includes the real-time camera translation amount and the real-time camera relative rotation angle, and the estimated camera posture includes the estimated camera translation amount and the estimated camera relative rotation angle; The fusion motion trajectory acquisition module includes: A fused camera translation acquisition module is used to perform data fusion on the real-time camera translation and the estimated camera translation to obtain a fused camera translation; A fusion camera relative rotation angle acquisition module is used to fuse the real-time camera relative rotation angle with the estimated camera relative rotation angle to obtain the fusion camera relative rotation angle; The trajectory generation module is used to form the fusion motion trajectory according to the translation amount of the fusion camera and the relative rotation angle of the fusion camera.

9. The high-performance image stabilization device based on video surveillance equipment according to claim 7, characterized in that: The clipping change matrix acquisition module includes: An objective function construction module, used to construct an objective function according to the fused motion trajectory; A change matrix generation module, used for adjusting the objective function to be minimized, obtaining a smoothing filter trajectory, and obtaining the clipping change matrix according to the smoothing filter trajectory and the fused motion trajectory; Wherein, the objective function is: A = B × C; A represents the smoothing filter trajectory, B represents the fusion motion trajectory, C represents the cropping change matrix, w1, w2, w3 and w4 represent the balance factors of the energy function, |D 1 (A) |1 represents the static data item corresponding to the static shooting of the camera, |D 2 (A) |1 represents the uniform speed data item corresponding to the camera moving at a uniform speed, |D 3 (A)|1 indicates the uniform acceleration data item corresponding to the uniformly accelerated movement of the camera. represents the data item between the fused motion trajectory and the smoothing filter trajectory, and t represents the frame number.

10. The high-performance image stabilization device based on video surveillance equipment according to claim 9, characterized in that: The change matrix generation module includes: An optimization function building module is used to build an optimization function based on the L1 norm and the L2 norm; The smoothing filter trajectory acquisition module is used to optimize the objective function according to the optimization function so as to minimize the objective function and obtain the smoothing filter trajectory.

11. The high-performance image stabilization device based on video surveillance equipment according to claim 10, characterized in that: In the smoothing filter trajectory acquisition module, the constraint conditions of the objective function are that the relative rotation angle of the camera does not exceed 3 degrees, the camera translation does not exceed 20 mm, and the static data item corresponding to the static shooting of the camera |D 1 (A)|1 remains unchanged before and after optimization.

12. The high-performance image stabilization device based on video surveillance equipment according to claim 7, characterized in that: The data acquisition module acquires the real-time camera posture according to the inertial measurement element; Wherein, the inertial measurement element is arranged on the connection mechanism between the camera and the base.

Images