Adaptive Image Stabilization Method and System for Vehicle-Mounted Cameras

Abstract

This invention provides an adaptive image stabilization method and system for vehicle-mounted cameras, relating to the field of image processing technology. The method includes: acquiring video streams from a vehicle-mounted camera and IMU data from an inertial measurement unit (IMU) on the vehicle; performing frequency domain analysis on the IMU data to obtain motion compensation parameters, extracting texture feature points from image frames of the video stream, and calculating visual displacement vectors; determining a jitter compensation strategy for the image frames based on the difference between the motion compensation parameters and the visual displacement vectors; determining the final compensated motion field under the jitter compensation strategy based on the motion compensation parameters and the visual displacement vectors; and performing an inverse transformation on the image frames to obtain a jitter-compensated image stream. This invention solves the problems of low fusion between IMU data and images and poor adaptability to complex road conditions in existing vehicle-mounted camera image jitter processing.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to an adaptive image stabilization imaging method and system for vehicle-mounted cameras. Background Technology

[0002] With the development of intelligent driving technology, vehicle cameras have become core sensors for environmental perception. However, during vehicle operation, due to engine idling resonance, road bumps, or acceleration and deceleration, the vehicle body experiences complex vibrations with multiple degrees of freedom (pitch, yaw, vertical sway). This vibration causes the camera's optical axis to shift, resulting in blurred images or unstable image sequences, severely affecting the accuracy of subsequent target detection and tracking algorithms. Currently, mainstream image stabilization technologies are mainly divided into three categories: the first is mechanical image stabilization, which compensates for the optical path by physically moving the lens or sensor module. Although it has a fast response, it is limited by vehicle space and cost, and its compensation range for large-amplitude vibrations is limited. The second is electronic image stabilization, which compensates by cropping edge pixels using algorithms and using inertial measurement unit (IMU) data or images for feature matching (such as optical flow). However, simple IMU integration suffers from drift problems, while pure vision algorithms are prone to losing feature points during high-speed movement or drastic changes in lighting. Thirdly, there is the panoramic virtual window technology: the direction of the virtual viewfinder is corrected by sensor data, but this method is mainly used to enhance the panoramic video viewing experience and does not provide sufficient support for the perception tasks of high-precision intelligent driving assistance systems.

[0003] Existing technologies use simple methods such as feature matching to fuse IMU data with image data from vehicle-mounted cameras for image jitter compensation. However, the degree of data fusion is low, and the technology is poorly adaptable to complex road conditions. In particular, factors such as rainy days and nighttime conditions have a significant impact on the inertial measurement unit and the imaging of the vehicle-mounted camera, resulting in poor jitter compensation. Summary of the Invention

[0004] In view of this, it is necessary to provide an adaptive image stabilization method and system for vehicle cameras to solve the problems of low fusion degree of IMU data and image in existing vehicle camera image stabilization processing and poor adaptability to complex road conditions.

[0005] To address the above problems, this invention provides an adaptive image stabilization method for vehicle-mounted cameras, comprising: Acquire video streams from vehicle-mounted cameras and IMU data collected by the vehicle's inertial measurement unit; Frequency domain analysis is performed on the IMU data to obtain motion compensation parameters, and texture feature points are extracted from the image frames of the video stream. The visual displacement vector is calculated based on the texture feature points between the image frames. Determine the difference between the motion compensation parameter and the visual displacement vector; The jitter compensation strategy for the image frame is determined based on the difference, and the final compensated motion field under the jitter compensation strategy is determined based on the motion compensation parameters and the visual displacement vector. The jitter compensation strategy includes a visual compensation strategy and an IMU compensation strategy. Based on the final compensated motion field, the image frame is inversely transformed to obtain a jitter-compensated image stream.

[0006] In one possible implementation, the method further includes: Determine the timestamps of image frames in the video stream; Virtual IMU data for each target timestamp is generated from the IMU data by interpolation, wherein the target timestamp is a timestamp for which no corresponding IMU data exists.

[0007] In one possible implementation, the step of performing frequency domain analysis on the IMU data to obtain motion compensation parameters includes: Temperature compensation is applied to the IMU data, and the gravity component is removed from the compensated IMU data to obtain preprocessed IMU data. The preprocessed IMU data is subjected to Fourier transform to obtain the main jitter frequency; Motion compensation parameters are obtained by adjusting the process noise covariance matrix and measurement noise covariance matrix of the Kalman filter based on the main jitter frequency. The Kalman filter is used to predict the IMU data based on the measurement data of the inertial measurement unit.

[0008] In one possible implementation, the step of adjusting the process noise covariance matrix and the measurement noise covariance matrix of the Kalman filter based on the main jitter frequency to obtain motion compensation parameters includes: When the main jitter frequency is greater than the preset first frequency threshold, the Kalman filter is adjusted as follows: increase the process noise covariance matrix and decrease the measurement noise covariance matrix; When the main jitter frequency is less than the preset second frequency threshold, the Kalman filter is adjusted as follows: the process noise covariance matrix is ​​reduced, the measurement noise covariance matrix is ​​increased, and the first frequency threshold is greater than the second frequency threshold. Motion compensation parameters are determined based on the state vector output by the adjusted Kalman filter.

[0009] In one possible implementation, determining the jitter compensation strategy for the image frame based on the difference includes: When the difference is less than a preset error threshold, the jitter compensation strategy for the image frame is determined to be an IMU compensation strategy. When the difference is not less than a preset error threshold and the number of texture feature points of the image frame is greater than a preset number threshold, the jitter compensation strategy of the image frame is determined to be a visual compensation strategy. When the difference is not less than a preset error threshold and the number of texture feature points of the image frame is not greater than a preset number threshold, the jitter compensation strategy of the image frame is determined to be an IMU compensation strategy.

[0010] In one possible implementation, determining the final compensated motion field under the jitter compensation strategy based on the motion compensation parameters and the visual displacement vector includes: When the jitter compensation strategy is an IMU compensation strategy, the motion compensation parameters are determined as the final compensated motion field; When the jitter compensation strategy is a visual compensation strategy, the motion compensation parameters are drift-corrected based on the visual displacement vector to obtain the final compensated motion field.

[0011] In one possible implementation, the step of performing an inverse transformation on the image frame based on the final compensated motion field to obtain a jitter-compensated image stream includes: The region of interest in each image frame is divided into an original image grid; The vertex positions of each original image grid are determined based on the final compensated motion field; The target image grid is obtained by filling the image region formed by the vertex positions using bilinear interpolation; The target image grid is stretched according to a preset edge stretching coefficient, and the non-preset edge regions of the target image grid are linearly transformed to obtain a jitter-compensated image stream.

[0012] This invention provides an adaptive image stabilization imaging system for vehicle-mounted cameras, comprising: The acquisition module is used to acquire video streams captured by the vehicle-mounted camera and IMU data collected by the inertial measurement unit on the vehicle. The processing module is used to perform frequency domain analysis on the IMU data to obtain motion compensation parameters, extract texture feature points from the image frames of the video stream, and calculate the visual displacement vector based on the texture feature points between the image frames. A calculation module is used to determine the difference between the motion compensation parameters and the visual displacement vector; The decision module is used to determine the jitter compensation strategy of the image frame based on the difference, and to determine the final compensated motion field under the jitter compensation strategy based on the motion compensation parameters and the visual displacement vector. The jitter compensation strategy includes a visual compensation strategy and an IMU compensation strategy. The transformation module is used to perform an inverse transformation on the image frame based on the final compensated motion field to obtain a jitter-compensated image stream.

[0013] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program; the processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps of the above-described adaptive image stabilization imaging method for vehicle cameras.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described adaptive image stabilization method for vehicle-mounted cameras.

[0015] The beneficial effects of the above implementation are as follows: The adaptive image stabilization method and system for vehicle cameras provided by this invention firstly designs two independent processing links for image and IMU data to generate two types of jitter compensation data: motion compensation parameters and visual displacement vectors. Furthermore, an adaptive jitter compensation method is proposed, which determines the jitter compensation strategy for the image frame by comparing the difference between the two jitter compensation data. The difference in jitter compensation data can reflect the degree of difference in the impact of jitter on the image frame and IMU data under complex road conditions. Therefore, this adaptive compensation method can effectively adapt to various complex road conditions and determine a reasonable jitter compensation strategy. Under the determined jitter compensation strategy, the two jitter compensation data are further combined to determine the final compensation motion field for jitter compensation. Compared with feature matching of the original IMU data and image data, its data fusion degree is higher, and the jitter compensation effect is better. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the adaptive image stabilization imaging method for vehicle-mounted cameras provided by the present invention. Figure 2 This is a schematic diagram illustrating the principle of the adaptive image stabilization imaging method for vehicle-mounted cameras provided by the present invention. Figure 3 A processing diagram illustrating an example of an image stabilization imaging scenario provided by the present invention; Figure 4 This is a schematic diagram of the structure of the vehicle-mounted camera adaptive image stabilization imaging system provided by the present invention; Figure 5A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0019] In the description of the embodiments of this application, unless otherwise stated, "a plurality of" means two or more.

[0020] In this embodiment of the invention, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to such process, method, product or device.

[0021] The naming or numbering of steps in the embodiments of the present invention does not mean that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] The adaptive image stabilization method and system for vehicle-mounted cameras provided by this invention can be applied to imaging scenarios using vehicle-mounted cameras. The executing entity can be an in-vehicle system, such as an intelligent driving assistance system, or a server, terminal, or remote cloud device communicating with the in-vehicle system. During vehicle operation, video streams and IMU data are acquired and uploaded in real time through the in-vehicle camera and inertial measurement unit (IMU). Then, the adaptive image stabilization method and system for vehicle-mounted cameras provided by this invention are called to process the image frames of the video stream, ultimately outputting a stable image stream after shake compensation, which can be used for target detection, tracking algorithms, etc.

[0024] The adaptive image stabilization method for vehicle cameras provided by this invention will be described in detail below.

[0025] Figure 1 This is a flowchart illustrating the adaptive image stabilization method for vehicle-mounted cameras provided by the present invention, as shown below. Figure 1 As shown, the adaptive image stabilization method for vehicle cameras can be implemented through the following steps 101 to 105, which are explained in detail below.

[0026] Step 101: Acquire the video stream captured by the vehicle-mounted camera and the IMU data collected by the inertial measurement unit on the vehicle.

[0027] During vehicle operation, onboard cameras and an inertial measurement unit (IMU) continuously capture video streams of road conditions and angular velocity / acceleration data. The onboard camera is typically a global shutter camera, fixed to the windshield, used to continuously acquire images of the external road environment. The IMU can be a six-axis IMU (including a three-axis accelerometer and a three-axis gyroscope), rigidly connected to the printed circuit board of the onboard camera to ensure consistent motion between the two. The angular velocity / acceleration data from the IMU is typically filtered using a Kalman filter to obtain the IMU data.

[0028] like Figure 2 As shown, the image frames of the video stream and the IMU data are acquired by different sensors. The vehicle-mounted camera's acquisition frequency is 30fps, while the IMU's sampling frequency is generally in the range of 200Hz to 400Hz, which is incompatible with the vehicle-mounted camera. To address this, this embodiment of the invention establishes a synchronization triggering mechanism to perform data synchronization alignment, thereby solving the data asynchrony problem, which is described in detail below.

[0029] First, determine the timestamps of the image frames in the video stream, and then generate virtual IMU data for each target timestamp in the IMU data through interpolation.

[0030] Here, the timestamps of image frames in the video stream are used as references, and then each timestamp is queried to see if corresponding IMU data exists. During the query process, the target timestamp is determined, which is the timestamp where no corresponding IMU data exists.

[0031] For the target timestamp, this embodiment of the invention generates virtual IMU data for each target timestamp in the IMU data by interpolation, thereby ensuring that each image frame for each timestamp corresponds to IMU data and achieving data synchronization alignment.

[0032] In this embodiment of the invention, virtual IMU data is generated by interpolation, which realizes the time alignment between image frames and IMU data, and can effectively improve the instantaneous accuracy of subsequent jitter calculation.

[0033] Step 102: Perform frequency domain analysis on the IMU data to obtain motion compensation parameters, extract texture feature points from the image frames of the video stream, and calculate the visual displacement vector based on the texture feature points between image frames.

[0034] like Figure 2 As shown, after data synchronization and alignment are completed, two independent data processing chains are then entered. One is the visual processing chain, which processes the image frames of the video stream, and the other is the IMU processing chain, which processes the IMU data. These will be described separately below.

[0035] In the IMU processing chain, this embodiment of the invention performs frequency domain analysis on the IMU data to determine the main jitter frequency, and then uses an adaptive Kalman filter to determine the filtering parameters to obtain motion compensation parameters.

[0036] In one possible implementation, frequency domain analysis is performed on the IMU data to obtain motion compensation parameters. This can be achieved in the following ways, which are explained in detail below.

[0037] Temperature compensation is applied to the IMU data, and the gravity component is removed from the compensated IMU data to obtain preprocessed IMU data. The preprocessed IMU data is subjected to Fourier transform to obtain the main jitter frequency; Motion compensation parameters are obtained by adjusting the process noise covariance matrix and the measurement noise covariance matrix of the Kalman filter based on the master jitter frequency.

[0038] Here, the IMU data acquired by the inertial measurement unit (IMU) is first preprocessed, which includes gravity elimination and temperature compensation. Both temperature and gravity affect IMU data, so these factors need to be removed to ensure accuracy. Temperature compensation is achieved through weighted calculations using temperature weights, while gravity elimination is achieved by calculating the projection of gravity onto the IMU's body coordinate system using the IMU's attitude information (such as roll and pitch angles) and subtracting this component from the original acceleration measurement, thus eliminating the gravity component.

[0039] Further frequency domain analysis is performed on the preprocessed IMU data to extract spectral information. Specifically, the time domain is transformed into the frequency domain through Fourier transform to identify the main frequency components and determine the main jitter frequency f.

[0040] The Kalman filter is used to predict IMU data based on the measurement data from the inertial measurement unit. When vehicle vibration affects the IMU data, the filtering parameters of the Kalman filter can be adjusted by identifying the dominant vibration frequency f. Vehicle vibration is generally divided into two types: low-frequency, large-amplitude disturbances, such as the large-amplitude vibrations that occur when a vehicle drives over speed bumps or ditches, but at a very low frequency; and high-frequency, small-amplitude flutter, such as the continuous small-amplitude vibrations emitted by the vehicle's engine during operation, but at a very high frequency.

[0041] The type of vehicle vibration can be determined by the main vibration frequency f. Then, the filtering parameters of the Kalman filter can be adjusted according to the corresponding vibration characteristics. Specifically, these parameters include the process noise covariance matrix and the measurement noise covariance matrix. By correcting the filtering parameters, the motion compensation parameters corresponding to the IMU data can be obtained.

[0042] In one possible implementation, motion compensation parameters are obtained by adjusting the process noise covariance matrix and the measurement noise covariance matrix of the Kalman filter based on the master jitter frequency. This can be achieved in the following ways, which are explained in detail below.

[0043] When the main jitter frequency is greater than the preset first frequency threshold, the Kalman filter is adjusted as follows: increase the process noise covariance matrix and decrease the measurement noise covariance matrix; When the main jitter frequency is less than the preset second frequency threshold, the Kalman filter is adjusted as follows: the process noise covariance matrix is ​​reduced and the measurement noise covariance matrix is ​​increased.

[0044] Motion compensation parameters are determined based on the state vector output by the adjusted Kalman filter.

[0045] Specifically, here the first frequency threshold is greater than the second frequency threshold. If the dominant jitter frequency identified by the Fourier transform is greater than the preset first frequency threshold, it indicates that the current vehicle jitter is high-frequency jitter. In this case, the process noise covariance matrix Q of the Kalman filter can be adaptively increased. Specifically, this can be done by increasing the diagonal elements of the corresponding state variables in the Q matrix to increase the adaptability to high-frequency dynamic changes. At the same time, the measurement noise covariance matrix R can be adaptively decreased. Specifically, the diagonal element values ​​of the R matrix can be reduced near the dominant jitter frequency, thereby reducing the confidence in sensor readings for high-frequency noise.

[0046] If the dominant jitter frequency identified by the Fourier transform is less than a preset second frequency threshold, it indicates that the current vehicle jitter is low-frequency. In this case, the process noise covariance matrix Q of the Kalman filter is adaptively reduced. Specifically, this can be achieved by reducing the diagonal elements of the Q matrix corresponding to the state variables. This reduces the uncertainty assumptions of the system model, making the Kalman filter more reliant on the prediction model. This avoids the Kalman filter over-responding to low-frequency jitter and improves estimation stability. Simultaneously, the measurement noise covariance matrix R can be adaptively increased. Specifically, the diagonal elements of the R matrix can be increased near the dominant jitter frequency to reduce the reliance on the inertial measurement unit's measurement data. This makes the Kalman filter rely more on the prediction model and reduces the impact of low-frequency jitter on the estimation results.

[0047] The adjusted Kalman filter outputs an optimized state estimation vector when processing IMU data. From this vector, attitude parameters (roll, pitch, yaw) can be extracted, along with angular velocity estimates and attitude errors, which serve as the corresponding motion compensation parameters, denoted as... .

[0048] In this embodiment of the invention, by adjusting the process noise covariance matrix and measurement noise covariance matrix of the Kalman filter through the main jitter frequency, different motion compensation parameters can be calculated according to different jitter types, thereby determining different compensation levels and avoiding poor results caused by a single compensation strategy.

[0049] On the other hand, such as Figure 2 As shown, for each image frame in a video stream, the region of interest (ROI) is first determined through semantic segmentation. Then, features are extracted from the ROI using either the FAST or SIFT algorithm to extract texture feature points. Finally, feature matching is performed, matching texture feature points from adjacent image frames and calculating the positional difference of identical texture feature points between the two frames to obtain the visual displacement vector, denoted as [image frame name missing]. .

[0050] Step 103: Determine the difference between the motion compensation parameters and the visual displacement vector.

[0051] After processing the image frames and IMU data separately, an error function is further constructed to calculate motion compensation parameters. With visual displacement vector The difference, taken as the error function, is denoted as Error. The error function can characterize the difference between visual jitter compensation and IMU jitter compensation. Therefore, it can be used as a basis to estimate the complex scene in which the current camera and inertial measurement unit are located, make decisions on jitter compensation strategies, and determine a more reasonable jitter compensation method.

[0052] Step 104: Determine the jitter compensation strategy for the image frame based on the difference, and determine the final compensated motion field under the jitter compensation strategy based on the motion compensation parameters and the visual displacement vector.

[0053] Here, jitter compensation strategies include visual compensation strategies and IMU compensation strategies. When making jitter compensation decisions based on the difference (Error) between motion compensation parameters and the visual displacement vector, it can be determined whether to adopt a visual compensation strategy or an IMU compensation strategy. The visual compensation strategy primarily utilizes the visual displacement vector... The strategy for jitter compensation is to utilize motion compensation parameters. The strategy for jitter compensation.

[0054] In one possible implementation, the jitter compensation strategy for determining the image frame based on the difference can be achieved in the following way, which is explained in detail below.

[0055] When the difference is less than the preset error threshold, the jitter compensation strategy for the image frame is determined to be the IMU compensation strategy. When the difference is not less than the preset error threshold and the number of texture feature points of the image frame is greater than the preset number threshold, the jitter compensation strategy of the image frame is determined to be the visual compensation strategy. When the difference is not less than the preset error threshold and the number of texture feature points of the image frame is less than the preset number threshold, the jitter compensation strategy for the image frame is determined to be the IMU compensation strategy.

[0056] First, this embodiment of the invention sets a preset error threshold to determine the motion compensation parameters. With visual displacement vector The degree of difference. If the difference Error is less than the preset error threshold Th, it means that the difference between motion compensation and visual displacement is not significant. In this case, the motion compensation parameters... With visual displacement vector Both can be used for jitter compensation. Considering the strong robustness of IMU data in complex road conditions such as darkness and rain, the jitter compensation strategy for image frames is determined to be the IMU compensation strategy, that is, using motion compensation parameters. Perform jitter compensation.

[0057] If the error value (Error) is not less than the preset error threshold, it indicates a significant difference between motion compensation and visual displacement. This may be due to a large deviation in visual displacement, as complex road conditions such as darkness or rain can severely impact image frame collection, leading to substantial visual shift errors. Therefore, further judgment is needed: specifically, whether the number of texture feature points in the image frame exceeds the preset threshold. If so, it means that complex road conditions such as darkness or rain have not been encountered, the current image frame still displays features, and the visual displacement is still effective. In this case, the jitter compensation strategy for the image frame can be determined to be a visual compensation strategy, i.e., using the visual displacement vector. Perform jitter compensation. If not, it indicates that the current situation is complex, such as nighttime or rainy weather, and features cannot be extracted from the current image frame, making the calculated visual displacement invalid. In this case, the jitter compensation strategy for the image frame can be determined to be an IMU compensation strategy, i.e., using motion compensation parameters. Perform jitter compensation.

[0058] This invention proposes an adaptive jitter compensation method that determines the jitter compensation strategy for an image frame by comparing the difference between two jitter compensation data. This method can effectively adapt to various complex road conditions and make the jitter compensation strategy decision more reasonable and accurate.

[0059] After deciding on the jitter compensation strategy, the final compensated motion field under the jitter compensation strategy is further determined based on the motion compensation parameters and the visual displacement vector.

[0060] like Figure 2 As shown, after determining the jitter compensation strategy, under each jitter compensation strategy, this embodiment of the invention continues to incorporate motion compensation parameters. With visual displacement vector This is used to determine the final jitter compensation parameters for the image frame. Specifically, it is based on the motion compensation parameters. With visual displacement vector To determine the final compensation sports field, denoted as... .

[0061] In one possible implementation, the final compensated motion field under the jitter compensation strategy is determined based on motion compensation parameters and visual displacement vectors. This can be achieved in the following way, which will be explained in detail below.

[0062] When the jitter compensation strategy is the IMU compensation strategy, the motion compensation parameter is determined as the compensation motion field; When the jitter compensation strategy is a visual compensation strategy, the motion compensation parameters are drift-corrected based on the visual displacement vector to obtain the final compensated motion field.

[0063] Specifically, when the jitter compensation strategy is an IMU compensation strategy, it mainly relies on IMU data for jitter compensation. The motion compensation parameters can be directly determined as the final compensated motion field. .

[0064] When the jitter compensation strategy is a visual compensation strategy, it mainly relies on image frames for jitter compensation, and can utilize visual displacement vectors. Motion compensation parameters Offset correction is then performed. Specifically, the difference between the visual displacement vector and the motion predicted by the motion compensation parameters is first compared to identify systematic biases in the IMU data. Then, the offset to be corrected is calculated based on the visual displacement vector and the systematic biases. Finally, the calculated offset is applied to the IMU compensation parameters to achieve dynamic correction of the motion compensation parameters, resulting in the final compensated motion field. .

[0065] In this embodiment of the invention, under different jitter compensation strategies, the final jitter correction parameters are determined by combining motion compensation parameters and visual displacement vectors, which effectively realizes the data fusion of image vision and IMU motion. Compared with simply combining the original IMU data and image data through feature matching, its jitter compensation effect is better.

[0066] Step 105: Perform an inverse transformation on the image frame based on the final compensated motion field to obtain the jitter-compensated image stream.

[0067] like Figure 2 As shown, after determining the final compensation motion field based on the motion compensation parameters and visual displacement vector under the corresponding jitter compensation strategy, it can be used to perform jitter compensation on the image frames of the video stream. By performing an inverse transformation on the image frames through the final compensation motion field, a stable image stream that is not affected by jitter offset is obtained.

[0068] Here, this invention proposes a grid flow method to inversely transform an image and achieve jitter compensation. Specifically, the image frame is divided into a grid, and then the final compensation motion field is used to adjust the image grid vertices. Furthermore, interpolation filling and edge stretching are used to correct the non-pixel areas and black border areas formed after the image grid vertices are adjusted.

[0069] In one possible implementation, the image frame is inversely transformed based on the final compensated motion field to obtain the jitter-compensated image stream. This can be achieved in the following way, which is explained in detail below.

[0070] The region of interest in each image frame is divided into an original image grid; The vertex positions of each original image grid are determined based on the final compensated motion field; The target image grid is obtained by filling the image regions formed by the vertex positions using bilinear interpolation; The target image grid is stretched according to a preset edge stretching coefficient, and the non-preset edge regions of the target image grid are linearly transformed to obtain the jitter-compensated image stream.

[0071] Specifically, the region of interest (ROI) is first identified within the image frame requiring jitter compensation, and then this region is divided into multiple original image grids. Further, the position of each vertex in the original image grid is moved using the final compensation motion field, and this position is used as the vertex position of the jitter-compensated image grid.

[0072] Because the positions of the grid vertices change, the size and shape of the original image grid change, and some non-pixel regions may appear in the image grid. To address this, this embodiment of the invention uses bilinear interpolation to fill these non-pixel regions, ensuring that each area of ​​the image grid is filled with pixels, thus obtaining the target image grid.

[0073] Finally, mesh detail processing is performed. The filled target image mesh may contain some black borders, creating a difference between the mesh and the image regions within the target image mesh. To eliminate these black borders, a preset edge stretching coefficient is used to stretch the pixels at preset edge positions of the target image mesh. Pixels at non-preset edge positions, which are the pixels in the center region, are simply transformed linearly, resulting in the jitter-compensated image stream.

[0074] This invention employs a grid flow method to compensate for image frame jitter. By offsetting the vertex positions of the image grid and filling image regions, it avoids generating a large number of black areas during image processing. Compared to direct zooming and cropping, this method ensures that the jitter-processed image does not lose resolution. Furthermore, by using an edge stretching coefficient to stretch the edge regions of the image grid while applying a linear transformation to the central region, it minimizes distortion of the central field of view while ensuring the absence of black borders, thus guaranteeing image quality.

[0075] The adaptive image stabilization imaging method for vehicle cameras provided by this invention will be illustrated below with a specific example.

[0076] like Figure 3 As shown, when a vehicle is driving normally, a shaking phenomenon occurs. At this time, the vehicle camera and inertial measurement unit collect video streams and IMU data respectively, and send the data to vehicle systems such as intelligent driving assistance systems. The preprocessing module of the vehicle system first generates virtual IMU data according to the timestamps corresponding to the image frames of the video stream to achieve data synchronization and alignment, and then sends the aligned image frames and IMU data to the algorithm module.

[0077] The algorithm module executes two data processing links to process image frames and IMU data respectively, obtaining visual displacement vectors and motion compensation parameters. Further jitter compensation decision-making is performed to determine the jitter compensation strategy, and under each strategy, the final compensated motion field is determined based on the visual displacement vectors and motion compensation parameters. This final compensated motion field enters the image reconstruction module of the in-vehicle system, where it is used to perform inverse transformation on the image frames to achieve jitter compensation, ultimately generating a jitter-compensated stable image stream for use in target recognition and tracking algorithms, etc.

[0078] This invention proposes two independent processing paths for image and IMU data to generate two types of jitter compensation data: motion compensation parameters and visual displacement vectors. Furthermore, an adaptive jitter compensation method is proposed, which determines the jitter compensation strategy for the image frame by comparing the difference between the two jitter compensation data. The difference in jitter compensation data reflects the varying degrees of impact of jitter on the image frame and IMU data under complex road conditions. Therefore, this adaptive compensation method can effectively adapt to complex road conditions and determine a more reasonable jitter compensation strategy. Finally, under the determined jitter compensation strategy, the two jitter compensation data are combined to determine the final compensation motion field for jitter compensation. This data fusion method achieves a higher degree of data fusion than simply combining the original IMU data and image data through feature matching, resulting in superior jitter compensation performance.

[0079] The adaptive image stabilization imaging system for vehicle cameras provided by this invention will be described in detail below.

[0080] like Figure 4 As shown, the vehicle-mounted camera adaptive image stabilization system specifically includes: acquisition module 401, processing module 402, calculation module 403, decision module 404, and transformation module 405.

[0081] Specifically, the acquisition module 401 is used to acquire video streams captured by the vehicle-mounted camera and IMU data collected by the inertial measurement unit on the vehicle; the processing module 402 is used to perform frequency domain analysis on the IMU data to obtain motion compensation parameters, extract texture feature points from the image frames of the video stream, and calculate visual displacement vectors based on the texture feature points between image frames; the calculation module 403 is used to determine the difference between the motion compensation parameters and the visual displacement vector; the decision module 404 is used to determine the jitter compensation strategy for the image frame based on the difference, and determine the final compensated motion field under the jitter compensation strategy based on the motion compensation parameters and the visual displacement vector, wherein the jitter compensation strategy includes a visual compensation strategy and an IMU compensation strategy; and the transformation module 405 is used to perform an inverse transformation on the image frame based on the final compensated motion field to obtain a jitter-compensated image stream.

[0082] In one possible implementation, the processing module 402 is further configured to determine the timestamps of image frames in the video stream; and to generate virtual IMU data for each target timestamp in the IMU data by interpolation, wherein the target timestamp is a timestamp for which no corresponding IMU data exists.

[0083] The vehicle-mounted camera adaptive image stabilization imaging system provided in the above embodiments can realize the technical solutions described in the above embodiments of the vehicle-mounted camera adaptive image stabilization imaging method. The specific implementation principles of each module or unit can be found in the corresponding content in the above embodiments of the vehicle-mounted camera adaptive image stabilization imaging method, and their technical effects can also be referred to each other. They will not be repeated here.

[0084] like Figure 5 As shown, the present invention also provides an electronic device 500. The electronic device 500 includes a processor 501, a memory 502, and a display 503. Figure 5 Only some components of the electronic device 500 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0085] In some embodiments, memory 502 may be an internal storage unit of electronic device 500, such as a hard disk or memory of electronic device 500. In other embodiments, memory 502 may also be an external storage device of electronic device 500, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 500.

[0086] Furthermore, the memory 502 may include both internal storage units of the electronic device 500 and external storage devices. The memory 502 is used to store application software and various types of data installed on the electronic device 500.

[0087] In some embodiments, processor 501 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 502 or process data, such as the adaptive image stabilization method for vehicle cameras in this invention.

[0088] In some embodiments, display 503 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 503 is used to display information from electronic device 500 and to display a visual user interface. Components 401-403 of electronic device 500 communicate with each other via a system bus.

[0089] In some embodiments of the present invention, when the processor 501 executes the computer program in the memory 502, the following steps can be implemented: acquiring video streams collected by vehicle-mounted cameras and IMU data collected by inertial measurement units on the vehicle; performing frequency domain analysis on the IMU data to obtain motion compensation parameters, and extracting texture feature points from image frames of the video stream, calculating visual displacement vectors based on texture feature points between image frames; determining the difference between the motion compensation parameters and the visual displacement vectors; determining a jitter compensation strategy for the image frames based on the difference, and determining the final compensated motion field under the jitter compensation strategy based on the motion compensation parameters and the visual displacement vectors, wherein the jitter compensation strategy includes a visual compensation strategy and an IMU compensation strategy; and performing an inverse transformation on the image frames based on the final compensated motion field to obtain a jitter-compensated image stream.

[0090] It should be understood that when the processor 501 executes the computer program in the memory 502, in addition to the functions described above, it can also perform other functions, as can be found in the description of the corresponding method embodiments above.

[0091] Furthermore, the embodiments of the present invention do not specifically limit the type of electronic device 500 mentioned. Electronic device 500 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, electronic device 500 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0092] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the adaptive image stabilization method for vehicle-mounted cameras provided by the methods described above. This method includes: acquiring a video stream captured by a vehicle-mounted camera and IMU data collected by an inertial measurement unit (IMU) on the vehicle; performing frequency domain analysis on the IMU data to obtain motion compensation parameters, and extracting texture feature points from image frames of the video stream, calculating a visual displacement vector based on the texture feature points between image frames; determining the difference between the motion compensation parameters and the visual displacement vector; determining a jitter compensation strategy for the image frame based on the difference, and determining a final compensated motion field under the jitter compensation strategy based on the motion compensation parameters and the visual displacement vector, wherein the jitter compensation strategy includes a visual compensation strategy and an IMU compensation strategy; and performing an inverse transformation on the image frame based on the final compensated motion field to obtain a jitter-compensated image stream.

[0093] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0094] The above provides a detailed description of the adaptive image stabilization method and system for vehicle-mounted cameras provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. An adaptive image stabilization imaging method for vehicle-mounted cameras, characterized in that, include: Acquire video streams from vehicle-mounted cameras and IMU data collected by the vehicle's inertial measurement unit; Frequency domain analysis is performed on the IMU data to obtain motion compensation parameters, and texture feature points are extracted from the image frames of the video stream. The visual displacement vector is calculated based on the texture feature points between the image frames. Determine the difference between the motion compensation parameter and the visual displacement vector; The jitter compensation strategy for the image frame is determined based on the difference, and the final compensated motion field under the jitter compensation strategy is determined based on the motion compensation parameters and the visual displacement vector. The jitter compensation strategy includes a visual compensation strategy and an IMU compensation strategy. Based on the final compensated motion field, the image frame is inversely transformed to obtain a jitter-compensated image stream.

2. The adaptive image stabilization imaging method for vehicle-mounted cameras according to claim 1, characterized in that, The method further includes: Determine the timestamps of image frames in the video stream; Virtual IMU data for each target timestamp is generated from the IMU data by interpolation, wherein the target timestamp is a timestamp for which no corresponding IMU data exists.

3. The adaptive image stabilization imaging method for vehicle-mounted cameras according to claim 1, characterized in that, The frequency domain analysis of the IMU data to obtain motion compensation parameters includes: Temperature compensation is applied to the IMU data, and the gravity component is removed from the compensated IMU data to obtain preprocessed IMU data. The preprocessed IMU data is subjected to Fourier transform to obtain the main jitter frequency; Motion compensation parameters are obtained by adjusting the process noise covariance matrix and measurement noise covariance matrix of the Kalman filter based on the main jitter frequency. The Kalman filter is used to predict the IMU data based on the measurement data of the inertial measurement unit.

4. The adaptive image stabilization imaging method for vehicle-mounted cameras according to claim 3, characterized in that, The process of adjusting the Kalman filter based on the main jitter frequency, using the process noise covariance matrix and the measurement noise covariance matrix to obtain motion compensation parameters, includes: When the main jitter frequency is greater than the preset first frequency threshold, the Kalman filter is adjusted as follows: increase the process noise covariance matrix and decrease the measurement noise covariance matrix; When the main jitter frequency is less than the preset second frequency threshold, the Kalman filter is adjusted as follows: the process noise covariance matrix is ​​reduced, the measurement noise covariance matrix is ​​increased, and the first frequency threshold is greater than the second frequency threshold. Motion compensation parameters are determined based on the state vector output by the adjusted Kalman filter.

5. The adaptive image stabilization imaging method for vehicle-mounted cameras according to claim 1, characterized in that, The step of determining the jitter compensation strategy for the image frame based on the difference includes: When the difference is less than a preset error threshold, the jitter compensation strategy for the image frame is determined to be an IMU compensation strategy. When the difference is not less than a preset error threshold and the number of texture feature points of the image frame is greater than a preset number threshold, the jitter compensation strategy of the image frame is determined to be a visual compensation strategy. When the difference is not less than a preset error threshold and the number of texture feature points of the image frame is not greater than a preset number threshold, the jitter compensation strategy of the image frame is determined to be an IMU compensation strategy.

6. The adaptive image stabilization imaging method for vehicle-mounted cameras according to claim 1, characterized in that, Determining the final compensated motion field under the jitter compensation strategy based on the motion compensation parameters and the visual displacement vector includes: When the jitter compensation strategy is an IMU compensation strategy, the motion compensation parameters are determined as the final compensated motion field; When the jitter compensation strategy is a visual compensation strategy, the motion compensation parameters are drift-corrected based on the visual displacement vector to obtain the final compensated motion field.

7. The adaptive image stabilization imaging method for vehicle-mounted cameras according to claim 1, characterized in that, The step of performing an inverse transformation on the image frame based on the final compensated motion field to obtain a jitter-compensated image stream includes: The region of interest in each image frame is divided into an original image grid; The vertex positions of each original image grid are determined based on the final compensated motion field; The target image grid is obtained by filling the image region formed by the vertex positions using bilinear interpolation; The target image grid is stretched according to a preset edge stretching coefficient, and the non-preset edge regions of the target image grid are linearly transformed to obtain a jitter-compensated image stream.

8. An adaptive image stabilization imaging system for vehicle-mounted cameras, characterized in that, include: The acquisition module is used to acquire video streams captured by the vehicle-mounted camera and IMU data collected by the inertial measurement unit on the vehicle. The processing module is used to perform frequency domain analysis on the IMU data to obtain motion compensation parameters, extract texture feature points from the image frames of the video stream, and calculate the visual displacement vector based on the texture feature points between the image frames. A calculation module is used to determine the difference between the motion compensation parameters and the visual displacement vector; The decision module is used to determine the jitter compensation strategy of the image frame based on the difference, and to determine the final compensated motion field under the jitter compensation strategy based on the motion compensation parameters and the visual displacement vector. The jitter compensation strategy includes a visual compensation strategy and an IMU compensation strategy. The transformation module is used to perform an inverse transformation on the image frame based on the final compensated motion field to obtain a jitter-compensated image stream.

9. An electronic device, characterized in that, Including memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps of the vehicle-mounted camera adaptive image stabilization imaging method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the vehicle-mounted camera adaptive image stabilization imaging method as described in any one of claims 1 to 7.

Images