A remote video picture processing method and system based on an automatic driving carrier platform

Abstract

This invention relates to the field of remote monitoring and image processing technology for autonomous driving, and discloses a remote video image processing method and system based on an autonomous driving vehicle platform. The method includes: acquiring images from multiple wide-angle cameras and six-axis IMU data, and synchronizing multi-source sensor data; performing attitude prediction and vibration mode separation based on the synchronized IMU data, outputting pixel displacement prediction values, predicted homography matrix, and relative rotational perturbations between cameras; obtaining an independent stabilized image through adaptive image stabilization transformation; performing dynamic compensation and spatiotemporal consistency fusion of the surround-view stitching seams based on the relative rotational perturbations to generate a surround-view panoramic image; and outputting the final image based on adaptive distortion suppression using a human spatial perception model. This invention improves the smoothness and spatial perception reliability of remotely controlled images under vibration conditions on unpaved roads by jointly optimizing IMU high-frequency prediction and sparse optical flow, and dynamically adjusting seams and fusion weights in the stitching transition zone.

Description

Technical Field

[0001] This invention relates to the field of remote monitoring and image processing technology for autonomous driving, and in particular to a method and system for remote video image processing based on an autonomous driving vehicle platform. Background Technology

[0002] With the development of autonomous driving technology, the demand for remote control or monitoring of autonomous vehicles is increasing. Remote operators need to obtain real-time panoramic video images of the vehicle's surroundings to make accurate driving decisions. However, during operation, the vibration and bumps of the autonomous vehicle itself, as well as the time synchronization issues of the multi-camera system, cause video images to become jittery, misaligned, and distorted, seriously affecting the accuracy and comfort of remote control.

[0003] In existing technologies, most common video stabilization methods rely solely on image features or simple inertial measurement unit (IMU) data, which are insufficient for separating complex motions and high-frequency vibrations during vehicle movement, resulting in limited stabilization effects. Regarding surround-view stitching, traditional methods assume rigid connections between cameras and a static environment, failing to adapt to relative pose changes caused by vehicle vibrations, leading to noticeable breaks or ghosting at stitching seams. Furthermore, panoramic image output does not consider human spatial perception characteristics, resulting in simplistic distortion handling and causing errors in remote operators' judgments of the distance and shape of objects around the vehicle. Therefore, a video processing method capable of effectively handling multi-sensor synchronization, vibration compensation, dynamic stitching, and adaptive distortion suppression is needed. Summary of the Invention

[0004] To address the aforementioned technical shortcomings, the purpose of this invention is to propose a remote video image processing method based on an autonomous driving vehicle platform. This method aims to solve the technical problems in existing technologies that rely solely on mechanical image stabilization or fixed-parameter electronic image stabilization, lack multi-sensor fusion and dynamic stitching compensation mechanisms, especially under conditions of severe vehicle vibration causing sudden changes in relative displacement between multiple cameras on unpaved roads, making it impossible to achieve real-time image stabilization and seamless panoramic stitching on low-computing-power embedded devices.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention provides a remote video image processing method based on an autonomous driving vehicle platform.

[0006] The remote video processing method based on an autonomous driving platform includes: Step S10: Acquire images from multiple wide-angle cameras and a set of six-axis inertial measurement unit (IMU) data. Based on timestamp adaptive interpolation alignment and adaptive low-pass filtering, a cubic spline interpolation resampling mechanism is used to perform the multi-source sensor data synchronous acquisition and alignment task, and output the aligned IMU data and multiple wide-angle camera images. Step S20: Based on the aligned IMU data, the camera attitude prediction and vibration compensation tasks are performed using IMU pre-integration and high-frequency prediction of camera attitude and vibration mode separation mechanism, and the pixel displacement prediction value, prediction homography matrix and relative rotation perturbation between adjacent cameras are output. Step S30: Based on the multi-channel wide-angle camera images and the pixel displacement prediction value and prediction homography matrix, an adaptive image stabilization transformation mechanism with sparse optical flow feature tracking and IMU joint constraints is used to perform the image stabilization task and output independent stabilized images of each camera. Step S40: Based on the independent shaking images of each camera and the relative rotation disturbance between adjacent cameras, a panoramic stitching task is performed using a dynamic compensation and spatiotemporal consistency fusion mechanism for surround view stitching seams, and a surround view panoramic image is output. Step S50: Based on the panoramic image, perform adaptive distortion suppression and image output based on the human spatial perception model, and output the final image.

[0007] Preferably, in step S10, the multi-source sensor data synchronous acquisition and alignment task is performed using a cubic spline interpolation resampling mechanism based on timestamp adaptive interpolation alignment and adaptive low-pass filtering, specifically including: Step S101: Multiple wide-angle cameras and a six-axis inertial measurement unit (IMU) are simultaneously connected to the vehicle-mounted embedded platform. Each wide-angle camera is synchronously exposed by an external trigger signal and outputs an image stream with a resolution of 1280×720 at a frame rate of 30fps. A precise timestamp is generated by hardware at the center of each frame. The IMU outputs three-axis angular velocity and three-axis acceleration at a frequency of 200Hz. Each data packet carries a timestamp generated by the same clock source. Step S102: Send the image stream and IMU data stream into the circular buffer. Perform adaptive low-pass filtering preprocessing on the raw IMU data. The filter cutoff frequency is dynamically adjusted based on the current vehicle speed and steering angle: when the vehicle speed is not lower than 0.5 m / s or the steering angle is not greater than 15°, the cutoff frequency increases linearly with the vehicle speed, with an upper limit of 40 Hz; when the vehicle speed is lower than 0.5 m / s and the steering angle is greater than 15°, a correction term proportional to the standard deviation of the IMU acceleration is superimposed on top of the above, with a correction coefficient of 2 Hz·s. 2 / m; Step S103: Perform cubic spline interpolation resampling on the filtered IMU angular velocity sequence to obtain the IMU virtual measurement value corresponding to the exposure time of each frame of the image, thereby obtaining a precisely aligned timestamp sequence. Provide the aligned IMU data to step S20 and provide the image to step S30.

[0008] Preferably, the step of performing camera attitude prediction and vibration compensation tasks based on the aligned IMU data, using IMU pre-integration and camera attitude high-frequency prediction, and vibration mode separation mechanisms, specifically includes: Step S201: Receive the aligned IMU angular velocity and acceleration, perform median integral pre-integration on the angular velocity and acceleration between adjacent image moments to obtain the rotation increment and position increment of the vehicle body coordinate system, use the pre-calibrated extrinsic parameters between each camera and the IMU to convert the vehicle body attitude change into the attitude change matrix of each camera, and calculate the relative rotational disturbance change between adjacent cameras. Step S202: Extract the camera attitude quaternion and convert it into pitch and roll angles in Euler angles. Use a second-order Butterworth filter to dynamically adjust the passband according to the real-time main frequency of the IMU to extract the vibration angle. Calculate the short-time vibration energy and determine the adaptive compensation gain to obtain the dynamic compensation attitude. Combine the camera intrinsic parameters to convert it into pixel displacement prediction values ​​and construct the prediction homography matrix. Step S203: Pass the pixel displacement prediction value and the prediction homography matrix to step S30, and pass the relative rotation perturbation change amount to step S40.

[0009] Preferably, in step S202, the adaptive compensation gain is limited to the range of [0.8, 1.5], exhibits a linear positive correlation with short-time vibration energy, has a base coefficient of 0.8, and a slope coefficient of k, where k = Furthermore, the slope coefficient is finely adjusted in real time by a fuzzy controller based on vehicle speed and suspension travel sensor data.

[0010] Preferably, in step S30, based on the multi-channel wide-angle camera images and the pixel displacement prediction values ​​and prediction homography matrix, an adaptive image stabilization transformation mechanism combining sparse optical flow feature tracking and IMU joint constraints is used to perform the image stabilization task, specifically including: Step S301: For each camera, extract FAST corner points from the previous frame after shaking, use the pixel displacement prediction value as the initial optical flow, use KLT pyramid optical flow to track in the current frame, and perform forward and backward error detection and consistency judgment with IMU prediction based on the predicted displacement constraint search radius, eliminate non-rigid body feature points or reduce weight, and obtain a set of effective feature point pairs. Step S302: Construct the optimization objective function for the joint constraints of IMU and sparse optical flow, and solve for the optimal image stabilization transformation matrix. The calculation formula is as follows:

[0011] in This is a Tukey robust kernel function used to suppress outlier effects, e i Let w be the reprojection error of the i-th feature point. IMU H is the adaptive weight for the IMU prediction term. IMU The output I of step S20 MU Predicting the homography matrix, w reg =0.5 is the weight of the regularization term. This is a perspective distortion penalty term used to limit excessive deformation, with adaptive weight w. IMU The optimal image stabilization transformation matrix H is obtained by dynamically adjusting the difference between the homography matrix estimated by the number of effective inliers and the homography matrix predicted by the IMU based on the pure optical flow. ; Step S303: Utilize the image stabilization transformation matrix H Image stabilization processing is performed on the images from multiple wide-angle cameras to obtain independent stabilized images from each camera, which are then output to step S40.

[0012] Preferably, in step S40, based on the independently shaken images from each camera and the relative rotational disturbance between adjacent cameras, a panoramic stitching task is performed using a dynamic compensation and spatiotemporal consistency fusion mechanism for surround-view stitching seams, specifically including: Step S401: Project each shaking image onto a unified top-down or cylindrical panoramic coordinate system according to the pre-calibrated panoramic projection table, and calculate the vibration offset field of the overlapping pixels using the relative rotation disturbance change and the preset ground plane distance. Step S402: Construct a joint cost function that considers vibration offset, and use dynamic programming to search for the optimal joint line. The calculation formula is as follows:

[0013] Where M is the total number of pixels of the seam line, and seam is the seam line image. , Let p be the grayscale values ​​of the i-th and j-th cameras at any pixel point p, respectively. Let be the grayscale value of the j-th adjacent camera at the corrected position, and D(p) be the vibration offset field calculated in step S401. Let λ be the gradient of the vibration offset field, and λ = 0.2 be the weight of the smoothing term, used to ensure the continuity of the seam line. The L2 norm squared is the gradient of the vibration migration field; a time-domain inertial constraint is applied to the optimal seam line obtained by the search, and adaptive fusion weights are calculated and time-domain IIR low-pass filtering is performed. Step S403: When the average offset within any triangular mesh of the overlapping area exceeds 35% of the width of the overlapping band and is greater than 30 pixels, discard the dynamic adjustment of the seam in the current frame, backtrack to the effective seam of the previous frame, and generate a panoramic image by extended Laplacian fusion; otherwise, generate a panoramic image by fusion of the current seam and pass the panoramic image to step S50.

[0014] Preferably, in step S50, adaptive distortion suppression and image output based on a human spatial perception model are performed on the panoramic image, specifically including: Step S501: Divide the panoramic image into a 32×32 grid, calculate the second-order Jacobian matrix of each grid node from the original camera coordinate system to the final image, and extract the local scaling anisotropy metric, which is the ratio of the maximum singular value to the minimum singular value of the Jacobian matrix. Step S502: When the local scaling anisotropy metric is greater than 1.08, perform similarity transformation interpolation on the local area, replace the local homography of the original mesh deformation with the similarity transformation of the nearest neighbor rotation plus equidistant scaling, perform gradient blending through radial blending weights, and force a rigid body transformation approximation on the lower 1 / 3 area of ​​the image. Step S503: Before rendering output, adjust the relevant parameters according to the anti-distortion configuration file of the remote control terminal display device to make the output screen adapt to the optical characteristics of the display and output the final screen.

[0015] The present invention also provides a remote video processing system based on an autonomous driving vehicle platform, comprising: Data synchronization and alignment module: used to acquire images from multiple wide-angle cameras and a set of six-axis inertial measurement unit (IMU) data. Based on timestamp adaptive interpolation alignment and adaptive low-pass filtering, it uses a cubic spline interpolation resampling mechanism to perform multi-source sensor data synchronization and alignment tasks, and outputs aligned IMU data and images from multiple wide-angle cameras. Attitude prediction and vibration separation module: Based on the aligned IMU data, it performs camera attitude prediction and vibration compensation tasks by using IMU pre-integration and high-frequency prediction of camera attitude and vibration mode separation mechanism, and outputs pixel displacement prediction value, prediction homography matrix and relative rotational perturbation between adjacent cameras. Image Stabilization Transformation Module: Based on the multi-channel wide-angle camera images and the pixel displacement prediction value and prediction homography matrix, it uses an adaptive image stabilization transformation mechanism with sparse optical flow feature tracking and IMU joint constraints to perform image stabilization tasks and output independent stabilized images for each camera. The stitching and fusion module is used to perform panoramic stitching tasks based on the independent shaking images of each camera and the relative rotation disturbance between adjacent cameras, and adopts a dynamic compensation and spatiotemporal consistency fusion mechanism for surround view stitching seams to output a surround view panoramic image. Distortion suppression and output module: Used to perform adaptive distortion suppression and image output based on the surround panoramic image and output the final image.

[0016] The present invention also provides a remote video processing device based on an autonomous driving vehicle platform. The remote video processing device based on the autonomous driving vehicle platform includes: a memory, a processor, and a remote video processing program based on the autonomous driving vehicle platform stored in the memory and executable on the processor. When the remote video processing program based on the autonomous driving vehicle platform is executed by the processor, it implements the above-described method.

[0017] The present invention also provides a computer program product, the computer program product including a remote video image processing program based on an autonomous driving vehicle platform, the remote video image processing program based on the autonomous driving vehicle platform implementing the above method when executed by a processor.

[0018] The beneficial effects of this invention are as follows: By using adaptive interpolation alignment, adaptive low-pass filtering, and cubic spline interpolation resampling, high-precision time synchronization of multi-source sensor data can be achieved, providing a reliable data foundation for subsequent image stabilization and stitching, and reducing errors caused by time misalignment.

[0019] By using IMU pre-integration and high-frequency prediction of camera attitude, vibration mode separation, and adaptive image stabilization transformation with joint constraints of sparse optical flow and IMU, the shaking during vehicle movement is effectively removed while maintaining image details. Through dynamic compensation of surround view stitching seams and spatiotemporal consistency fusion, the continuity and stability of the panoramic image are achieved. Through adaptive distortion suppression based on the human spatial perception model, the output image is made more in line with human eye observation habits, improving the accuracy and efficiency of remote control. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the first embodiment of a remote video processing method based on an autonomous driving vehicle platform according to the present invention. Detailed Implementation

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0022] Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1: As Figure 1 The diagram shown is a flowchart of a first embodiment of a remote video processing method based on an autonomous driving vehicle platform according to the present invention.

[0024] In the first embodiment, the remote video processing method based on an autonomous driving vehicle platform includes: Step S10: Acquire images from multiple wide-angle cameras and a set of six-axis inertial measurement unit (IMU) data. Based on timestamp adaptive interpolation alignment and adaptive low-pass filtering, a cubic spline interpolation resampling mechanism is used to perform the multi-source sensor data synchronous acquisition and alignment task, and output the aligned IMU data and multiple wide-angle camera images. It should be noted that the multi-source sensor data synchronous acquisition and alignment refers to simultaneously accessing multiple wide-angle cameras and a six-axis IMU via an in-vehicle embedded platform, and using hardware triggering and a unified clock source to generate timestamps for each frame of image and each IMU data packet. Considering the uncertainty of actual transmission and sampling delays, an adaptive low-pass filter is used to preprocess the raw IMU data. The filter cutoff frequency is not a fixed value, but is dynamically calculated based on the current vehicle speed and steering angle to adapt to the noise characteristics under different motion states. Based on this, cubic spline interpolation resampling is performed on the filtered IMU angular velocity sequence to generate virtual IMU measurements corresponding to the exposure time of each frame of image, thereby obtaining a time-strictly aligned IMU data sequence and multi-source wide-angle camera images. This step provides a precise and synchronized basic input for subsequent camera pose prediction and image stabilization.

[0025] Understandably, this step, through the combination of adaptive filtering and spline interpolation, can maintain high-precision time alignment between IMU data and image data even when the noise spectrum changes due to vehicle speed changes and steering operations. This avoids ghosting or misalignment problems in subsequent image stabilization and stitching caused by sensor timestamp deviations, providing a reliable data source for the entire processing chain.

[0026] It should be understood that, compared to traditional methods that use low-pass filtering with a fixed cutoff frequency or simple linear interpolation alignment, the adaptive mechanism in this step can match the vehicle's motion state in real time, retain high-frequency attitude information when driving at high speeds, and enhance noise suppression when performing complex operations at low speeds. At the same time, cubic spline interpolation can more accurately reproduce the dynamic changes between sampling points than linear interpolation, reducing the impact of interpolation errors on subsequent integration and attitude prediction.

[0027] For example, when an autonomous vehicle makes a low-speed turn in a narrow passage, with a speed below 0.5 m / s and a steering angle greater than 15°, the vibration noise introduced by the suspension and steering system has specific low-frequency characteristics. Adaptive filtering determines the noise based on vehicle speed and steering angle, and adds a correction term proportional to the standard deviation of IMU acceleration to dynamically reduce the cutoff frequency to filter out this noise. At the same time, cubic spline interpolation accurately captures the angular velocity changes during the steering process, ensuring that the aligned IMU data can truly reflect the subtle changes in vehicle attitude, thus preventing overcompensation or undercompensation due to data distortion in subsequent image stabilization.

[0028] Step S20: Based on the aligned IMU data, the camera attitude prediction and vibration compensation tasks are performed using IMU pre-integration and high-frequency prediction of camera attitude and vibration mode separation mechanism, and the pixel displacement prediction value, prediction homography matrix and relative rotation perturbation between adjacent cameras are output. It should be noted that in the camera attitude prediction and vibration compensation task, the median integral is first used to pre-integrate the IMU angular velocity and acceleration between adjacent image moments to obtain the vehicle's rotation increment and position increment. Combined with pre-calibrated extrinsic parameters of each camera and IMU, the vehicle attitude change is converted into an attitude change matrix for each camera, while simultaneously calculating the relative rotational perturbation change between adjacent cameras. Subsequently, the pitch and roll angles are extracted from the camera attitude quaternions, and vibration angular components are extracted using a second-order Butterworth filter, where the filter's passband is dynamically adjusted according to the IMU's real-time master frequency. The adaptive compensation gain is determined by calculating short-time vibration energy, and the dynamically compensated attitude is obtained after compensating for the vibration angles. Then, the pixel displacement prediction value and predicted homography matrix are calculated using the camera intrinsic parameters. Finally, the pixel displacement prediction value, predicted homography matrix, and relative rotational perturbation between adjacent cameras are output, which are used for image stabilization and panoramic stitching, respectively.

[0029] Understandably, this step, through IMU pre-integration and high-frequency vibration mode separation, can predict in real time the high-frequency changes in camera posture caused by vehicle vibration, and provide them to the image stabilization module in the form of pixel displacement and homography matrix. This allows the image stabilization transformation to follow the vibration trend in advance, reducing the computational delay and matching error of the image stabilization process. At the same time, it provides quantitative information on the relative pose disturbance between cameras for panoramic stitching, which is helpful for subsequent dynamic seam adjustment.

[0030] It should be understood that in traditional methods, camera pose prediction often relies solely on simple integration by the IMU or pure visual odometry. The former is prone to accumulating errors and cannot separate vibration from subject motion, while the latter is unstable when features are sparse or illumination changes. This step, by explicitly separating vibration modes and using pre-integration to ensure relative pose accuracy between adjacent frames, ensures that high-frequency vibration compensation and low-frequency motion tracking do not interfere with each other, thereby improving the accuracy and robustness of prediction.

[0031] For example, when an autonomous vehicle is driving on a continuously bumpy gravel road, the IMU data contains a mixture of normal low-frequency pitch from the vehicle body and high-frequency vibrations from the suspension. A second-order Butterworth filter dynamically sets the passband based on the current IMU master frequency (e.g., 20Hz), effectively extracting the high-frequency vibration components of 8-15Hz. Short-time vibration energy analysis indicates that the current energy is high, and the adaptive compensation gain is adjusted to around 1.2, thereby generating a pixel displacement prediction value that can offset this vibration. Simultaneously, the change in relative rotational disturbance reflects the additional torsion between the left front camera and the front front camera caused by micro-deformation of the chassis, providing key compensation parameters for subsequent seam processing.

[0032] Step S30: Based on the multi-channel wide-angle camera images and the pixel displacement prediction value and prediction homography matrix, an adaptive image stabilization transformation mechanism with sparse optical flow feature tracking and IMU joint constraints is used to perform the image stabilization task and output independent stabilized images of each camera. It should be noted that the specific process of the adaptive image stabilization transformation mechanism is as follows: First, FAST corner points are extracted from the previous frame's stabilized image. The pixel displacement prediction value output by S20 is used as the initial optical flow. KLT optical flow tracking is performed at each layer of the pyramid, and the search radius is constrained according to the predicted displacement to reduce mismatches. Through forward and backward error detection and consistency judgment with IMU prediction, non-rigid motion feature points are eliminated or their weights are reduced, retaining the effective feature point set. Then, a joint optimization objective function is constructed, which combines a robust kernel function term based on reprojection error, an IMU predicted homography matrix constraint term, and a perspective distortion penalty term. The adaptive weights of the IMU constraint term are dynamically adjusted based on the number of effective interior points and the difference between the pure optical flow estimated homography matrix and the IMU predicted homography matrix. Solving this optimization problem yields the optimal image stabilization transformation matrix. Finally, this matrix is ​​applied to transform the image, outputting an independently stabilized image.

[0033] Understandably, this step, through the joint constraint of sparse optical flow and IMU, can provide reliable transformation priors through IMU prediction when the camera shakes violently or there are insufficient feature points, thus avoiding the failure of the pure optical flow method. In normal scenarios, optical flow features can provide more refined transformation details, and adaptive weights balance the relationship between the two, thereby ensuring the stability and accuracy of image stabilization under various working conditions.

[0034] It should be understood that, compared to stabilization methods that rely solely on optical flow and fail to track when faced with blurred or large displacements, or stabilization methods that rely solely on IMU and lack adaptability to scene content, the joint constraint mechanism in this step effectively integrates the advantages of the two modes through dynamic weights. It can suppress the interference of non-rigid moving objects on image stabilization, while avoiding excessive perspective distortion and ensuring the naturalness of the output image.

[0035] For example, in remote control of autonomous driving, other moving vehicles may appear in the frame, and their movement is not considered camera jitter. After FAST corner tracking, consistency judgment marks these feature points located on moving vehicles as outliers and reduces their weight. Simultaneously, if the number of effective inliers in the current frame is large and the difference between the pure optical flow estimation transformation and the IMU prediction is small, adaptive weights favor the optical flow result to preserve details; if dense fog or low-texture road sections are encountered, the number of effective inliers sharply decreases, and the IMU constraint weights automatically increase to a dominant position to prevent image stabilization failure. The final solved matrix eliminates jitter while avoiding local distortions of moving vehicles in the image.

[0036] Step S40: Based on the independent shaking images of each camera and the relative rotation disturbance between adjacent cameras, a panoramic stitching task is performed using a dynamic compensation and spatiotemporal consistency fusion mechanism for surround view stitching seams, and a surround view panoramic image is output. It should be noted that in the panoramic stitching task, firstly, based on a pre-calibrated surround projection table, each independent shaky image is projected onto a unified top-down or cylindrical panoramic coordinate system. Using the relative rotational perturbation change between adjacent cameras output by S20 and the preset ground plane distance, the vibration offset field of pixels in the overlapping area is calculated. Then, a seam cost function considering this vibration offset is constructed. This function includes a pixel grayscale difference term and a vibration offset field gradient smoothing term, and the optimal seam line is searched through dynamic programming. To maintain temporal stability, a temporal inertial constraint is applied to the current seam line to avoid abrupt changes in seam positions between adjacent frames. Simultaneously, adaptive fusion weights are calculated and temporal IIR low-pass filtering is performed. If the average offset within any triangulated grid exceeds 35% of the overlap band width and is greater than 30 pixels, the current offset is deemed too large, dynamic adjustment is discarded, and the frame is reverted to the previous effective seam, and extended Laplacian fusion is used to generate the panoramic image; otherwise, the currently calculated seam and fusion weights are used to generate the surround panoramic image.

[0037] Understandably, this step, by introducing a vibration offset field for dynamic seam compensation, can correct pixel misalignment in overlapping areas caused by vehicle vibration in real time. Combined with spatiotemporal consistency constraints, this ensures that the seams of the panoramic image are smooth and natural in dynamic environments, without obvious breaks or flickering. The backoff mechanism ensures that the panoramic image will not fail to stitch under extreme vibration, improving overall reliability.

[0038] It should be understood that traditional surround view stitching often uses fixed seams calibrated offline, which cannot adapt to relative pose changes caused by vehicle vibration and chassis deformation during driving, and misalignment and ghosting are prone to occur at the seams. This step calculates the vibration offset field and performs dynamic seam optimization, which can actively follow the relative rotational disturbances between cameras. At the same time, temporal filtering and inertial constraints reduce seam jitter, making the panoramic image more suitable for continuous observation by remote control.

[0039] For example, when an autonomous vehicle traverses an uneven road surface at high speed, the relative rotational perturbation between adjacent cameras causes a shift of up to approximately 25 pixels in the overlapping area. The dynamic programming algorithm incorporates this shift into the cost function, adjusting the optimal seam line towards the area with the smaller shift to avoid cutting into high-contrast objects. Simultaneously, temporal IIR filtering limits the inter-frame movement of the seam line to within a few pixels, making the seams in the panoramic image less noticeable to remote operators. When the vehicle passes over a deep pothole, causing an instantaneous shift exceeding 40 pixels, a backtracking mechanism is triggered, reusing the effective seam from the previous frame. The image only experiences a brief blur rather than tearing before resuming normal stitching.

[0040] Step S50: Based on the panoramic image, perform adaptive distortion suppression and image output based on the human spatial perception model, and output the final image.

[0041] It should be noted that in the adaptive distortion suppression and image output process, the panoramic image is first divided into a 32×32 grid. The second-order Jacobian matrix of each grid node from the original camera coordinate system to the final image is calculated, and the local scaling anisotropy metric, i.e., the ratio of the maximum singular value to the minimum singular value, is extracted. When this ratio exceeds 1.08, similarity transformation interpolation is performed on the local region. Specifically, the local homography of the original grid deformation is replaced with a similarity transformation of nearest neighbor rotation plus equidistant scaling, and then gradually blended with the surrounding region through radial mixing weights. For the lower 1 / 3 of the image, a rigid body transformation approximation is forcibly applied to maintain the shape. Finally, the relevant parameters are adjusted according to the anti-distortion configuration file of the remote control display device to adapt the output image to the optical characteristics of the display, generating the final image.

[0042] Understandably, this step is based on a human spatial perception model, which assesses the human eye’s sensitivity to distortion by measuring anisotropy through local scaling and applies similarity or rigid body transformation constraints to areas where distortion exceeds limits. This reduces the interference of image stretching and compression on the human eye’s spatial judgment while maintaining a panoramic field of view, making the sense of distance and angle obtained by remote operators closer to reality and helping to improve control accuracy.

[0043] It should be understood that in existing technologies, panoramic image output typically uses a uniform perspective or projection transformation, failing to consider the differences in human eye perception of distortion in different areas of the image. This results in near objects being stretched and distorted, and distant objects appearing excessively compressed, affecting the operator's judgment of the vehicle's surrounding environment. This step quantifies and suppresses local anisotropic distortion and applies rigid body constraints to the near-vehicle area at the bottom of the image, making the generated surround view image more consistent with human visual habits and reducing the risk of misjudgment caused by image distortion.

[0044] For example, in a remote parking scenario, if the parking space markings and guardrails displayed at the bottom of the panoramic view are stretched and distorted, the operator may misjudge the distance. This step detects that the anisotropy metric of the mesh in this area reaches 1.12, exceeding the threshold of 1.08. Therefore, it automatically replaces the local homography with a similarity transformation, maintaining the straightness and angle of the markings, while using a gradual radial weight transition to avoid abrupt changes. The upper distant view area retains its original projection to preserve a large field of view. The final output is an image adapted to the display's optical characteristics, allowing the operator to clearly judge the lateral and longitudinal distances between the vehicle and surrounding objects.

[0045] Example 2: Furthermore, the present invention provides a remote video processing system based on an autonomous driving vehicle platform, employing a remote video processing method based on an autonomous driving vehicle platform as described in the above embodiments, which can solve a technical problem related to remote video processing based on an autonomous driving vehicle platform. The beneficial effects of the remote video processing system based on an autonomous driving vehicle platform provided by the present invention are the same as those of the remote video processing method based on an autonomous driving vehicle platform provided in the above embodiments, and other technical features of the remote video processing system based on an autonomous driving vehicle platform are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0046] Example 3: This invention provides a remote video processing device based on an autonomous driving platform. The device includes: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed, enable the at least one processor to perform the remote video processing method based on an autonomous driving platform described in Example 1. The remote video processing device based on an autonomous driving platform in this invention can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. This remote video processing device based on an autonomous driving platform is merely an example and should not limit the functionality or scope of the invention. A remote video processing device based on an autonomous driving platform may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory or a program loaded from a storage device into a random access memory. The random access memory also stores various programs and data required for the operation of the remote video processing device based on the autonomous driving platform. The processing unit, the read-only memory, and the random access memory are interconnected via a bus. An I / O interface is also connected to the bus. Typically, the following systems can be connected to the I / O interface: input devices including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices including, for example, magnetic tapes, hard disks, etc.; and communication devices. The communication device allows the remote video processing device based on the autonomous driving platform to communicate wirelessly or wiredly with other devices to exchange data. While a remote video processing device based on an autonomous driving platform with various systems has been described, it should be understood that implementation or possession of all described systems is not required. It can be implemented alternatively or with more or fewer systems.

[0047] Example 4: This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the remote video image processing method based on an autonomous driving vehicle platform as described above. The computer program product provided by this invention can solve a technical problem related to remote video image processing based on an autonomous driving vehicle platform. Compared with the prior art, the beneficial effects of the computer program product provided by this invention are the same as those of the remote video image processing method based on an autonomous driving vehicle platform provided in the above embodiments, and will not be repeated here.

[0048] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a read-only memory. When the computer program is executed by a processing device, it performs the functions defined in the methods of the embodiments disclosed in this invention.

[0049] It should be understood that the various parts disclosed in this invention can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0050] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A method for remote video image processing based on an autonomous driving vehicle platform, characterized in that, The methods include: Step S10: Acquire images from multiple wide-angle cameras and a set of six-axis inertial measurement unit (IMU) data. Based on timestamp adaptive interpolation alignment and adaptive low-pass filtering, a cubic spline interpolation resampling mechanism is used to perform multi-source sensor data synchronous acquisition and alignment tasks, and output the aligned IMU data. Step S20: Based on the aligned IMU data, the camera attitude prediction and vibration compensation tasks are performed using IMU pre-integration and high-frequency prediction of camera attitude and vibration mode separation mechanism, and the pixel displacement prediction value, prediction homography matrix and relative rotation perturbation between adjacent cameras are output. Step S30: Based on the multi-channel wide-angle camera images and the pixel displacement prediction value and prediction homography matrix, an adaptive image stabilization transformation mechanism with sparse optical flow feature tracking and IMU joint constraints is used to perform the image stabilization task and output independent stabilized images of each camera. Step S40: Based on the independent shaking images of each camera and the relative rotation disturbance between adjacent cameras, a panoramic stitching task is performed using a dynamic compensation and spatiotemporal consistency fusion mechanism for surround view stitching seams, and a surround view panoramic image is output. Step S50: Based on the panoramic image, perform adaptive distortion suppression and image output based on the human spatial perception model, and output the final image.

2. The remote video processing method based on an autonomous driving platform according to claim 1, characterized in that, In step S10, based on timestamp adaptive interpolation alignment and adaptive low-pass filtering, a cubic spline interpolation resampling mechanism is used to perform the multi-source sensor data synchronous acquisition and alignment task, specifically including: Step S101: Multiple wide-angle cameras and a six-axis inertial measurement unit (IMU) are simultaneously connected to the vehicle-mounted embedded platform. Each wide-angle camera is synchronously exposed by an external trigger signal and outputs an image stream with a resolution of 1280×720 at a frame rate of 30fps. A precise timestamp is generated by hardware at the center of each frame. The IMU outputs three-axis angular velocity and three-axis acceleration at a frequency of 200Hz. Each data packet carries a timestamp generated by the same clock source. Step S102: Send the image stream and IMU data stream into the circular buffer. Perform adaptive low-pass filtering preprocessing on the raw IMU data. The filter cutoff frequency is dynamically adjusted based on the current vehicle speed and steering angle: when the vehicle speed is not lower than 0.5 m / s or the steering angle is not greater than 15°, the cutoff frequency increases linearly with the vehicle speed, with an upper limit of 40 Hz; when the vehicle speed is lower than 0.5 m / s and the steering angle is greater than 15°, a correction term proportional to the standard deviation of the IMU acceleration is superimposed on top of the above, with a correction coefficient of 2 Hz·m. 2 / s; Step S103: Perform cubic spline interpolation resampling on the filtered IMU angular velocity sequence to obtain the IMU virtual measurement value corresponding to the exposure time of each frame of the image, thereby obtaining the precisely aligned timestamp sequence, and outputting the aligned IMU data and multi-channel wide-angle camera images.

3. The remote video processing method based on an autonomous driving platform according to claim 2, characterized in that, Based on the aligned IMU data, the camera attitude prediction and vibration compensation tasks are performed using IMU pre-integration and a high-frequency prediction and vibration mode separation mechanism. Specifically, this includes: Step S201: Receive the aligned IMU angular velocity and acceleration, perform median integral pre-integration on the angular velocity and acceleration between adjacent image moments to obtain the rotation increment and position increment of the vehicle body coordinate system, use the pre-calibrated extrinsic parameters between each camera and the IMU to convert the vehicle body attitude change into the attitude change matrix of each camera, and calculate the relative rotational disturbance change between adjacent cameras. Step S202: Extract the camera attitude quaternion and convert it into pitch and roll angles in Euler angles. Use a second-order Butterworth filter to dynamically adjust the passband according to the real-time main frequency of the IMU to extract the vibration angle. Calculate the short-time vibration energy and determine the adaptive compensation gain to obtain the dynamic compensation attitude. Combine the camera intrinsic parameters to convert it into pixel displacement prediction values ​​and construct the prediction homography matrix.

4. The remote video processing method based on an autonomous driving platform according to claim 3, characterized in that, In step S202, the adaptive compensation gain is limited to the range of [0.8, 1.5], and is linearly positively correlated with the short-time vibration energy. The basic coefficient is 0.8, and the slope coefficient is k, where k = Furthermore, the slope coefficient is adjusted in real time by a fuzzy controller based on data from vehicle speed and suspension travel sensors.

5. The remote video processing method based on an autonomous driving platform according to claim 1, characterized in that, In step S30, based on the multi-channel wide-angle camera images and the predicted pixel displacement values ​​and the predicted homography matrix, an adaptive image stabilization transformation mechanism combining sparse optical flow feature tracking and IMU joint constraints is used to perform the image stabilization task, specifically including: Step S301: For each camera, extract FAST corner points from the previous frame after shaking, use the pixel displacement prediction value as the initial optical flow, use KLT pyramid optical flow to track in the current frame, and perform forward and backward error detection and consistency judgment with IMU prediction based on the predicted displacement constraint search radius, eliminate non-rigid body feature points or reduce weight, and obtain a set of effective feature point pairs. Step S302: Construct the optimization objective function for the joint constraints of IMU and sparse optical flow, and solve for the optimal image stabilization transformation matrix. The calculation formula is as follows: in This is a Tukey robust kernel function used to suppress outlier effects, e i Let w be the reprojection error of the i-th feature point. IMU H is the adaptive weight for the IMU prediction term. IMU The output I of step S20 MU Predicting the homography matrix, w reg For the weight of the regularization term, This is a perspective distortion penalty term used to limit excessive deformation, with adaptive weight w. IMU The optimal image stabilization transformation matrix H is obtained by dynamically adjusting the difference between the homography matrix estimated by the number of effective inliers and the homography matrix predicted by the IMU based on the pure optical flow. ; Step S303: Utilize the image stabilization transformation matrix H Image stabilization is performed on images from multiple wide-angle cameras to obtain independent stabilized images from each camera.

6. The remote video processing method based on an autonomous driving platform according to claim 1, characterized in that, In step S40, based on the independently shaken images from each camera and the relative rotational disturbances between adjacent cameras, a panoramic stitching task is performed using a dynamic compensation and spatiotemporal consistency fusion mechanism for surround-view stitching seams. Specifically, this includes: Step S401: Project each shaking image onto a unified top-down or cylindrical panoramic coordinate system according to the pre-calibrated panoramic projection table, and calculate the vibration offset field of the overlapping pixels using the relative rotation disturbance change and the preset ground plane distance. Step S402: Construct a joint cost function that considers vibration offset, and use dynamic programming to search for the optimal joint line. The calculation formula is as follows: Where M is the total number of pixels of the seam line, and seam is the seam line image. , Let p be the grayscale values ​​of the i-th and j-th cameras at any pixel point p, respectively. Let be the grayscale value of the j-th adjacent camera at the corrected position, and D(p) be the vibration offset field calculated in step S401. Let λ be the gradient of the vibration offset field, and λ = 0.2 be the weight of the smoothing term, used to ensure the continuity of the seam line. The L2 norm squared is the gradient of the vibration migration field; a time-domain inertial constraint is applied to the optimal seam line obtained by the search, and adaptive fusion weights are calculated and time-domain IIR low-pass filtering is performed. Step S403: When the average offset within any triangular mesh of the overlapping area exceeds 35% of the width of the overlapping band and is greater than 30 pixels, discard the dynamic adjustment of the seam in the current frame, revert to the effective seam of the previous frame, and generate a panoramic image using extended Laplacian fusion; otherwise, generate a panoramic image using the current seam fusion.

7. The remote video image processing method based on an autonomous driving vehicle platform according to claim 1, characterized in that, In step S50, adaptive distortion suppression and image output based on a human spatial perception model are performed on the panoramic image, specifically including: Step S501: Divide the panoramic image into a 32×32 grid, calculate the second-order Jacobian matrix of each grid node from the original camera coordinate system to the final image, and extract the local scaling anisotropy metric, which is the ratio of the maximum singular value to the minimum singular value of the Jacobian matrix. Step S502: When the local scaling anisotropy metric is greater than 1.08, perform similarity transformation interpolation on the local area, replace the local homography of the original mesh deformation with the similarity transformation of the nearest neighbor rotation plus equidistant scaling, perform gradient blending through radial blending weights, and force a rigid body transformation approximation on the lower 1 / 3 area of ​​the image. Step S503: Before rendering output, adjust the relevant parameters according to the anti-distortion configuration file of the remote control terminal display device to make the output screen adapt to the optical characteristics of the display and output the final screen.

8. A remote video processing system based on an autonomous driving vehicle platform, applied to the remote video processing method based on an autonomous driving vehicle platform as described in any one of claims 1 to 7, characterized in that, include: Data synchronization and alignment module: used to acquire images from multiple wide-angle cameras and a set of six-axis inertial measurement unit (IMU) data. Based on timestamp adaptive interpolation alignment and adaptive low-pass filtering, it uses a cubic spline interpolation resampling mechanism to perform multi-source sensor data synchronization and alignment tasks, and outputs aligned IMU data and images from multiple wide-angle cameras. Attitude prediction and vibration separation module: Based on the aligned IMU data, it performs camera attitude prediction and vibration compensation tasks by using IMU pre-integration and high-frequency prediction of camera attitude and vibration mode separation mechanism, and outputs pixel displacement prediction value, prediction homography matrix and relative rotational perturbation between adjacent cameras. Image Stabilization Transformation Module: Based on the multi-channel wide-angle camera images and the pixel displacement prediction value and prediction homography matrix, it uses an adaptive image stabilization transformation mechanism with sparse optical flow feature tracking and IMU joint constraints to perform image stabilization tasks and output independent stabilized images for each camera. The stitching and fusion module is used to perform panoramic stitching tasks based on the independent shaking images of each camera and the relative rotation disturbance between adjacent cameras, and adopts a dynamic compensation and spatiotemporal consistency fusion mechanism for surround view stitching seams to output a surround view panoramic image. Distortion suppression and output module: Used to perform adaptive distortion suppression and image output based on the surround panoramic image and output the final image.

9. A remote video processing device based on an autonomous driving platform, characterized in that, The remote video processing device based on the autonomous driving vehicle platform includes: a memory, a processor, and a remote video processing program based on the autonomous driving vehicle platform stored in the memory and executable on the processor. When the remote video processing program based on the autonomous driving vehicle platform is executed by the processor, it implements a remote video processing method based on the autonomous driving vehicle platform according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a remote video processing program based on an autonomous driving vehicle platform. When the remote video processing program based on the autonomous driving vehicle platform is executed by a processor, it implements a remote video processing method based on an autonomous driving vehicle platform as described in any one of claims 1 to 7.

Images