Unmanned vehicle large-view-field imaging video anti-shake system and method

Abstract

The invention provides an unmanned vehicle large-view-field imaging video anti-shake system and method. The system is composed of an unmanned vehicle, a multi-aperture camera, an acquisition card, a calculation card and a display. The multi-aperture camera is carried on the unmanned vehicle, the acquisition card is connected with the multi-aperture camera, a multi-aperture video acquisition algorithm is deployed in the acquisition card, and a real-time video stitching algorithm and a large-view-field video anti-shake algorithm are deployed in the calculation card. A video acquisition module, a real-time image splicing module and a large-view-field video anti-shake module are included in the unmanned vehicle large-view-field video anti-shake method. The system and the method specially carry out jitter smoothing processing on the large-view-field video collected by the unmanned vehicle, and compared with a traditional small-view-field video anti-jitter method, the stability of the large-view-field video is improved, and information loss of video images is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a large-field-of-view imaging video anti-shake system and method for an unmanned vehicle. Background Art

[0002] During the movement of the unmanned vehicle, factors such as uneven ground, poor road conditions, and the vibration of the vehicle itself will cause the image captured by the camera to shake, making the image unstable. This image instability makes it difficult for the unmanned vehicle to accurately identify and understand the surrounding environment, thereby reducing its ability to perceive roads and obstacles. Secondly, image jitter will also affect the path planning of the unmanned vehicle, resulting in unstable or even deviations in the driving route, increasing the risk and uncertainty of driving. Therefore, solving the problem of image jitter during the movement of the unmanned vehicle and improving the stability and quality of the image are of great significance to improving the visual perception ability of the car and the accuracy of path planning. The use of an effective video anti-shake algorithm can effectively suppress image jitter and ensure that the car obtains clear and stable visual information during driving, thereby improving its autonomous navigation and obstacle avoidance capabilities and ensuring driving safety and stability.

[0003] Unmanned vehicles usually need to obtain extensive environmental information when implementing functions such as navigation and obstacle avoidance, so it is particularly important to have a large field of view. This allows the car to perceive the surrounding environment more comprehensively and improve its ability to identify road conditions and obstacles. In order to meet this demand, using multiple cameras to expand the field of view has become a common choice for unmanned vehicles. Multiple cameras combined into a large field of view image can cover a wider area, thereby providing more environmental information for the car. By analyzing large field of view video images, unmanned vehicles can detect road conditions, identify traffic signs, and find obstacles in real time, which helps to achieve accurate navigation and effective obstacle avoidance. However, if this large field of view video is severely jittery, it will affect the video quality and greatly affect the realization of the unmanned vehicle function. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a large-field-of-view imaging video stabilization system and method for unmanned vehicles. By combining a deep learning network and using a large-field-of-view video data set for training, compared with traditional small-field-of-view video stabilization methods, the present invention improves the stability of large-field-of-view videos and reduces information loss in video images.

[0005] To achieve the above-mentioned object, the present invention adopts the following technical scheme: an unmanned vehicle large-field-of-view imaging video anti-shake system, comprising an unmanned vehicle, a multi-aperture camera, a capture card, a computing card, a display, a video capture module, a real-time image stitching module and a large-field-of-view video anti-shake module; The unmanned vehicle is equipped with a computing card, an acquisition card and a multi-aperture camera. The unmanned vehicle will shake during the movement, which is the source of the jitter of the unmanned vehicle's large field of view video; The video acquisition module includes an acquisition card, a multi-aperture camera, a multi-threaded parallel acquisition module and a frame synchronization control module; the multi-aperture camera is used for multi-aperture video; the acquisition card is connected to the multi-aperture camera to deploy a multi-aperture video acquisition algorithm; the multi-threaded parallel acquisition module is used to control the operation of the multi-aperture camera; the frame synchronization control module is used to control the multi-aperture camera to achieve synchronous acquisition; The real-time image stitching module is connected to the video acquisition module and is used to stitch the received multi-aperture images into a complete large-field-of-view image; The large-field-of-view video stabilization module is connected to the real-time image stitching module; the large-field-of-view video stabilization module includes a feature point detection module, a key point motion estimation module, a grid motion propagation module and a motion smoothing module, which are used to de-shake the stitched large-field-of-view video and output a large-field-of-view stabilized video; The computing card is used to deploy real-time stitching algorithms and large-field-of-view video stabilization algorithms, and is responsible for executing corresponding computing tasks; the display is used to present stabilized large-field-of-view videos.

[0006] In a preferred embodiment, a multi-aperture camera is fixed on the unmanned vehicle at a certain angle to expand the field of view, an acquisition card is connected to the multi-aperture camera, and a video acquisition algorithm is deployed to control the camera to perform multi-aperture video acquisition; a real-time image stitching module and a large-field-of-view video stabilization algorithm module are deployed on a computing card, which is responsible for stabilizing the received large-field-of-view video; a display is connected to the acquisition card to display the stabilized large-field-of-view video.

[0007] In a preferred embodiment, the video acquisition module includes a multi-aperture camera module, a multi-threaded parallel acquisition module, an acquisition card and a frame synchronization control module; the acquisition card is respectively connected to the multi-aperture camera module, the multi-threaded parallel acquisition module and the frame synchronization control module; the multi-threaded parallel acquisition module controls the transmission of multi-aperture images in a multi-threaded parallel manner; the frame synchronization control module uses a continuous soft trigger method to send signals to the multi-aperture camera and controls parameters such as the frame rate; the multi-threaded parallel acquisition module and the frame synchronization control module are used in combination to obtain synchronized multi-aperture images.

[0008] In a preferred embodiment, the real-time image stitching includes an image dedistortion module, an image registration module and an image transformation module; the image dedistortion module uses the calibrated camera internal parameters to dedistort the multi-aperture images respectively, and records the image dedistortion matrix at the same time; the image registration module is connected to the image dedistortion module, firstly, the image registration module receives a multi-aperture image with sufficient features, detects the feature information of the input image, uses the multi-view geometry calibration method to record the internal and external parameters of the camera, and calculates the transformation relationship parameters of the stitched image; the image transformation module is connected to the image registration module, uses the recorded image dedistortion parameters and image transformation parameters to distort the multi-aperture image, and fuses the transformed multi-aperture image into one image to obtain a large field of view image.

[0009] In a preferred embodiment, the large field of view video stabilization module includes a feature point detection module, a key point motion estimation module, a grid motion propagation module and a motion smoothing module; the feature point detection module detects the feature points of the input large field of view image; the key point motion estimation module is connected to the feature point detection module, and is used to screen the detected feature points, and obtain the key point motion vector by matching the feature points of the current frame with the feature points of the past frame; the grid motion propagation module is connected to the key point motion estimation module, and firstly divides the large field of view video frame into M×N grids, and then propagates the key point motion data into grid vertex motion data according to a certain weight, and the weight is related to the distance between the key point and the grid vertex, and finally uses multiple one-dimensional convolutions to extract motion features, fuse information and generate an optimized grid motion trajectory; the motion smoothing module is connected to the large field of view video stabilization module, and is used to smooth the calculation to obtain the grid vertex motion, perform large field of view image transformation according to the offset grid motion vector, and output a large field of view stable video; The grid motion propagation module and the motion smoothing module first use the system's video acquisition module and real-time image stitching module to produce a large-field-of-view video data set, which is an unstable large-field-of-view video data set collected when the unmanned vehicle shakes during movement; the data set is respectively input into the grid motion propagation module and the motion smoothing module for training, thereby improving the robustness of the anti-shake algorithm in smoothing complex large-field-of-view shakes.

[0010] The present invention also provides a method for stabilizing video of unmanned vehicle with large field of view imaging, which adopts the unmanned vehicle with large field of view imaging video stabilization system, and comprises the following steps: Step S1: construct a large field of view video dataset based on the unmanned vehicle, including image data of different scenes; Step S2: using the constructed data set to train the grid motion propagation module and the motion smoothing module, to obtain a grid motion propagation module and a motion smoothing module that are robust to image stabilization of a large field of view of an unmanned vehicle; Step S3: Acquire a calibration image for the real-time image stitching module, the image must contain enough feature points to ensure the accuracy of the image transformation parameters, and calculate the image transformation parameters through the image; Step S4: driving the unmanned vehicle. The unmanned vehicle may vibrate due to some uneven roads or the vehicle itself. Step S5: The video acquisition module can collect a jittery multi-aperture image through a multi-aperture camera mounted on the unmanned vehicle due to the influence of the jittery posture information of the unmanned vehicle; Step S6: inputting the jittered multi-aperture image into a real-time image stitching module, transforming the multi-aperture image by using pre-calculated image transformation parameters to obtain a large field of view image; Step S7: The large-field-of-view video stabilization module receives the jittered large-field-of-view image, performs feature point detection, key point motion estimation, grid motion propagation and motion smoothing on the jittered image, and finally outputs a stable large-field-of-view video.

[0011] Compared with the prior art, the present invention has the following beneficial effects: the present invention performs jitter smoothing processing on the large field of view video collected by the unmanned vehicle, which improves the stability of the large field of view video and reduces the information loss of the video image compared with the traditional small field of view video anti-shake method. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Schematic diagram of a method and system for stabilizing video of unmanned vehicle with large field of view according to a preferred embodiment of the present invention; Figure 2 is a schematic diagram of the multi-aperture camera structure of a preferred embodiment of the present invention; Figure 3 It is a schematic diagram of the multi-aperture video acquisition algorithm flow in a preferred embodiment of the present invention; Figure 4 It is a flowchart of a real-time image stitching algorithm according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0013] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0014] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0015] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0016] See also Figure 1-4 , a large-field-of-view imaging video anti-shake system for an unmanned vehicle, specifically as follows: a large-field-of-view imaging video anti-shake method and system based on an unmanned vehicle, characterized in that it includes an unmanned vehicle, a multi-aperture camera, a capture card, a computing card, a display, a video capture module, a real-time image stitching module and a large-field-of-view video anti-shake module; The unmanned vehicle is equipped with a computing card, an acquisition card, and a multi-aperture camera. It will shake while moving, which is the source of the jitter of the unmanned vehicle's large field of view video. The video acquisition module includes an acquisition card, a multi-aperture camera, a multi-threaded parallel acquisition module and a frame synchronization control module. The multi-aperture camera is used for multi-aperture video; the acquisition card is connected to the multi-aperture camera to deploy the multi-aperture video acquisition algorithm; the multi-threaded parallel acquisition module is used to control the multi-aperture camera to work; the frame synchronization control module is used to control the multi-aperture camera to achieve synchronous acquisition, providing the necessary conditions for stitching; A real-time image stitching module, connected to the video acquisition module, is used to stitch the received multi-aperture images into a complete large-field-of-view image; The large-field-of-view video anti-shake module is connected to the real-time image stitching module. It includes a feature point detection module, a key point motion estimation module, a grid motion propagation module, and a motion smoothing module, which are used to de-shake the stitched large-field-of-view video and output a large-field-of-view stabilized video; The computing card is used to deploy the real-time stitching algorithm and the large-field-of-view video stabilization algorithm, and is responsible for executing the corresponding computing tasks; the display is used to present the large-field-of-view video that has been stabilized.

[0017] The video acquisition module includes a multi-aperture camera module, a multi-threaded parallel acquisition module, an acquisition card and a frame synchronization control module. The multi-aperture camera uses an industrial USB camera with a resolution of 3920 × 2080 and a maximum adjustable frame rate of 60 FPS; the acquisition card uses Jeston Xavier NX; and the video acquisition algorithm is programmed using QT5.12.12.

[0018] The acquisition card is connected to the multi-aperture camera module, the multi-threaded parallel acquisition module and the frame synchronization control module respectively; the multi-threaded parallel acquisition module controls the transmission of the multi-aperture image in a multi-threaded parallel manner; the frame synchronization control module uses a continuous soft trigger method to send signals to the multi-aperture camera and controls parameters such as the frame rate at the same time; the multi-threaded parallel acquisition module and the frame synchronization control module are used in combination to obtain synchronized multi-aperture images, providing the necessary conditions for stitching.

[0019] Multi-aperture camera Figure 1 The acquisition card is fixed in a fixed way and installed on the unmanned vehicle. The acquisition card is connected to the multi-aperture camera by wire. The flow chart of the multi-aperture video acquisition algorithm is as follows: Figure 3 As shown, the specific steps include: Step SA1: Enumerate and configure camera devices: Scan all accessible camera devices and access them one by one. Set acquisition parameters for each camera, including exposure time, gain, soft trigger mode, etc., to ensure the quality and consistency of the acquired images; Step SA2: Independent thread trigger acquisition: Create an independent thread for each camera and continuously perform soft trigger operations in the thread. After each trigger, send a command to the camera to acquire a single frame image to ensure that the acquisition process is efficient and not affected by other cameras; Step SA3: Parallel callback image acquisition: Register callback functions in parallel in the independent threads of each camera. Whenever the camera completes image acquisition, the callback function will be triggered to obtain a frame of image data; Step SA4: Main thread image display: the acquired image is transmitted to the main thread, interacting with the interface controls to realize real-time output display of the image; Step SA5: Calculate and evaluate performance: record the time of each image update, calculate and analyze performance parameters such as frame rate to evaluate the real-time performance and efficiency of the system; Step SA6: Multithreaded image saving: Three independent threads are started, which are dedicated to saving multi-aperture images and supporting offline processing and subsequent analysis of images.

[0020] The unmanned vehicle large field of view imaging video stabilization method is characterized in that the real-time image stitching includes an image dedistortion module, an image registration module and an image transformation module. The image dedistortion module uses the calibrated camera internal parameters to dedistort the multi-aperture images separately, and records the image dedistortion matrix at the same time; the image registration module is connected to the image dedistortion module, and firstly the module receives a frame of multi-aperture image with sufficient features, detects the feature information of the input image, and uses the multi-view geometry calibration method to record the internal and external parameters of the camera, and calculates the transformation relationship parameters of the stitched image. The image transformation module is respectively connected to the image registration module, and uses the recorded image dedistortion parameters and image transformation parameters to perform a distortion transformation on the multi-aperture image, and fuses the transformed multi-aperture image into one image to obtain a large field of view image. The schematic diagram of the real-time image stitching process is shown as follows. Figure 4 As shown, the specific steps include: Step SB1: Use Zhang’s calibration method to accurately estimate the camera intrinsic parameters, and calculate the distortion correction matrix based on these parameters to ensure that the image distortion is effectively corrected; Step SB2: Use the video acquisition module to obtain a multi-aperture image containing enough feature points to provide rich geometric information for subsequent processing.

[0021] Step SB3: Calculate the camera's external parameters through a multi-view geometry estimation method, and then derive the projection parameters required in the panoramic image generation process.

[0022] Step SB4: Create a stitching weight map to seamlessly fuse the multi-aperture images into the large field of view image, and record the weight parameters of each pixel to optimize the fusion effect.

[0023] Step SB5: Pass the image stitching parameters, weight parameters and the size of the large field of view image to the stitching thread to ensure the smoothness and accuracy of real-time processing.

[0024] Step SB6: Perform frame transformation on the multi-aperture image according to the aforementioned parameters, and finally generate a high-quality large-field-of-view image to achieve real-time stitching.

[0025] The unmanned vehicle large field of view imaging video anti-shake method is characterized in that the large field of view video anti-shake module includes a feature point detection module, a key point motion estimation module, a grid motion propagation module and a motion smoothing module. The feature point detection module detects the input large field of view image feature points; the key point motion estimation module is connected to the feature point detection module, and is used to screen the detected feature points, and obtain the key point motion vector by matching the feature points of the current frame with the feature points of the past frame; the grid motion propagation module is connected to the key point motion estimation module, and first divides the large field of view video frame into M×N grids, and then propagates the key point motion data into grid vertex motion data according to a certain weight, and the weight is related to the distance between the key point and the grid vertex; the motion smoothing module is connected to the large field of view video anti-shake module, and is used to smooth the calculation to obtain the grid vertex motion, and perform large field of view image transformation according to the offset grid motion vector, and output large field of view stable video.

[0026] The grid motion propagation module and motion smoothing module first use the system's video acquisition module and real-time image stitching module to create a large field of view video data set, which is a large field of view video data set collected when the unmanned vehicle shakes during movement. The data set is input into the grid motion propagation module and the motion smoothing module for training, respectively, to improve the robustness of the anti-shake algorithm in smoothing complex large field of view jitters.

[0027] The unmanned vehicle large field of view imaging video stabilization method based on the unmanned vehicle large field of view imaging video stabilization system comprises the following steps: Step SC1: construct a large field of view video dataset based on the unmanned vehicle, including image data of different scenes; Step SC2: using the constructed data set to train the grid motion propagation module and the motion smoothing module, to obtain a grid motion propagation module and a motion smoothing module that are robust to image stabilization of a large field of view of an unmanned vehicle; Step SC3: Acquire a calibration image for the real-time image stitching module. The image must contain enough feature points to ensure the accuracy of the image transformation parameters, and calculate the image transformation parameters through the image; Step SC4: driving the unmanned vehicle. The unmanned vehicle may vibrate due to some uneven roads or the vehicle itself. Step SC5: The video acquisition module can collect a jittery multi-aperture image through a multi-aperture camera mounted on the unmanned vehicle due to the influence of the jittery posture information of the unmanned vehicle; Step SC6: input the jittered multi-aperture image into the real-time image stitching module, transform the multi-aperture image by using pre-calculated image transformation parameters to obtain a large field of view image; Step SC7: The large field of view video stabilization module receives the jittered large field of view image, performs feature point detection, key point motion estimation, grid motion propagation and motion smoothing on the jittered image, and finally outputs a stable large field of view video.

Claims

1. Unmanned vehicle large field of view imaging video anti-shake system, characterized by It includes unmanned vehicle, multi-aperture camera, acquisition card, computing card, display, video acquisition module, real-time image stitching module and large field of view video anti-shake module; The unmanned vehicle is equipped with a computing card, an acquisition card and a multi-aperture camera. The unmanned vehicle will shake during the movement, which is the source of the jitter of the unmanned vehicle's large field of view video; The video acquisition module includes an acquisition card, a multi-aperture camera, a multi-threaded parallel acquisition module and a frame synchronization control module; the multi-aperture camera is used to acquire multi-aperture videos; the acquisition card is connected to the multi-aperture camera to deploy a multi-aperture video acquisition algorithm; the multi-threaded parallel acquisition module is used to control the operation of the multi-aperture camera; the frame synchronization control module is used to control the multi-aperture camera to achieve synchronous acquisition; The real-time image stitching module is connected to the video acquisition module and is used to stitch the received multi-aperture images into a complete large-field-of-view image; The large-field-of-view video stabilization module is connected to the real-time image stitching module; the large-field-of-view video stabilization module includes a feature point detection module, a key point motion estimation module, a grid motion propagation module and a motion smoothing module, which are used to de-shake the stitched large-field-of-view video and output a large-field-of-view stabilized video; The computing card is used to deploy real-time stitching algorithms and large-field-of-view video stabilization algorithms, and is responsible for executing corresponding computing tasks; the display is used to present stabilized large-field-of-view videos.

2. The unmanned vehicle large field of view imaging video stabilization system according to claim 1, characterized in that: The multi-aperture camera is fixed on the unmanned vehicle at a certain angle to expand the field of view. The acquisition card is connected to the multi-aperture camera, and a video acquisition algorithm is deployed to control the camera to perform multi-aperture video acquisition. The real-time image stitching module and the large-field-of-view video anti-shake algorithm module are deployed on the computing card, which is responsible for anti-shake processing of the received large-field-of-view video. The display is connected to the acquisition card to display the stabilized video with a large field of view.

3. The unmanned vehicle large field of view imaging video stabilization system according to claim 1, characterized in that: The video acquisition module includes a multi-aperture camera module, a multi-threaded parallel acquisition module, an acquisition card and a frame synchronization control module; the acquisition card is respectively connected to the multi-aperture camera module, the multi-threaded parallel acquisition module and the frame synchronization control module; the multi-threaded parallel acquisition module controls the transmission of multi-aperture images in a multi-threaded parallel manner; the frame synchronization control module uses a continuous soft trigger method to send signals to the multi-aperture camera and controls parameters such as the frame rate at the same time; the multi-threaded parallel acquisition module and the frame synchronization control module are used in combination to obtain synchronized multi-aperture images.

4. The unmanned vehicle large field of view imaging video stabilization system according to claim 1, characterized in that The real-time image stitching comprises an image dedistortion module, an image registration module and an image transformation module; the image dedistortion module uses the calibrated camera internal parameters to dedistort the multi-aperture images respectively, and records the image dedistortion matrix at the same time; the image registration module is connected to the image dedistortion module, firstly, the image registration module receives a multi-aperture image with sufficient features, detects the feature information of the input image, records the internal and external parameters of the camera using the multi-view geometry calibration method, and calculates the transformation relationship parameters of the stitching image; the image transformation module is connected to the image registration module, uses the recorded image dedistortion parameters and image transformation parameters to perform a distortion transformation on the multi-aperture image, fuses the transformed multi-aperture image into one image, and obtains a large field of view image.

5. The unmanned vehicle large field of view imaging video stabilization system according to claim 1, characterized in that The large-field-of-view video anti-shake module includes a feature point detection module, a key point motion estimation module, a grid motion propagation module and a motion smoothing module; the feature point detection module detects feature points of the input large-field-of-view image; The key point motion estimation module is connected to the feature point detection module to filter the detected feature points and obtain the key point motion vector by matching the feature points of the current frame with the feature points of the past frame; the grid motion propagation module is connected to the key point motion estimation module, firstly, the large field of view video frame is divided into M×N grids, and then the key point motion data is propagated as grid vertex motion data according to a certain weight, which is related to the distance between the key point and the grid vertex, and finally multiple one-dimensional convolutions are used to extract motion features, fuse information and generate optimized grid motion trajectories; the motion smoothing module is connected to the large field of view video anti-shake module to smooth the calculation of grid vertex motion, perform large field of view image transformation according to the offset grid motion vector, and output large field of view stable video; The grid motion propagation module and the motion smoothing module first use the system's video acquisition module and real-time image stitching module to produce a large-field-of-view video data set, which is an unstable large-field-of-view video data set collected when the unmanned vehicle shakes during movement; the data set is respectively input into the grid motion propagation module and the motion smoothing module for training, thereby improving the robustness of the anti-shake algorithm in smoothing complex large-field-of-view shakes.

6. Unmanned vehicle large field of view imaging video anti-shake method, characterized by The unmanned vehicle large field of view imaging video stabilization system according to any one of claims 1 to 5 is adopted, comprising the following steps: Step S1: construct a large field of view video dataset based on the unmanned vehicle, including image data of different scenes; Step S2: using the constructed data set to train the grid motion propagation module and the motion smoothing module, to obtain a grid motion propagation module and a motion smoothing module that are robust to image stabilization of a large field of view of an unmanned vehicle; Step S3: Acquire a calibration image for the real-time image stitching module. The image must contain enough feature points to ensure the accuracy of the image transformation parameters, and calculate the image transformation parameters through the image; Step S4: driving the unmanned vehicle. The unmanned vehicle may vibrate due to some uneven roads or the vehicle itself. Step S5: The video acquisition module can collect a jittery multi-aperture image through a multi-aperture camera mounted on the unmanned vehicle due to the influence of the jittery posture information of the unmanned vehicle; Step S6: inputting the jittered multi-aperture image into a real-time image stitching module, transforming the multi-aperture image by using pre-calculated image transformation parameters to obtain a large field of view image; Step S7: The large-field-of-view video stabilization module receives the jittered large-field-of-view image, performs feature point detection, key point motion estimation, grid motion propagation and motion smoothing on the jittered image, and finally outputs a stable large-field-of-view video.

Images