A method and device for vehicle-mounted near and long-distance real-time panoramic splicing and video stabilization for armored vehicles

Abstract

This application provides a vehicle-mounted real-time panoramic stitching and video stabilization method and device for armored vehicles at both near and far distances, relating to the field of armored vehicle technology. The vehicle-mounted real-time panoramic stitching and video stabilization device for armored vehicles includes a vehicle body. A panoramic camera array is arranged around the periphery of the vehicle body, and a processing mechanism is arranged inside the vehicle body. The panoramic camera array includes a close-up camera, a mid-range camera, and a long-range camera. The close-up cameras are mounted around the perimeter of the vehicle body, while the mid-range and long-range cameras are mounted on top of the vehicle body. This vehicle-mounted real-time panoramic stitching and video stabilization method and device for armored vehicles utilizes multi-screen panoramic stitching technology based on homography matrix transformation, combined with camera intrinsic and extrinsic parameters to establish an initial stitching model. This enables near- and far-range panoramic observation of armored vehicles without blind spots, effectively eliminating ghosting between adjacent images and blind spots.

Description

Technical Field

[0001] This application relates to the field of armored vehicle technology, specifically to a method and apparatus for real-time panoramic stitching and video stabilization of armored vehicles at near and far distances. Background Technology

[0002] Armored vehicles refer to a general term for armored land vehicles that use high-strength metal or composite armor structures to achieve overall protection, have excellent land mobility and complex terrain traversal capabilities, can carry various weapons and equipment and personnel, and possess comprehensive performance in protection, mobility, and logistical support.

[0003] Traditional solutions often involve mounting a commander's or gunner's panoramic observation scope on the top of the vehicle. Some newer armored vehicles use multiple optical cameras mounted around the vehicle to observe its surroundings. The problem with panoramic observation scopes is that they can only observe a specific area at a time, failing to provide a comprehensive view of the entire environment. While multiple optical cameras around the vehicle can observe a larger area simultaneously, blind spots are easily created due to vehicle structural limitations. Furthermore, since armored vehicles are mostly used in field environments with extremely harsh road conditions and severe vibrations, the video footage from optical cameras is prone to significant shaking, affecting the observation results. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this application provides a vehicle-mounted real-time panoramic stitching and video stabilization method and apparatus for armored vehicles, which solves the problems mentioned in the background art.

[0005] To achieve the above objectives, this application provides the following technical solution: a vehicle-mounted real-time panoramic stitching and video stabilization device for armored vehicles, comprising a vehicle body, a panoramic camera mechanism disposed around the periphery of the vehicle body, and a processing mechanism disposed inside the vehicle body. The panoramic camera mechanism includes a close-up camera, a mid-range camera, and a long-range camera. The close-up camera is mounted around the vehicle body, and the mid-range camera and the long-range camera are both mounted on top of the vehicle body.

[0006] Preferably, mounting bases are fixedly installed on both sides of the vehicle body by bolts, and the close-up camera is fixedly installed on one side of the mounting base.

[0007] Preferably, a mounting plate is fixedly mounted on one side of the mounting base by bolts, a protective cover is fixedly mounted on one side of the mounting plate, and a flat glass is fixedly mounted on one end of the protective cover.

[0008] Preferably, a cantilever support is fixedly installed on the top of the vehicle body by bolts, and the mid-range camera is fixedly installed on one side of the cantilever support.

[0009] Preferably, a gimbal base is fixedly mounted on the upper surface of the vehicle turret platform by bolts, and the long-range camera is fixedly mounted on one end of the gimbal base.

[0010] Preferably, the processing mechanism includes a mounting plate, which is fixedly installed in the center of gravity region of the vehicle body floor. Two-stage damping shock absorbers are fixedly installed on the upper surface of the mounting plate. An anti-resonance bracket is fixedly installed at one end of each of the two-stage damping shock absorbers. An inertial measurement unit is fixedly installed on the upper surface of the anti-resonance bracket.

[0011] Preferably, the processing mechanism further includes a vehicle-mounted terminal, which is fixedly installed inside the vehicle body. The vehicle-mounted terminal integrates a video preprocessing module, a panoramic stitching module, a collaborative image stabilization module, and a central control unit.

[0012] Preferably, the vehicle-mounted real-time panoramic stitching and video stabilization method for armored vehicles includes the following steps: synchronous hardware acquisition of multiple video streams from near-field blind spots, mid-field paths, and distant targets using a vehicle-mounted distributed camera array; denoising, contrast enhancement, and lens distortion correction of the multiple video streams to output standard distortion-free video frames; feature extraction and matching of the corrected video frames to solve the global homography transformation matrix, and mapping the near-field, mid-field, and distant videos to a unified panoramic coordinate system; weighted fusion and boundary smoothing of the overlapping areas of the multiple video streams in the panoramic coordinate system to generate an integrated real-time panoramic image from near and far distances; global motion compensation using Kalman filtering based on vehicle attitude data acquired by the IMU and the global motion vector of the panoramic image to complete the synchronous processing of stitching and stabilization; and real-time output of the stabilized panoramic image to the vehicle-mounted display terminal for panoramic display, zooming, and alarm prompts.

[0013] Preferably, a dynamic fusion strategy is adopted during panoramic stitching, and the detail weight of obstacles is increased in the near-field area.

[0014] Preferably, the target sharpness weight is increased in the far-distance area, and the collaborative image stabilization is used for global motion compensation. Only high-frequency vibration components are filtered out, while the normal steering and pitch motion of the vehicle body is retained, thus avoiding image drift and distortion.

[0015] This application provides a method and apparatus for real-time panoramic stitching and video stabilization at near and long distances for armored vehicles. It has the following beneficial effects: 1. The vehicle-mounted near- and far-range real-time panoramic stitching and video stabilization method and device for armored vehicles, through multi-screen panoramic stitching technology based on homography matrix transformation, and by combining the camera's intrinsic and extrinsic parameters to establish an initial stitching model, can achieve near- and far-range panoramic observation of armored vehicles without blind spots, effectively eliminating ghosting of adjacent images and blind spots; at the same time, it combines the vehicle's vibration and impact characteristics to complete motion estimation and panoramic stabilization, significantly suppressing image shaking caused by harsh road conditions in the field.

[0016] 2. The vehicle-mounted near- and far-range real-time panoramic stitching and video stabilization method and device for armored vehicles combines the calculation of the stabilization affine matrix and the panoramic stitching affine matrix, and completes the integrated processing of image stabilization and stitching with only one affine transformation. Compared with the traditional step-by-step processing, it significantly improves the algorithm's running efficiency and ensures the real-time performance of the video. The use of a fade-in and fade-out image fusion algorithm achieves a natural transition of the image, and the fusion effect is better than the conventional weighted average fusion, with no obvious stitching traces, thus improving the overall user experience and efficiency of the vehicle-mounted observation system. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the main appearance structure of this application; Figure 2 This is a schematic diagram of the close-up camera and its connection structure according to this application; Figure 3 This is a schematic diagram of the mid-range camera and its connection structure in this application; Figure 4 This is a schematic diagram of the long-range camera and its connection structure according to this application; Figure 5 This is a schematic diagram of the inertial measurement unit and its connection structure according to this application; Figure 6 This is a flowchart of the panoramic stitching and video stabilization process in this application.

[0019] In the diagram: 1. Vehicle body; 2. Panoramic camera mechanism; 201. Mounting base; 202. Mounting plate; 203. Protective cover; 204. Flat glass; 205. Close-up camera; 206. Cantilever support; 207. Mid-range camera; 208. Gimbal base; 209. Long-range camera; 3. Processing mechanism; 301. Mounting plate; 302. Two-stage damping shock absorber; 303. Anti-resonance bracket; 304. Inertial measurement unit. Detailed Implementation

[0020] It should be noted that in the description of the embodiments of this application, the terms "front," "rear," "left," "right," "up," "down," etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. The terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication between two elements. For those skilled in the art, the specific meaning of the above terms in this application can be understood according to the specific circumstances.

[0021] The present application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0022] Reference Figure 1 and Figure 2 This application provides a vehicle-mounted near-long-range real-time panoramic stitching and video stabilization device for armored vehicles, including a vehicle body 1, a panoramic camera mechanism 2 arranged around the outer periphery of the vehicle body 1, and a processing mechanism 3 arranged inside the vehicle body 1. The panoramic camera mechanism 2 includes a close-range camera 205, a mid-range camera 207 and a long-range camera 209. The close-range camera 205 is installed around the vehicle body 1, and the mid-range camera 207 and the long-range camera 209 are both installed above the vehicle body 1. Using the armored vehicle hull 1 as the overall mounting carrier, a panoramic camera array 2 is arranged around the exterior of the hull 1. Near-field cameras 205 are installed around the hull 1 to cover near-field blind spots, while mid-field cameras 207 and long-field cameras 209 are installed on top of the hull 1 to cover mid-to-long-range observation areas. At the same time, a processing mechanism 3 is set up inside the hull 1 to receive multiple video signals collected by the panoramic camera array 2 and complete integrated processing such as panoramic stitching and image stabilization. This can construct a three-level distributed visual perception system of near, mid, and long distances, realize panoramic observation of the surrounding environment of the armored vehicle without blind spots, and provide complete hardware support for real-time perception.

[0023] Reference Figure 2 , Figure 3 and Figure 4In one aspect of this embodiment, mounting bases 201 are fixedly installed on both sides of the vehicle body 1 by bolts, and a close-range camera 205 is fixedly installed on one side of the mounting base 201. By firmly fixing the mounting base 201 to both sides of the vehicle body 1 with bolts, and using the mounting base 201 as a rigid reference platform, the close-range camera 205 is stably installed on one side of the mounting base 201. This ensures that the installation posture of the close-range camera 205 is accurate and the viewing angle is fixed. It can not only achieve reliable fixation and convenient disassembly and maintenance of the close-range camera 205, but also ensure stable observation of the blind spot at close range, adapting to the harsh working conditions of armored vehicles driving in the field with bumps and impacts.

[0024] A mounting plate 202 is bolted to one side of the mounting base 201, and a protective cover 203 is bolted to one side of the mounting plate 202. A flat glass 204 is bolted to one end of the protective cover 203. The mounting plate 202 is fixed to one side of the mounting base 201 with bolts, and the protective cover 203 is fixed with the mounting plate 202 as a connecting structure, so that the protective cover 203 completely covers and protects the close-range camera 205. The flat glass 204 at the front end of the protective cover 203 has high light transmittance and can block sand, rain, and impacts without obstructing the imaging light path, providing all-round physical protection for the close-range camera 205, ensuring that the camera can continuously and stably image in harsh environments and extending the service life of the equipment.

[0025] A cantilever support 206 is bolted to the top of the vehicle body 1, and a mid-range camera 207 is fixedly mounted on one side of the cantilever support 206. By bolting the cantilever support 206 to the top of the vehicle body 1, and utilizing the outward extension of the cantilever structure to create an unobstructed installation space, the mid-range camera 207 can be fixed to one side of the cantilever support 206. This effectively avoids the obstruction of the mid-range field of view by the vehicle body 1 itself, and obtains a continuous and open mid-range path observation image. This provides stable and complete mid-section image data for panoramic stitching, ensuring that there are no missing or discontinuous areas in the middle of the panoramic image.

[0026] A gimbal base 208 is bolted to the upper surface of the turret platform of vehicle hull 1, and a long-range camera 209 is fixedly mounted on one end of the gimbal base 208. The gimbal base 208 is securely mounted to the upper surface of the turret platform of vehicle hull 1 with bolts. The long-range camera 209 is mounted on the gimbal base 208 as a load-bearing and driving structure. The gimbal can drive the long-range camera 209 to achieve horizontal rotation and pitch adjustment, enabling flexible aiming and tracking of long-range targets, achieving high-definition observation and accurate identification of long-range targets, and meeting the operational requirements of armored vehicles.

[0027] Reference Figure 1 and Figure 5In one aspect of this embodiment, the processing mechanism 3 includes a mounting plate 301, which is fixedly installed in the center of gravity region of the vehicle body 1's floor. A two-stage damping shock absorber 302 is fixedly installed on the upper surface of the mounting plate 301, and an anti-resonance bracket 303 is fixedly installed at one end of the two-stage damping shock absorber 302. An inertial measurement unit 304 is fixedly installed on the upper surface of the anti-resonance bracket 303. By fixing the mounting plate 301 to the center of gravity region of the vehicle body 1's floor, and using this as a basis to install the two-stage damping shock absorber 302 and the anti-resonance bracket 303, the damping shock absorber 302 can significantly absorb the high-frequency vibrations and impacts generated during vehicle movement, and the anti-resonance bracket 303 avoids structural resonance interference. Ultimately, this provides a highly stable and low-disturbance measurement environment for the inertial measurement unit 304, ensuring that the inertial measurement unit 304 accurately collects data such as vehicle body attitude, angular velocity, and acceleration, providing reliable core parameters for subsequent global motion compensation and video stabilization.

[0028] The processing unit 3 also includes an on-board terminal, which is fixedly installed inside the vehicle body 1. The on-board terminal integrates a video preprocessing module, a panoramic stitching module, a collaborative image stabilization module, and a central control unit. The on-board terminal serves as the core processing unit inside the vehicle body 1. The terminal integrates four main modules: video preprocessing, panoramic stitching, collaborative image stabilization, and central control. The preprocessing module performs video denoising and distortion correction; the panoramic stitching module performs multi-view fusion mapping; the collaborative image stabilization module performs jitter suppression and motion compensation; and the central control unit coordinates the collaborative work of each module, achieving integrated and efficient operation of the entire process from video acquisition, preprocessing, panoramic stitching, image stabilization, to display output, ensuring system real-time performance, image clarity, and stability.

[0029] Reference Figure 6 In one aspect of this embodiment, an armored vehicle-mounted real-time panoramic stitching and video stabilization method for near and far distances includes the following steps: synchronous hardware acquisition of multiple video streams from near-field blind spots, mid-field paths, and far-field targets using an onboard distributed camera array; denoising, contrast enhancement, and lens distortion correction of the multiple video streams to output standard distortion-free video frames; feature extraction and matching of the corrected video frames to solve the global homography transformation matrix, and mapping the near-field, mid-field, and far-field videos to a unified panoramic coordinate system; weighted fusion and boundary smoothing of the overlapping areas of the multiple video streams in the panoramic coordinate system to generate an integrated real-time panoramic image for near and far distances; global motion compensation using Kalman filtering based on vehicle attitude data acquired by the IMU and the global motion vector of the panoramic image to complete the synchronous processing of stitching and stabilization; and real-time output of the stabilized panoramic image to an onboard display terminal for panoramic display, zooming, and alarm prompts.

[0030] A dynamic fusion strategy is used during panoramic stitching, with increased weighting of obstacle details in the foreground area.

[0031] In long-distance areas, target sharpness is weighted more effectively. Collaborative image stabilization provides global motion compensation, filtering out only high-frequency vibration components while preserving normal vehicle steering and pitch motion, thus avoiding image drift and distortion. Based on the installation pose (extrinsic parameters) and intrinsic parameters of panoramic camera module 2, an initial panoramic stitching parameter model for the panoramic observation components is established. Multiple panoramic observation components acquire camera video streams in real time. The video processing module acquires video streams from multiple panoramic observation components, performs motion estimation on the video images, and obtains the stabilization parameters for each panoramic observation component. Based on the basic pose model and stabilization parameters of the panoramic observation components, a real-time pose model for the panoramic observation components is established. Using the real-time pose model, a homography transformation model is employed to achieve real-time stabilization and stitching of multiple panoramic video images, and a fade-in / fade-out fusion algorithm is used to fuse adjacent images.

[0032] Electronic image stabilization uses the ORB feature extraction operator to extract feature points from the previous frame and the current frame, and solves the affine matrix using the RANSAC method to obtain the image stabilization parameters (image stabilization affine matrix) for the current frame.

[0033] Let the input image be I(u,v), the transformed output image be I(x,y), and the affine transformation matrix be T.

[0034] The basic pose model of panoramic camera module 2 is the basis for panoramic stitching of multiple panoramic camera module 2 videos. Based on the installation position (extrinsic parameters) and camera intrinsic parameters of panoramic camera module 2, and taking the first panoramic camera module 2 image as the reference, an affine matrix of each panoramic camera module 2 image is established as the basic pose model of each panoramic camera module 2.

[0035] Traditional electronic image stabilization and panoramic stitching processes typically involve first stabilizing a single image to generate a stabilized image, and then using the stabilized image for panoramic stitching. Because this requires two affine transformations, the processing efficiency is relatively low.

[0036] This invention proposes an innovative method: first, the image stabilization affine matrix of a single image and the panoramic stitching affine matrix are merged and calculated to generate a set of end-to-end affine matrices. Then, only one affine transformation is needed to generate the stabilized panoramic image from the original image. Compared with the method of stabilizing multiple camera video frames individually and then performing panoramic stitching transformation, this reduces one set of image transformation operations, significantly improving the algorithm's efficiency and ensuring the real-time performance of video display.

[0037] Image fusion employs a progressive-in / progressive fusion algorithm, which assigns linear weights to pixels in the overlapping regions of the two images: Where d is the distance from the pixel to the boundary of the overlapping region.

[0038] This algorithm can achieve a natural transition and its fusion effect is better than that of the weighted average fusion algorithm.

[0039] All electrical devices in this plan are powered by an external power source.

[0040] Working Principle: When in use, the system first performs geometric calibration based on the installation pose of the panoramic camera module 2 and its intrinsic parameters, establishing an initial panoramic stitching parameter model and a basic pose model to provide a unified coordinate reference for multiple video streams. The panoramic camera modules 2, distributed around and above the vehicle, acquire video streams in real-time from the vehicle's 360° near-field blind spots, mid-field driving path, and distant targets using synchronous hardware acquisition, and transmit them at high speed to the video processing module. The video processing module uses the ORB feature extraction operator to obtain inter-frame feature points, uses the RANSAC algorithm to remove mismatched points and solves the affine matrix to obtain image stabilization parameters, accurately separating the high-frequency jitter components caused by normal vehicle movement and road bumps. Then, the image stabilization affine matrix and the panoramic stitching affine matrix are merged to generate an end-to-end composite transformation matrix. Video stabilization and panoramic stitching can be completed simultaneously with only one affine transformation, significantly improving processing efficiency. Subsequently, a fade-in / fade-out fusion algorithm is adopted, which assigns linear weights based on the distance from the pixel to the boundary of the overlapping area, and performs pixel-level smooth fusion on the stitched area to eliminate ghosting and stitching marks. Finally, the processed real-time stabilized panoramic image is output to the vehicle display terminal, providing armored vehicles with a blind-spot-free, high-definition, and low-shake observation effect.

[0041] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0042] Although embodiments of this application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A vehicle-mounted real-time panoramic stitching and video stabilization device for armored vehicles, comprising a vehicle body (1), characterized in that: A panoramic camera mechanism (2) is provided on the outer periphery of the vehicle body (1), and a processing mechanism (3) is provided inside the vehicle body (1). The panoramic camera mechanism (2) includes a close-up camera (205), a medium-range camera (207) and a long-range camera (209). The close-up camera (205) is installed around the vehicle body (1), and the medium-range camera (207) and the long-range camera (209) are both installed above the vehicle body (1).

2. The vehicle-mounted near- and long-range real-time panoramic stitching and video stabilization device for armored vehicles according to claim 1, characterized in that: The vehicle body (1) has mounting bases (201) fixedly installed on both sides by bolts, and the close-up camera (205) is fixedly installed on one side of the mounting base (201).

3. The vehicle-mounted near-long-range real-time panoramic stitching and video stabilization device for armored vehicles according to claim 2, characterized in that: The mounting base (201) is fixedly mounted with a mounting plate (202) on one side by bolts, and a protective cover (203) is fixedly mounted on one side of the mounting plate (202). A flat glass (204) is fixedly mounted on one end of the protective cover (203).

4. The vehicle-mounted near- and long-range real-time panoramic stitching and video stabilization device for armored vehicles according to claim 1, characterized in that: The top of the vehicle body (1) is fixedly mounted with a cantilever support (206) by bolts, and the mid-range camera (207) is fixedly mounted on one side of the cantilever support (206).

5. The vehicle-mounted near-long-range real-time panoramic stitching and video stabilization device for armored vehicles according to claim 1, characterized in that: The upper surface of the turret platform of the vehicle body (1) is fixedly installed with a gimbal base (208) by bolts, and the long-range camera (209) is fixedly installed at one end of the gimbal base (208).

6. The vehicle-mounted near-long-range real-time panoramic stitching and video stabilization device for armored vehicles according to claim 1, characterized in that: The processing mechanism (3) includes a mounting plate (301), which is fixedly installed in the center of gravity area of ​​the bottom plate of the vehicle body (1). A two-stage damping shock absorber (302) is fixedly installed on the upper surface of the mounting plate (301). An anti-resonance bracket (303) is fixedly installed at one end of the two-stage damping shock absorber (302). An inertial measurement unit (304) is fixedly installed on the upper surface of the anti-resonance bracket (303).

7. The vehicle-mounted near- and long-range real-time panoramic stitching and video stabilization device for armored vehicles according to claim 1, characterized in that: The processing mechanism (3) also includes a vehicle-mounted terminal, which is fixedly installed inside the vehicle body (1). The vehicle-mounted terminal integrates a video preprocessing module, a panoramic stitching module, a collaborative image stabilization module, and a central control unit.

8. A vehicle-mounted real-time panoramic stitching and video stabilization method for armored vehicles, applied to the device described in any one of claims 1-7, characterized in that: Includes the following steps: The system uses a vehicle-mounted distributed camera array to simultaneously acquire multiple video feeds from near-field blind spots, mid-field paths, and distant targets. Noise reduction, contrast enhancement, and lens distortion correction are applied to these multiple video feeds, outputting standard distortion-free video frames. Feature extraction and matching are performed on the corrected video frames to solve the global homography transformation matrix, unifying the mapping of near-field, mid-field, and distant videos to a panoramic coordinate system. Weighted fusion and boundary smoothing are then applied to the overlapping areas of the multiple video feeds in the panoramic coordinate system to generate a unified real-time panoramic image integrating near and far distances. Based on the vehicle posture data collected by the IMU and the global motion vector of the panoramic image, Kalman filtering is used for global motion compensation to complete the stitching and image stabilization synchronization processing. The stabilized panoramic image is output to the vehicle display terminal in real time, enabling panoramic display, zooming, and alarm prompts.

9. A method for real-time panoramic stitching and video stabilization for armored vehicles at near and far distances according to claim 8, characterized in that: A dynamic fusion strategy is used during panoramic stitching, with increased weighting of obstacle details in the foreground area.

10. A method for real-time panoramic stitching and video stabilization for armored vehicles at near and far distances according to claim 8, characterized in that: In distant areas, the target clarity weight is increased, and collaborative image stabilization is used for global motion compensation. Only high-frequency vibration components are filtered out, while the normal steering and pitch motion of the vehicle body is preserved, avoiding image drift and distortion.

Images