Video panoramic stitching and three-dimensional fusion device and system

Abstract

The invention relates to the technical field of video stitching, in particular to a video panoramic stitching and three-dimensional fusion device and system.The video panoramic stitching and three-dimensional fusion device comprises a picture acquisition module, a picture correction module, a stitching optimization module, a three-dimensional reconstruction module and a model optimization module, and the picture acquisition module is used for acquiring video pictures in all directions; the picture correction module is used for eliminating noise and errors in the collected video picture; the splicing optimization module is used for splicing the video streams collected by the multiple cameras into a panoramic video and optimizing the splicing part; the three-dimensional reconstruction module is used for reconstructing a three-dimensional scene from the spliced panoramic video; the model optimization module is used for optimizing the reconstructed three-dimensional grid model; therefore, detail correction is carried out on the collected picture, the picture is optimized, gaps, black edges, picture overlapping or picture tearing are eliminated, a high-precision three-dimensional model is constructed, and the watching experience of a user is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video splicing, and in particular to a device and system for panoramic video splicing and three-dimensional fusion. Background Art

[0002] Currently, in order to ensure that all angles or areas are captured and recorded during monitoring or filming, multiple cameras (or mobile devices) are used to synchronously capture video streams from different angles to cover 360° or larger scenes. However, these multiple images need to be displayed using corresponding display devices. Too many displayed images will reduce the user experience. Therefore, video images from multiple directions are usually spliced ​​together to form a 360-degree panoramic video, which can not only greatly improve the user experience, but also enable users to observe more content more conveniently.

[0003] In the aforementioned existing technologies, traditional splicing optimization simply combines multiple images together through a simple three-dimensional model. Although it can help users watch more images at one time, there are usually gaps, black edges, overlapping or tearing of images at the joints of different images, which seriously affects the user's viewing experience. Summary of the Invention

[0004] The purpose of the present invention is to provide a video panoramic stitching and three-dimensional fusion device and system to solve the problem that traditional stitching optimization in the prior art only combines multiple pictures together through a simple three-dimensional model. Although it can help users watch more pictures at one time, there are usually gaps, black edges, picture overlap or picture tearing at the splicing points of different pictures, which seriously affects the user's viewing experience.

[0005] To achieve the above-mentioned object, the present invention provides a video panoramic stitching and 3D fusion system, comprising an image acquisition module, an image correction module, a stitching optimization module, a 3D reconstruction module, and a model optimization module, wherein the image acquisition module, the image correction module, the stitching optimization module, the 3D reconstruction module, and the model optimization module are connected in sequence;

[0006] The picture acquisition module is used to acquire video pictures from all directions;

[0007] The image correction module is used to eliminate noise and errors in the collected video images;

[0008] The splicing optimization module is used to splice the video streams collected by multiple cameras into a panoramic video and optimize the splicing points;

[0009] The 3D reconstruction module is used to reconstruct a 3D scene from the stitched panoramic video;

[0010] The model optimization module is used to optimize the reconstructed three-dimensional mesh model.

[0011] The image acquisition module includes a camera deployment unit, a camera adjustment unit and an image transmission unit, and the camera deployment unit, the camera adjustment unit and the image transmission unit are connected in sequence;

[0012] The camera deployment unit is used to select a high-precision camera model and determine the camera installation position and angle to ensure coverage of the target scene;

[0013] The camera adjustment unit is used to adjust the exposure time, white balance and focus parameters of the camera;

[0014] The image transmission unit is used to transmit the captured images to the picture correction module via a network.

[0015] The image correction module includes an image preprocessing unit, a spatiotemporal synchronization unit, and a distortion correction unit, and the image preprocessing unit, the spatiotemporal synchronization unit, and the distortion correction unit are connected in sequence;

[0016] The image preprocessing unit is used to perform preprocessing operations of denoising and enhancing the video image;

[0017] The spatiotemporal synchronization unit is used to keep the video images captured by multiple cameras consistent in time and space;

[0018] The distortion correction unit is used to eliminate the influence of camera lens distortion on the image.

[0019] Wherein, the spatiotemporal synchronization unit includes a hardware triggering subunit and a software compensation subunit, and the hardware triggering subunit and the software compensation subunit are connected;

[0020] The hardware trigger subunit is used to generate a synchronization pulse signal using FPGA to trigger all cameras to expose and ensure time synchronization;

[0021] The software compensation subunit is used to dynamically model network transmission delay and align asynchronous frames using an interpolation algorithm to improve time synchronization accuracy;

[0022] The distortion correction unit includes an offline calibration subunit and an online calibration subunit, and the offline calibration subunit is connected to the online calibration subunit;

[0023] The offline calibration subunit is used to obtain camera parameters using a checkerboard calibration plate and establish a spatial mapping relationship between cameras;

[0024] The online calibration subunit is used to apply the undistort() function of OpenCV to correct image distortion in real time and improve image quality.

[0025] The splicing optimization module includes a semantic segmentation unit, a dynamic object processing unit and a gap elimination unit, and the semantic segmentation unit, the dynamic object processing unit and the gap elimination unit are connected in sequence;

[0026] The semantic segmentation unit is used to build a segmentation network based on MobileNetV3-Small, perform pixel-level semantic segmentation on the image, and identify different objects, including but not limited to pedestrians, vehicles and buildings.

[0027] The dynamic object processing unit is used to adopt a multi-view fusion strategy for the segmented dynamic objects, apply the optical flow method to track the object's motion trajectory, optimize the splicing weight, and avoid overlap or breakage at the splicing seams;

[0028] The seam elimination unit is used to use a Poisson fusion algorithm to eliminate illumination and geometric differences at the seams to achieve seamless splicing.

[0029] The three-dimensional reconstruction module includes a depth estimation unit, a grid generation unit and a semantic fusion unit, which are connected in sequence;

[0030] The depth estimation unit is used to generate a dense depth map based on a multi-view stereo vision algorithm, fuse IMU data to optimize the depth estimation result, and improve the accuracy of depth estimation;

[0031] The mesh generation unit uses the Marching Cubes algorithm to extract isosurfaces from the depth map to generate a three-dimensional mesh model, and simultaneously applies a mesh simplification algorithm to reduce the number of faces and improve rendering efficiency;

[0032] The semantic fusion unit is used to map semantic labels to the three-dimensional grid surface and optimize the grid topology based on semantic information.

[0033] Wherein, the model optimization module includes a texture mapping unit and a real-time rendering unit, and the texture mapping unit and the real-time rendering unit are connected;

[0034] The texture mapping unit is used to select the texture with the most correct viewing angle and the highest resolution for mapping each mesh surface;

[0035] The real-time rendering unit is used to utilize the parallel computing capability of the GPU to accelerate texture mapping and lighting calculations, and dynamically adjust the model detail level according to the camera distance to eliminate invisible faces.

[0036] The present invention also provides a video panoramic stitching and three-dimensional fusion device, which adopts the above-mentioned video panoramic stitching and three-dimensional fusion system, including a mounting plate, a plurality of cameras and a plurality of camera adjustment components, wherein the plurality of camera adjustment components are sequentially arranged on the mounting plate, and each of the cameras is mounted on a corresponding camera adjustment component;

[0037] The camera adjustment assembly includes an arcuate slide rail, an arcuate rack, a first gear, a first gear driving component, a sliding shell, a quick-release shell and a quick-release unit. The arcuate slide rail is arranged on one side of the mounting plate, the arcuate rack is arranged inside the arcuate slide rail, the sliding shell is slidingly connected to the arcuate slide rail, the sliding shell has a groove, the first gear driving component is arranged inside the groove, the output end of the first gear driving component is fixedly connected to the first gear, the first gear and the arcuate rack are engaged with each other, the quick-release shell is connected to the sliding shell through the quick-release unit, and the camera is arranged on one side of the quick-release shell.

[0038] In which, the quick-release unit includes two locking mechanisms, an unlocking rotating shell, a U-shaped block, a connecting shaft, two linkage mechanisms, a pop-up plate, a pop-up spring and a pop-up telescopic rod, the two locking mechanisms are symmetrically arranged inside the quick-release shell, the unlocking rotating shell is sleeved on the outside of the camera, the quick-release shell has two arc grooves, the two ends of the U-shaped block respectively pass through the corresponding arc grooves, and are both fixedly connected to the unlocking rotating shell, the connecting shaft is arranged inside the quick-release shell, one side of the U-shaped block is fixedly connected to the connecting shaft, the two linkage mechanisms are symmetrically arranged on both sides of the connecting shaft, the quick-release shell also has a slot, the two ends of the pop-up telescopic rod are respectively fixedly connected to the inner wall of the slot and the pop-up plate, the two ends of the pop-up spring are respectively movably connected to the inner wall of the slot and the pop-up plate, and the pop-up spring is sleeved on the outside of the pop-up telescopic rod.

[0039] Wherein, the locking mechanism includes a locking plate, a plurality of locking bevel blocks, a plurality of locking telescopic rods and a plurality of locking springs, the plurality of locking bevel blocks are fixedly connected to the locking plate, the ends of the plurality of locking telescopic rods are respectively fixedly connected to the inner wall of the sliding housing and the locking plate, the ends of the plurality of locking springs are respectively movably connected to the inner wall of the sliding housing and the locking plate, the locking springs are sleeved on the outside of the locking telescopic rods, the quick-release housing has a plurality of locking grooves, and the locking bevel blocks are adapted to the locking grooves;

[0040] The linkage mechanism includes a push plate, multiple push blocks, a shift rod, a rotating plate, a linkage rod, a linkage rotating block, a second gear, a linkage rack and a return spring, multiple push blocks are respectively slidably connected to the corresponding locking grooves, multiple and one end of the push blocks are fixedly connected to the push plate, one end of the shift rod is arranged on the outside of the connecting shaft, the other end of the shift rod is located on one side of the rotating plate, the rotating plate and the second gear in the zone are both rotatably connected to the inside of the quick-release housing, the linkage block is arranged on one side of the second gear, two ends of the linkage rod are respectively rotatably connected to the other side of the rotating plate and the linkage block, the linkage rack is arranged on one side of the push plate, the linkage rack and the second gear are meshed with each other, and the two ends of the return spring are respectively movably connected to the rotating plate and the inner wall of the quick-release housing.

[0041] The present invention provides a video panoramic stitching and 3D fusion device and system, wherein the image acquisition module is used to acquire video images from various directions; the image correction module is used to eliminate noise and errors in the acquired video images; the stitching optimization module is used to stitch video streams acquired by multiple cameras into a panoramic video and optimize the stitching; the 3D reconstruction module is used to reconstruct a 3D scene from the stitched panoramic video; and the model optimization module is used to optimize the reconstructed 3D mesh model.

[0042] Therefore, by correcting the details of the captured images and then optimizing them, gaps, black edges, image overlap or image tearing are eliminated, and finally a high-precision three-dimensional model is constructed, which significantly improves the user's viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art.

[0044] Figure 1 It is a schematic diagram of the video panoramic stitching and 3D fusion system of the present invention.

[0045] Figure 2 It is a principle diagram of the picture acquisition module of the present invention.

[0046] Figure 3 It is a principle diagram of the image correction module of the present invention.

[0047] Figure 4 It is a schematic diagram of the space-time synchronization unit of the present invention.

[0048] Figure 5 It is a principle diagram of the distortion correction unit of the present invention.

[0049] Figure 6 It is a schematic diagram of the splicing optimization module of the present invention.

[0050] Figure 7 It is a schematic diagram of the three-dimensional reconstruction module of the present invention.

[0051] Figure 8 It is a schematic diagram of the model optimization module of the present invention.

[0052] Figure 9 It is a structural diagram of the video panoramic stitching and three-dimensional fusion device of the present invention.

[0053] Figure 10 It is a cross-sectional view of the video panoramic stitching and three-dimensional fusion device of the present invention.

[0054] Figure 11 The present invention Figure 10 A magnified view of the local structure at point A.

[0055] Figure 12 It is a structural schematic diagram of the sliding housing of the present invention.

[0056] Figure 13 It is a structural schematic diagram of the quick-release housing of the present invention.

[0057] Figure 14 It is a cross-sectional view of the quick-release housing of the present invention.

[0058] Figure 15 It is a diagram of the internal structure of the quick-release housing of the present invention.

[0059] 1-Image acquisition module, 101-Camera deployment unit, 102-Camera adjustment unit, 103-Image transmission unit, 2-Image correction module, 201-Image preprocessing unit, 202-Spatiotemporal synchronization unit, 2021-Hardware trigger subunit, 2022-Software compensation subunit, 203-Distortion correction unit, 2031-Offline calibration subunit, 2032-Online calibration subunit, 3-Splice optimization module, 301-Semantic segmentation unit, 302-Dynamic object processing unit, 303-Gap elimination unit, 4-3D reconstruction module, 401-Depth estimation unit, 402-Grid generation unit, 403-Semantic fusion unit, 5-Model optimization module, 501-Texture mapping unit Yuan, 502-real-time rendering unit, 6-mounting plate, 7-camera, 8-arc-shaped slide rail, 9-arc-shaped rack, 10-first gear, 11-first gear driving component, 12-sliding housing, 13-quick-release housing, 14-groove, 15-unlocking rotating housing, 16-U-shaped block, 17-connecting shaft, 18-pop-up plate, 19-pop-up spring, 20-pop-up telescopic rod, 21-arc-shaped groove, 22-slot, 23-locking plate, 24-locking oblique block, 25-locking telescopic rod, 26-locking spring, 27-locking groove, 28-push plate, 29-push block, 30-dial lever, 31-rotating plate, 32-linkage rod, 33-linkage rotating block, 34-second gear, 35-linkage rack, 36-reset spring. DETAILED DESCRIPTION

[0060] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be understood as limiting the present invention.

[0061] See also Figures 1 to 8 The present invention provides a video panoramic stitching and three-dimensional fusion system.

[0062] The picture acquisition module 1 is used to acquire video pictures from all directions;

[0063] Specifically include:

[0064] The camera deployment unit 101 is used to select a high-precision camera 7 model and determine the installation position and angle of the camera 7 to ensure coverage of the target scene;

[0065] Sensors can be installed on the camera 7. A high-precision six-axis IMU, specifically the Bosch BMI088, can be selected for attitude compensation and motion estimation. RTK-GPS can also be optionally installed to provide centimeter-level positioning accuracy, suitable for vehicle-mounted or drone-mounted scenarios. This provides a basis for the camera adjustment unit 102. The deployment plan needs to consider the scene characteristics and subsequent processing requirements.

[0066] The camera adjustment unit 102 is used to adjust the exposure time, white balance and focus parameters of the camera 7;

[0067] Based on the result of the camera deployment unit 101, the camera 7 is fine-tuned to ensure the quality of the collected data.

[0068] The image transmission unit 103 is used to transmit the captured image to the image correction module 2 via a network.

[0069] The collected video stream is transmitted to the picture correction module 2 to provide a data basis for subsequent processing.

[0070] The image correction module 2 is used to eliminate noise and errors in the collected video images;

[0071] Specifically include:

[0072] The image pre-processing unit 201 is used to perform pre-processing operations of denoising and enhancing the video image;

[0073] After preprocessing, the image quality is improved, and processed image data is provided to the spatiotemporal synchronization unit 202 and the distortion correction unit 203.

[0074] The spatiotemporal synchronization unit 202 is used to keep the video images captured by the multiple cameras 7 consistent in time and space;

[0075] Specifically include:

[0076] The hardware trigger subunit 2021 is used to generate a synchronization pulse signal using FPGA to trigger all cameras 7 to expose and ensure time synchronization;

[0077] The software compensation subunit 2022 is used to dynamically model network transmission delay and align asynchronous frames using an interpolation algorithm to improve time synchronization accuracy;

[0078] Through hardware triggering and software compensation, the temporal and spatial consistency of the video streams captured by the multiple cameras 7 is ensured.

[0079] The distortion correction unit 203 is used to eliminate the influence of the lens distortion of the camera 7 on the image.

[0080] Specifically include:

[0081] The offline calibration subunit 2031 is used to obtain the parameters of the camera 7 using a checkerboard calibration plate and establish a spatial mapping relationship between the cameras 7;

[0082] The online calibration subunit 2032 is used to apply the undistort() function of OpenCV to correct image distortion in real time to improve image quality.

[0083] By means of correction, the influence of the lens distortion of the camera 7 on the image can be eliminated, thereby further improving the image quality.

[0084] The splicing optimization module 3 is used to splice the video streams captured by multiple cameras 7 into a panoramic video and optimize the splicing points;

[0085] Specifically include:

[0086] The semantic segmentation unit 301 is used to build a segmentation network based on MobileNetV3-Small, perform pixel-level semantic segmentation on the image, and identify different objects, including but not limited to pedestrians, vehicles and buildings.

[0087] The semantic-aware stitching algorithm is used to divide objects into different semantics, thereby providing semantic information to the dynamic object processing unit 302 and the gap elimination unit 303.

[0088] The dynamic object processing unit 302 is used to adopt a multi-view fusion strategy for the segmented dynamic objects, apply the optical flow method to track the object's motion trajectory, optimize the splicing weight, and avoid overlap or breakage at the splicing seams;

[0089] For dynamic objects, image information from different cameras 7 or different time frames of the same camera 7 is integrated, and cross-perspective associations are established through feature point matching or deep learning models, thereby eliminating the loss of object information caused by single-perspective occlusion or perspective changes. For example, in an intersection scene, multiple cameras 7 can work together to fully track the vehicle trajectory; at the same time, the optical flow field (Lucas-Kanade, Farneback or deep learning FlowNet) is calculated based on the pixel motion between adjacent frames to predict the position changes of dynamic objects in consecutive frames, and then accurately track the speed and direction of the object. For example, in video surveillance, optical flow can be used to identify whether a vehicle is speeding or a pedestrian is moving abnormally. Therefore, based on the results of the semantic segmentation unit 301, dynamic objects are specially processed to improve the stitching quality.

[0090] The seam elimination unit 303 is configured to use a Poisson blending algorithm to eliminate illumination and geometric differences at the seams, thereby achieving seamless splicing.

[0091] Based on the result of the dynamic object processing unit 302, the stitching seams are post-processed to improve the visual effect of the panoramic video.

[0092] Poisson blending transforms the seam elimination problem into solving the Poisson equation, achieving a natural transition by minimizing the difference in gradients on both sides of the seam. Compared with traditional methods (such as linear blending and feathering), Poisson blending achieves a more natural transition by aligning gradients, avoiding blur at the seams.

[0093] The three-dimensional reconstruction module 4 is used to reconstruct a three-dimensional scene from the stitched panoramic video;

[0094] Specifically include:

[0095] The depth estimation unit 401 is used to generate a dense depth map based on a multi-view stereo vision algorithm, fuse IMU data to optimize the depth estimation result, and improve the accuracy of depth estimation;

[0096] Provide depth information to the mesh generation unit 402 and the semantic fusion unit 403 to guide the 3D reconstruction process.

[0097] The mesh generation unit 402 uses the Marching Cubes algorithm to extract isosurfaces from the depth map to generate a three-dimensional mesh model, and simultaneously applies a mesh simplification algorithm to reduce the number of faces and improve rendering efficiency;

[0098] Based on the result of the depth estimation unit 401, a three-dimensional mesh model is generated to provide a basis for subsequent processing.

[0099] The semantic fusion unit 403 is used to map semantic labels to the three-dimensional mesh surface and optimize the mesh topology based on semantic information.

[0100] Based on the results of the mesh generation unit 402 and the semantic segmentation unit 301, the fusion of semantics and geometry is achieved to improve the quality of the three-dimensional reconstructed model.

[0101] The model optimization module 5 is used to optimize the reconstructed three-dimensional mesh model.

[0102] Specifically include:

[0103] The texture mapping unit 501 is used to select the texture with the most normal viewing angle and the highest resolution for mapping each mesh surface;

[0104] The real-time rendering unit 502 is provided with a three-dimensional model with high-quality textures to improve visual effects.

[0105] The real-time rendering unit 502 is used to utilize the parallel computing capability of the GPU to accelerate texture mapping and lighting calculations, and dynamically adjust the model detail level according to the distance from the camera 7 to eliminate invisible faces.

[0106] Based on the result of the texture mapping unit 501, the three-dimensional model is rendered in real time to provide a smooth visual experience.

[0107] See also Figures 9 to 15 The present invention also provides a video panoramic stitching and three-dimensional fusion device, including a mounting plate 6, multiple cameras 7 and multiple camera 7 adjustment components, the camera 7 adjustment components include an arcuate slide rail 8, an arcuate rack 9, a first gear 10, a first gear driving component 11, a sliding shell 12, a quick-release shell 13 and a quick-release unit, the quick-release unit includes two locking mechanisms, an unlocking rotating shell 15, a U-shaped block 16, a connecting shaft 17, two linkage mechanisms, a pop-up plate 18, a pop-up spring 19 and a pop-up telescopic rod 20, the locking mechanism includes a locking plate 23, multiple locking oblique blocks 24, multiple locking telescopic rods 25 and multiple locking springs 26, the linkage mechanism includes a push plate 28, multiple push blocks 29, a dial rod 30, a rotating plate 31, a linkage rod 32, a linkage rotating block 33, a second gear 34, a linkage rack 35 and a return spring 36.

[0108] Among them, multiple camera 7 adjustment components are arranged on the mounting plate 6 in sequence, and each camera 7 is installed on the corresponding camera 7 adjustment component; the arcuate slide rail 8 is arranged on one side of the mounting plate 6, and the arcuate rack 9 is arranged inside the arcuate slide rail 8, the sliding shell 12 is slidingly connected to the arcuate slide rail 8, and the sliding shell 12 has a groove 14, the first gear drive component 11 is arranged inside the groove 14, the output end of the first gear drive component is fixedly connected to the first gear 10, the first gear 10 and the arcuate rack 9 are engaged with each other, the quick-release shell 13 is connected to the sliding shell 12 through the quick-release unit, and the camera 7 is arranged on one side of the quick-release shell 13. The camera 7 adjustment assembly is used to install and adjust the position angle of the camera 7, and the mounting plate 6 is used to be installed in an indoor or outdoor area where video needs to be shot; the first gear drive component 11 is a self-locking motor, and the first gear drive component 11 is started to drive the first gear 10 to rotate, and then cooperate with the arc-shaped rack 9, thereby driving the sliding shell 12 to slide in the arc-shaped slide rail 8. The quick-release shell 13 can be quickly installed and disassembled through the quick-release unit, thereby facilitating maintenance and replacement of the camera 7.

[0109] Secondly, the two locking mechanisms are symmetrically arranged inside the quick-release housing 13, and the unlocking rotating shell 15 is sleeved on the outside of the camera 7. The quick-release housing 13 has two arc-shaped grooves 21, and the two ends of the U-shaped block 16 respectively pass through the corresponding arc-shaped grooves 21 and are fixedly connected to the unlocking rotating shell 15. The connecting shaft 17 is arranged inside the quick-release housing 13, and one side of the U-shaped block 16 is fixedly connected to the connecting shaft 17. The two linkage mechanisms are symmetrically arranged on both sides of the connecting shaft 17. The quick-release housing 13 also has a slot 22, and the two ends of the pop-up telescopic rod 20 are respectively fixedly connected to the inner wall of the slot 22 and the pop-up plate 18, and the two ends of the pop-up spring 19 are respectively movably connected to the inner wall of the slot 22 and the pop-up plate 18, and the pop-up spring 19 is sleeved on the outside of the pop-up telescopic rod 20. When the camera 7 is quickly disassembled, the unlocking rotating shell 15 is rotated to drive the U-shaped block 16 to slide in the arc groove 21, and at the same time drive the connecting shaft 17 to rotate. At this time, the connecting shaft 17 drives the linkage mechanism to operate, and then drives the locking mechanism to unlock. After unlocking, the pop-up spring 19 rebounds, driving the pop-up plate 18 to move. At the same time, the pop-up telescopic rod 20 maintains the stability of the pop-up plate 18, and then the quick-release shell 13 is ejected from the sliding shell.

[0110] Finally, the plurality of locking bevel blocks 24 are fixedly connected to the locking plate 23, the two ends of the plurality of locking telescopic rods 25 are respectively fixedly connected to the inner wall of the sliding shell and the locking plate 23, the two ends of the plurality of locking springs 26 are respectively movably connected to the inner wall of the sliding shell and the locking plate 23, the locking springs 26 are sleeved on the outside of the locking telescopic rod 25, the quick-release shell 13 has a plurality of locking grooves 27, the locking bevel blocks 24 and the locking grooves 27 are adapted to each other; the plurality of push blocks 29 are respectively slidably connected to the corresponding locking grooves 27, and one end of the plurality of push blocks 29 is fixed to the push plate 28 The lever 30 is connected to the outside of the connecting shaft 17, and the other end of the lever 30 is located on one side of the rotating plate 31. The rotating plate 31 and the second gear 34 are both rotatably connected to the inside of the quick-release housing 13. The linkage block is provided on one side of the second gear 34. The two ends of the linkage rod 32 are respectively rotatably connected to the other side of the rotating plate 31 and the linkage block. The linkage rack 35 is provided on one side of the push plate 28. The linkage rack 35 is meshed with the second gear 34. The two ends of the return spring 36 are respectively movably connected to the rotating plate 31 and the inner wall of the quick-release housing 13. The connecting shaft 17 rotates, driving the shift rod 30 to move, so that the shift rod 30 contacts the inclined surface of the rotating plate 31, thereby pushing the rotating plate 31 to rotate, driving the linkage rod 32 to move, and the linkage rod 32 drives the linkage rotating block 33 to rotate, so that the second gear 34 rotates, and then engages with the linkage rack 35, driving the push plate 28 and the plurality of push blocks 29 to move, and the push block 29 slides in the locking groove 27, thereby pushing out the locking oblique block 24, thereby completing the unlocking, and the Quick release housing 13. In addition, after unlocking is completed, the reset spring 36 can drive the rotating plate 31 to reset so that it can be used repeatedly later. In addition, when installing the quick release housing 13, the quick release housing 13 enters the sliding housing, pushing the inclined surface of the locking bevel 24, and then causing it to rebound. After the quick release housing 13 completely enters the sliding housing, the locking spring 26 rebounds, driving the locking plate 23 to move, so that the locking bevel 24 enters the locking groove 27, and at the same time the locking telescopic rod 25 maintains movement stability.

[0111] When using a video panoramic stitching and three-dimensional fusion system of the present embodiment, when shooting the images required for video panoramic stitching and three-dimensional fusion, the mounting plate 6 is fixed to the area to be shot. At this time, the camera 7 is installed first, the quick-release shell 13 is inserted into the sliding shell, and the inclined surface of the locking bevel 24 is pushed to make it rebound. After the quick-release shell 13 completely enters the sliding shell, the locking spring 26 rebounds, driving the locking plate 23 to move, so that the locking bevel 24 enters the locking groove 27, and at the same time the locking telescopic rod 25 maintains movement stability, thereby completing the installation of the camera 7, and then starting the camera 7 to shoot the images; when it is required When maintaining or replacing the camera 7 of other precision, the unlocking rotating shell 15 is rotated to drive the U-shaped block 16 to slide in the arc groove 21, and at the same time drive the connecting shaft 17 to rotate. At this time, the connecting shaft 17 drives the linkage mechanism to operate, and then drives the locking mechanism to unlock. After unlocking, the pop-up spring 19 rebounds, driving the pop-up plate 18 to move. At the same time, the pop-up telescopic rod 20 maintains the stability of the pop-up plate 18, and then the quick-release shell 13 is ejected from the sliding shell; thereby, the camera 7 can be quickly installed and adjusted, which significantly improves the use efficiency and user experience, increases the shooting effect, and is more convenient for video panoramic stitching and three-dimensional fusion.

[0112] The above disclosure is merely one or more preferred embodiments of the present application and is not intended to limit the scope of the present application. A person skilled in the art will understand that all or part of the processes of the above embodiments and equivalent changes made in accordance with the claims of the present application are still within the scope of the present application.

Claims

1. A video panoramic stitching and 3D fusion system, characterized by: It includes an image acquisition module, an image correction module, a splicing optimization module, a 3D reconstruction module and a model optimization module, wherein the image acquisition module, the image correction module, the splicing optimization module, the 3D reconstruction module and the model optimization module are connected in sequence; The picture acquisition module is used to acquire video pictures from all directions; The image correction module is used to eliminate noise and errors in the collected video images; The splicing optimization module is used to splice the video streams collected by multiple cameras into a panoramic video and optimize the splicing points; The 3D reconstruction module is used to reconstruct a 3D scene from the stitched panoramic video; The model optimization module is used to optimize the reconstructed three-dimensional mesh model.

2. The video panorama stitching and 3D fusion system according to claim 1, characterized in that: The picture acquisition module includes a camera deployment unit, a camera adjustment unit and an image transmission unit, and the camera deployment unit, the camera adjustment unit and the image transmission unit are connected in sequence; The camera deployment unit is used to select a high-precision camera model and determine the camera installation position and angle to ensure coverage of the target scene; The camera adjustment unit is used to adjust the exposure time, white balance and focus parameters of the camera; The image transmission unit is used to transmit the captured images to the picture correction module via a network.

3. The video panorama stitching and 3D fusion system according to claim 2, characterized in that: The picture correction module includes an image preprocessing unit, a spatiotemporal synchronization unit and a distortion correction unit, wherein the image preprocessing unit, the spatiotemporal synchronization unit and the distortion correction unit are connected in sequence; The image preprocessing unit is used to perform preprocessing operations of denoising and enhancing the video image; The spatiotemporal synchronization unit is used to keep the video images captured by multiple cameras consistent in time and space; The distortion correction unit is used to eliminate the influence of camera lens distortion on the image.

4. The video panoramic stitching and 3D fusion system according to claim 3, characterized in that: The spatiotemporal synchronization unit includes a hardware triggering subunit and a software compensation subunit, and the hardware triggering subunit and the software compensation subunit are connected; The hardware trigger subunit is used to generate a synchronization pulse signal using FPGA to trigger all cameras to expose and ensure time synchronization; The software compensation subunit is used to dynamically model network transmission delay and align asynchronous frames using an interpolation algorithm to improve time synchronization accuracy; The distortion correction unit includes an offline calibration subunit and an online calibration subunit, and the offline calibration subunit is connected to the online calibration subunit; The offline calibration subunit is used to obtain camera parameters using a checkerboard calibration plate and establish a spatial mapping relationship between cameras; The online calibration subunit is used to apply the undistort() function of OpenCV to correct image distortion in real time and improve image quality.

5. The video panorama stitching and 3D fusion system according to claim 4, characterized in that: The splicing optimization module includes a semantic segmentation unit, a dynamic object processing unit and a gap elimination unit, and the semantic segmentation unit, the dynamic object processing unit and the gap elimination unit are connected in sequence; The semantic segmentation unit is used to build a segmentation network based on MobileNetV3-Small, perform pixel-level semantic segmentation on the image, and identify different objects, including but not limited to pedestrians, vehicles and buildings. The dynamic object processing unit is used to adopt a multi-view fusion strategy for the segmented dynamic objects, apply the optical flow method to track the object's motion trajectory, optimize the splicing weight, and avoid overlap or breakage at the splicing seams; The seam elimination unit is used to use a Poisson fusion algorithm to eliminate illumination and geometric differences at the seams to achieve seamless splicing.

6. The video panoramic stitching and 3D fusion system according to claim 5, characterized in that: The three-dimensional reconstruction module includes a depth estimation unit, a grid generation unit and a semantic fusion unit, which are connected in sequence; The depth estimation unit is used to generate a dense depth map based on a multi-view stereo vision algorithm, fuse IMU data to optimize the depth estimation result, and improve the accuracy of depth estimation; The mesh generation unit uses the Marching Cubes algorithm to extract isosurfaces from the depth map to generate a three-dimensional mesh model, and simultaneously applies a mesh simplification algorithm to reduce the number of faces and improve rendering efficiency; The semantic fusion unit is used to map semantic labels to the three-dimensional grid surface and optimize the grid topology based on semantic information.

7. The video panoramic stitching and 3D fusion system according to claim 6, characterized in that: The model optimization module includes a texture mapping unit and a real-time rendering unit, and the texture mapping unit is connected to the real-time rendering unit; The texture mapping unit is used to select the texture with the most correct viewing angle and the highest resolution for mapping each mesh surface; The real-time rendering unit is used to utilize the parallel computing capability of the GPU to accelerate texture mapping and lighting calculations, and dynamically adjust the model detail level according to the camera distance to eliminate invisible faces.

8. A video panoramic stitching and 3D fusion device, using the video panoramic stitching and 3D fusion system according to claim 7, characterized in that: It includes a mounting plate, a plurality of cameras and a plurality of camera adjustment components, wherein the plurality of camera adjustment components are sequentially arranged on the mounting plate, and each of the cameras is mounted on a corresponding camera adjustment component; The camera adjustment assembly includes an arcuate slide rail, an arcuate rack, a first gear, a first gear driving component, a sliding shell, a quick-release shell and a quick-release unit. The arcuate slide rail is arranged on one side of the mounting plate, the arcuate rack is arranged inside the arcuate slide rail, the sliding shell is slidingly connected to the arcuate slide rail, the sliding shell has a groove, the first gear driving component is arranged inside the groove, the output end of the first gear driving component is fixedly connected to the first gear, the first gear and the arcuate rack are engaged with each other, the quick-release shell is connected to the sliding shell through the quick-release unit, and the camera is arranged on one side of the quick-release shell.

9. The video panoramic stitching and 3D fusion system according to claim 8, characterized in that: The quick-release unit includes two locking mechanisms, an unlocking rotating shell, a U-shaped block, a connecting shaft, two linkage mechanisms, a pop-up plate, a pop-up spring and a pop-up telescopic rod. The two locking mechanisms are symmetrically arranged inside the quick-release shell, and the unlocking rotating shell is sleeved on the outside of the camera. The quick-release shell has two arc grooves, and the two ends of the U-shaped block respectively pass through the corresponding arc grooves and are fixedly connected to the unlocking rotating shell. The connecting shaft is arranged inside the quick-release shell, one side of the U-shaped block is fixedly connected to the connecting shaft, and the two linkage mechanisms are symmetrically arranged on both sides of the connecting shaft. The quick-release shell also has a slot, and the two ends of the pop-up telescopic rod are respectively fixedly connected to the inner wall of the slot and the pop-up plate, and the two ends of the pop-up spring are respectively movably connected to the inner wall of the slot and the pop-up plate, and the pop-up spring is sleeved on the outside of the pop-up telescopic rod.

10. The video panoramic stitching and 3D fusion system according to claim 9, characterized in that: The locking mechanism includes a locking plate, a plurality of locking bevel blocks, a plurality of locking telescopic rods and a plurality of locking springs, wherein the plurality of locking bevel blocks are fixedly connected to the locking plate, the two ends of the plurality of locking telescopic rods are respectively fixedly connected to the inner wall of the sliding housing and the locking plate, the two ends of the plurality of locking springs are respectively movably connected to the inner wall of the sliding housing and the locking plate, the locking springs are sleeved on the outside of the locking telescopic rod, the quick-release housing has a plurality of locking grooves, and the locking bevel blocks and the locking grooves are adapted to each other; The linkage mechanism includes a push plate, multiple push blocks, a shift rod, a rotating plate, a linkage rod, a linkage rotating block, a second gear, a linkage rack and a return spring, multiple push blocks are respectively slidably connected to the corresponding locking grooves, multiple and one end of the push blocks are fixedly connected to the push plate, one end of the shift rod is arranged on the outside of the connecting shaft, the other end of the shift rod is located on one side of the rotating plate, the rotating plate and the second gear in the zone are both rotatably connected to the inside of the quick-release housing, the linkage block is arranged on one side of the second gear, two ends of the linkage rod are respectively rotatably connected to the other side of the rotating plate and the linkage block, the linkage rack is arranged on one side of the push plate, the linkage rack and the second gear are meshed with each other, and the two ends of the return spring are respectively movably connected to the rotating plate and the inner wall of the quick-release housing.

Images