Panoramic image acquisition system

Abstract

The invention discloses a panoramic image acquisition system, relates to the technical field of panoramic image acquisition, and solves the problems that splicing dislocation or image blurring is likely to occur during panoramic image acquisition when illumination changes drastically or interference is caused by a moving object, and equipment is large in size and difficult to adapt to an unmanned aerial vehicle for use due to the fact that a multi-camera layout is generally adopted. The invention relates to a panoramic image acquisition system, which comprises an unmanned aerial vehicle module, an inertial navigation auxiliary module, a multispectral image acquisition module, an adjusting structure, an edge calculation module, a self-adaptive transmission module and a distributed power supply management module, and two multispectral image acquisition modules are arranged in the inertial navigation auxiliary module. According to the invention, through the cooperation of the multispectral image acquisition module and the edge calculation module and through the infrared sensor and the dynamic mask technology, high-robustness image splicing is realized, the interference of a moving object is effectively inhibited, and the splicing accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of panoramic image acquisition, and in particular to a panoramic image acquisition system. Background Art

[0002] Panoramic photography is to stitch all the multiple pictures into one panoramic picture. Its basic shooting principle is to search for the edge of two pictures and overlap the area with the closest imaging effect to complete the automatic stitching of pictures; When taking graduation photos, drones are usually used to capture panoramic views.

[0003] The existing panoramic image acquisition is prone to splicing misalignment or image blur when the lighting changes drastically or there is interference from moving objects. In addition, the multi-camera layout is usually used, which makes the equipment bulky and difficult to adapt to drones. Therefore, it does not meet the existing needs. In this regard, we propose a panoramic image acquisition system. Summary of the invention

[0004] The purpose of the present invention is to provide a panoramic image acquisition system to solve the problems raised in the above background technology that panoramic image acquisition is prone to splicing misalignment or image blur when the lighting changes drastically or there is interference from moving objects, and that the multi-camera layout usually adopts a large device size and is difficult to adapt to use with drones.

[0005] To achieve the above-mentioned objectives, the present invention provides the following technical solutions: a panoramic image acquisition system, comprising a drone module, an inertial navigation auxiliary module, a multispectral image acquisition module, an adjustment structure, an edge computing module, an adaptive transmission module and a distributed power management module, an inertial navigation auxiliary module being provided at the bottom of the drone module, and two multispectral image acquisition modules being provided inside the inertial navigation auxiliary module.

[0006] Preferably, the multispectral image acquisition module includes an acquisition camera and an ultra-wide-angle lens. The ultra-wide-angle lens is provided on the outside of the acquisition camera. The ultra-wide-angle lens is integrated with an RGB sensor, an infrared sensor and a dynamic exposure control unit. The dynamic exposure control unit uses the RGB sensor and the infrared sensor to adjust the exposure parameters of the ultra-wide-angle lens in real time to ensure the consistency of multi-viewing angle lighting.

[0007] Preferably, the edge computing module includes a CPU and NPU heterogeneous computing architecture, and the edge computing module integrates a distortion correction unit, a feature fusion and splicing unit, and a dynamic mask generation unit.

[0008] Preferably, the distortion correction unit adopts a polynomial fitting model to correct wide-angle distortion in real time, the feature fusion and stitching unit adopts ORB and SuperPoint mixed feature descriptors to achieve multi-image feature matching, and introduces motion compensated IMU data to optimize stitching parameters, and the dynamic mask generation unit uses infrared sensor data to identify moving objects and generate dynamic area masks to avoid stitching errors caused by motion blur.

[0009] Preferably, the adaptive transmission module supports adaptive adjustment of resolution based on network status and is equipped with a chip supporting H. encoding.

[0010] Preferably, the distributed power management module is internally equipped with a solar auxiliary charging unit, adopts a time-sharing power supply strategy, and dynamically allocates power according to the working status of other modules.

[0011] Preferably, the inertial navigation auxiliary module includes a rotating disk, a six-axis IMU sensor, a rotating shaft, a connecting frame, a rotating gimbal and a camera rotating slot, a six-axis IMU sensor is provided at the top of the rotating disk, a rotating shaft is rotatably installed inside the six-axis IMU sensor, a connecting frame is fixedly installed at the bottom end of the rotating disk, a rotating gimbal is movably installed at the bottom end of the connecting frame, camera rotating slots are provided on both sides of the rotating gimbal, and a motion compensation algorithm is installed inside the inertial navigation auxiliary module, and the motion compensation algorithm is used to eliminate device jitter to avoid affecting image stitching.

[0012] Preferably, the adjustment structure includes a driven gear, a mounting frame, an active rack, a T-shaped sliding rod, a T-shaped sliding groove, a connecting plate, an electric push rod, an assembly frame, a fixing plate, a placement table, an L-shaped fixing rod, an L-shaped slide plate and a lens mounting hole, two driven gears are movably installed on both sides of the rotating pan-tilt head, mounting frames are fixedly installed on both sides of the rotating pan-tilt head, the mounting frames are rotatably connected to the driven gears, fixing plates are fixedly installed on the inner sides of the mounting frames, the placement table is fixedly installed on the bottom ends of the fixing plates, the placement table is fixedly installed on the bottom ends of the placement tables, the L-shaped fixing rods are movably installed on the bottom ends of the placement tables, the L-shaped slide plates are movably installed, and the L-shaped fixing rods are slidably inserted into the The invention relates to a method for realizing the present invention of a rotary pan-tilt platform and a rotary pan-tilt platform, wherein the rotary pan-tilt platform is provided with a plurality of movable members, the movable members are inserted into the L-shaped slide plate, the surface of the L-shaped slide plate is provided with a lens mounting hole, the ultra-wide-angle lens moves through the lens mounting hole, the inner sides of the two driven gears on the same side are meshed and connected with active racks, the two sides of the rotary pan-tilt platform close to the active racks are provided with T-shaped slide grooves, the surface of the active racks are fixedly installed with T-shaped slide bars, the T-shaped slide bars are slidably inserted into the T-shaped slide grooves, the bottom ends of the active racks are fixedly installed with connecting plates, the two sides of the rotary pan-tilt platform close to the active racks are fixedly installed with assembly frames, the interiors of the assembly frames are provided with electric push rods, and the output ends of the electric push rods are fixedly connected with the connecting plates.

[0013] A method for collecting panoramic images by a system comprises the following steps: S1: The multispectral image acquisition module synchronously captures the surrounding panoramic RGB and infrared images, and the six-axis IMU sensor collects the speed and acceleration information during image shooting; S2: Input the data and image data collected by the six-axis IMU sensor into the edge computing module for motion compensation and distortion correction; S3: Generate a distortion-free panoramic image by combining static scene features and dynamic masks through a feature fusion and stitching unit; S4: The adaptive transmission module selects local storage or real-time streaming output according to user needs.

[0014] Preferably, step S1 also includes the following steps: S1.1: First, integrate the inertial navigation auxiliary module with the bottom of the drone module, and install two multispectral image acquisition modules in the inertial navigation auxiliary module. The ultra-wide-angle lens is first installed vertically downward, and its rotation is controlled by the adjustment structure so that it can cover a 360°×150° field of view; S1.2: The data collected by the six-axis IMU sensor is used to compensate for the change in the drone's attitude in real time and dynamically adjust the stitching reference surface; S1.3: Preload the deep learning model in the CPU and NPU heterogeneous computing architecture to cooperate with the feature fusion and splicing unit for feature matching.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention realizes highly robust image stitching through the cooperation of a multispectral image acquisition module and an edge computing module, through infrared sensors and dynamic masking technology, effectively suppresses the interference of moving objects, and improves the stitching accuracy.

[0016] 2. The present invention reduces processing delay through the CPU and NPU heterogeneous computing architecture to meet the real-time requirements of the drone module and the mobile terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the overall structure of the present invention; Figure 2 A cross-sectional front view of the present invention as a whole; Figure 3 A sectional side view of the present invention as a whole; Figure 4 A cross-sectional top view of the present invention as a whole; Figure 5 For the present invention as a whole Figure 2 Schematic diagram of the local structure of part A; Figure 6is a flow chart of the panoramic image acquisition method of the present invention; Figure 7 is a flow chart of step S1 in the panoramic image acquisition method of the present invention; Figure 8 It is a structural schematic diagram of the panoramic image acquisition system of the present invention.

[0018] In the figure: 1. Inertial navigation auxiliary module; 101. Rotating disk; 102. Six-axis IMU sensor; 103. Rotating shaft; 104. Connecting frame; 105. Rotating gimbal; 106. Camera rotation slot; 2. Multispectral image acquisition module; 201. Acquisition camera; 202. Ultra-wide-angle lens; 3. Adjustment structure; 301. Driven gear; 302. Mounting frame; 303. Active rack; 304. T-shaped slide bar; 305. T-shaped slide slot; 306. Connecting plate; 307. Electric push rod; 308. Assembly frame; 309. Fixed plate; 310. Placement table; 311. L-shaped fixing rod; 312. L-shaped slide plate; 313. Lens mounting hole. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0020] See also Figures 1 to 8 An embodiment of the present invention is as follows: a panoramic image acquisition system includes a drone module, an inertial navigation auxiliary module 1, a multispectral image acquisition module 2, an adjustment structure 3, an edge computing module, an adaptive transmission module and a distributed power management module. An inertial navigation auxiliary module 1 is provided at the bottom of the drone module, and two multispectral image acquisition modules 2 are provided inside the inertial navigation auxiliary module 1.

[0021] The multispectral image acquisition module 2 includes an acquisition camera 201 and an ultra-wide-angle lens 202. The ultra-wide-angle lens 202 is provided on the outside of the acquisition camera 201. The ultra-wide-angle lens 202 is integrated with an RGB sensor, an infrared sensor and a dynamic exposure control unit. The dynamic exposure control unit uses the RGB sensor and the infrared sensor to adjust the exposure parameters of the ultra-wide-angle lens 202 in real time to ensure the consistency of multi-viewing angle lighting.

[0022] The edge computing module includes a CPU and NPU heterogeneous computing architecture, and the edge computing module integrates a distortion correction unit, a feature fusion and splicing unit, and a dynamic mask generation unit.

[0023] The distortion correction unit uses a polynomial fitting model to correct wide-angle distortion in real time. The feature fusion and stitching unit uses ORB and SuperPoint mixed feature descriptors to achieve multi-image feature matching, and introduces motion-compensated IMU data to optimize stitching parameters. The dynamic mask generation unit uses infrared sensor data to identify moving objects and generates dynamic area masks to avoid stitching errors caused by motion blur.

[0024] The adaptive transmission module supports adaptive adjustment of resolution based on network status and is equipped with a chip that supports H.265 encoding.

[0025] The distributed power management module is equipped with a solar auxiliary charging unit, adopts a time-sharing power supply strategy, and dynamically allocates power according to the working status of other modules.

[0026] The inertial navigation auxiliary module 1 includes a rotating disk 101, a six-axis IMU sensor 102, a rotating shaft 103, a connecting frame 104, a rotating gimbal 105 and a camera rotating slot 106. The six-axis IMU sensor 102 is provided at the top of the rotating disk 101, and the rotating shaft 103 is rotatably installed inside the six-axis IMU sensor 102. The bottom end of the rotating disk 101 is fixedly installed with a connecting frame 104, and the bottom end of the connecting frame 104 is movably installed with a rotating gimbal 105. Camera rotating slots 106 are provided on both sides of the rotating gimbal 105. The inertial navigation auxiliary module 1 is equipped with a motion compensation algorithm, and the motion compensation algorithm is used to eliminate device jitter to avoid affecting image stitching.

[0027] The adjusting structure 3 includes a driven gear 301, a mounting frame 302, an active rack 303, a T-shaped slide bar 304, a T-shaped slide groove 305, a connecting plate 306, an electric push rod 307, an assembly frame 308, a fixing plate 309, a placement table 310, an L-shaped fixing rod 311, an L-shaped slide plate 312 and a lens mounting hole 313. Two driven gears 301 are movably installed on both sides of the rotating pan-tilt head 105, and mounting frames 302 are fixedly installed on both sides of the rotating pan-tilt head 105. The mounting frames 302 are rotatably connected to the driven gear 301. The inner side of the mounting frame 302 is fixedly installed with a fixing plate 309, and the bottom end of the fixing plate 309 is fixedly installed with a placement table 310. The bottom end of the placement table 310 is fixedly installed with an L-shaped fixing rod 311, and the bottom end of the placement table 310 is movably installed with an L-shaped slide plate 312. The L-shaped fixing rod 311 are all slidably inserted into the interior of the L-shaped slide plate 312, and a lens mounting hole 313 is provided on the surface of the L-shaped slide plate 312. The ultra-wide-angle lens 202 moves through the interior of the lens mounting hole 313. The inner sides of the two driven gears 301 located on the same side are meshed and connected with the active rack 303. T-shaped slide grooves 305 are provided on both sides of the rotating pan-tilt head 105 close to the active rack 303. T-shaped slide bars 304 are fixedly installed on the surface of the active rack 303. The T-shaped slide bars 304 are slidably inserted into the interior of the T-shaped slide grooves 305. The bottom end of the active rack 303 is fixedly installed with a connecting plate 306. The rotating pan-tilt head 105 is fixedly installed with an assembly frame 308 on both sides close to the active rack 303. Electric push rods 307 are provided inside the assembly frame 308, and the output ends of the electric push rods 307 are fixedly connected to the connecting plate 306.

[0028] A method for collecting panoramic images by a system comprises the following steps: S1: The surrounding panoramic RGB and infrared images are synchronously captured by the multispectral image acquisition module 2, and the speed, acceleration and other information when the image is taken are collected by the six-axis IMU sensor 102; S2: Input the data and image data collected by the six-axis IMU sensor 102 into the edge computing module for motion compensation and distortion correction; S3: Generate a distortion-free panoramic image by combining static scene features and dynamic masks through a feature fusion and stitching unit; S4: The adaptive transmission module selects local storage or real-time streaming output according to user needs.

[0029] Step S1 also includes the following steps: S1.1: First, the inertial navigation auxiliary module 1 is integrated with the bottom of the drone module, and two multispectral image acquisition modules 2 are installed in the inertial navigation auxiliary module 1, and the ultra-wide-angle lens 202 is first installed vertically downward, and its rotation is controlled by the adjustment structure 3 so that it can cover a 360°×150° field of view; S1.2: using the data collected by the six-axis IMU sensor 102 to compensate for the change in the attitude of the drone in real time, and dynamically adjust the stitching reference surface; S1.3: Preload the deep learning model in the CPU and NPU heterogeneous computing architecture to cooperate with the feature fusion and splicing unit for feature matching.

[0030] When the panoramic image acquisition system is in use, the inertial navigation auxiliary module 1 is first installed at the bottom of the drone module, and the multispectral image acquisition module 2 is installed inside the camera rotation slot 106. At the same time, the L-shaped slide plate 312 at the bottom of the placement table 310 can be pulled to adjust the position of the lens mounting hole 313 to accommodate different sizes of acquisition cameras 201 for placement, and the ultra-wide-angle lens 202 is inserted into the lens mounting hole 313. Then, the output end is pushed to move by the electric push rod 307, so that it drives the active rack 303 to move inside the T-shaped slide slot 305, so that the active rack 303 The driven gears 301 on both sides are driven to rotate. At this time, the driven gears 301 will drive the fixed plate 309 to rotate, thereby driving the acquisition camera 201 to rotate, so that the initial orientation direction of the ultra-wide-angle lens 202 is perpendicular to the ground, so that when the device is performing panoramic image acquisition, the driven gears 301 can be driven to rotate by the active rack 303, so as to adjust the position of the acquisition camera 201, and then adjust the angle of the ultra-wide-angle lens 202, and the rotating shaft 103 drives the rotating pan-tilt head 105 to rotate, so as to adjust the angle of the acquisition camera 201, so that the ultra-wide-angle lens 202 can cover a 360°×150° field of view; Then, the surrounding panoramic RGB and infrared images are synchronously captured by the multispectral image acquisition module 2, and the speed, acceleration and other information when the image is taken are collected by the six-axis IMU sensor 102; The data and image data collected by the six-axis IMU sensor 102 are then input into the edge computing module for motion compensation and distortion correction; The feature fusion and stitching unit combines static scene features with dynamic masks to generate distortion-free panoramic images; The adaptive transmission module can be used to select local storage or real-time streaming output according to user needs.

[0031] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A panoramic image acquisition system, characterized in that: The invention comprises a drone module, an inertial navigation auxiliary module (1), a multispectral image acquisition module (2), an adjustment structure (3), an edge computing module, an adaptive transmission module and a distributed power management module, wherein the bottom of the drone module is provided with an inertial navigation auxiliary module (1), and the interior of the inertial navigation auxiliary module (1) is provided with two multispectral image acquisition modules (2).

2. A panoramic image acquisition system according to claim 1, characterized in that: The multispectral image acquisition module (2) comprises an acquisition camera (201) and an ultra-wide-angle lens (202); the ultra-wide-angle lens (202) is provided on the outside of the acquisition camera (201); an RGB sensor, an infrared sensor and a dynamic exposure control unit are integrated inside the ultra-wide-angle lens (202); the dynamic exposure control unit uses the RGB sensor and the infrared sensor to adjust the exposure parameters of the ultra-wide-angle lens (202) in real time, thereby ensuring the consistency of multi-viewing angle illumination.

3. A panoramic image acquisition system according to claim 1, characterized in that: The edge computing module includes a CPU and NPU heterogeneous computing architecture, and the edge computing module integrates a distortion correction unit, a feature fusion and splicing unit, and a dynamic mask generation unit.

4. A panoramic image acquisition system according to claim 3, characterized in that: The distortion correction unit adopts a polynomial fitting model to correct wide-angle distortion in real time. The feature fusion and stitching unit adopts ORB and SuperPoint mixed feature descriptors to achieve multi-image feature matching, and introduces motion-compensated IMU data to optimize stitching parameters. The dynamic mask generation unit uses infrared sensor data to identify moving objects and generates dynamic area masks to avoid stitching errors caused by motion blur.

5. The panoramic image acquisition system according to claim 1, characterized in that: The adaptive transmission module supports adaptive adjustment of resolution based on network status and is equipped with a chip supporting H.265 encoding.

6. The panoramic image acquisition system according to claim 1, characterized in that: The distributed power management module is internally equipped with a solar auxiliary charging unit, adopts a time-sharing power supply strategy, and dynamically allocates power according to the working status of other modules.

7. The panoramic image acquisition system according to claim 2, characterized in that: The inertial navigation auxiliary module (1) comprises a rotating disk (101), a six-axis IMU sensor (102), a rotating shaft (103), a connecting frame (104), a rotating platform (105) and a camera rotating slot (106); the six-axis IMU sensor (102) is provided at the top of the rotating disk (101); the rotating shaft (103) is rotatably mounted inside the six-axis IMU sensor (102); the connecting frame (104) is fixedly mounted at the bottom of the rotating disk (101); the rotating platform (105) is movably mounted at the bottom of the connecting frame (104); camera rotating slots (106) are provided on both sides of the rotating platform (105); and a motion compensation algorithm is mounted inside the inertial navigation auxiliary module (1), and the motion compensation algorithm is used to eliminate device jitter to avoid affecting image stitching.

8. A panoramic image acquisition system according to claim 7, characterized in that: The adjustment structure (3) comprises a driven gear (301), a mounting frame (302), an active rack (303), a T-shaped slide bar (304), a T-shaped slide groove (305), a connecting plate (306), an electric push rod (307), an assembly frame (308), a fixing plate (309), a placement table (310), an L-shaped fixing rod (311), an L-shaped slide plate (312) and a lens mounting hole (313). Two driven gears (301) are movably mounted on both sides of the rotating pan-tilt platform (105). Mounting frames (302) are fixedly installed on both sides of the rotating pan-tilt platform (105), and the mounting frames (302) are rotatably connected to the driven gear (301). A fixing plate (309) is fixedly installed on the inner side of the mounting frame (302), and a placement platform (310) is fixedly installed on the bottom end of the fixing plate (309). An L-shaped fixing rod (311) is fixedly installed on the bottom end of the placement platform (310), and an L-shaped sliding plate (312) is movably installed on the bottom end of the placement platform (310). The fixing rods (311) are slidably inserted into the interior of the L-shaped slide plate (312); a lens mounting hole (313) is provided on the surface of the L-shaped slide plate (312); the ultra-wide-angle lens (202) is movable through the interior of the lens mounting hole (313); the inner sides of the two driven gears (301) located on the same side are meshedly connected with active racks (303); the two sides of the rotating pan-tilt platform (105) close to the active racks (303) are provided with T-shaped slide grooves (305); the surface of the active racks (303) is The surfaces are fixedly installed with T-shaped slide bars (304), and the T-shaped slide bars (304) are slidably inserted into the interior of the T-shaped slide grooves (305). The bottom ends of the active racks (303) are fixedly installed with connecting plates (306). Both sides of the rotating platform (105) close to the active racks (303) are fixedly installed with assembly racks (308), and the interiors of the assembly racks (308) are provided with electric push rods (307), and the output ends of the electric push rods (307) are fixedly connected to the connecting plates (306).

9. A method for collecting panoramic images according to any one of claims 1 to 8, characterized in that: The following steps are involved: S1: synchronously capture the surrounding panoramic RGB and infrared images through the multi-spectral image acquisition module (2), and collect the speed and acceleration information when the image is taken through the six-axis IMU sensor (102); S2: inputting the data and image data collected by the six-axis IMU sensor (102) into the edge computing module to perform motion compensation and distortion correction; S3: Generate a distortion-free panoramic image by combining static scene features and dynamic masks through a feature fusion and stitching unit; S4: The adaptive transmission module selects local storage or real-time streaming output according to user needs.

10. The acquisition method of a panoramic image acquisition system according to claim 9, characterized in that: Step S1 also includes the following steps: S1.1: First, an inertial navigation auxiliary module (1) is integrated with the bottom of the drone module, and two multispectral image acquisition modules (2) are installed in the inertial navigation auxiliary module (1), and the ultra-wide-angle lens (202) is first installed vertically downward, and the adjustment structure (3) controls its rotation so that it can cover a 360°×150° field of view; S1.2: using the data collected by the six-axis IMU sensor (102) to compensate for the change in the attitude of the drone in real time, and dynamically adjust the splicing reference surface; S1.3: Preload the deep learning model in the CPU and NPU heterogeneous computing architecture to cooperate with the feature fusion and splicing unit for feature matching.

Images