Aerial all-round panoramic system supporting remote control operation of unmanned aerial vehicle

Abstract

The invention discloses an aerial all-around panoramic system supporting remote control of an unmanned aerial vehicle, and aims to better display the flight situation of the unmanned aerial vehicle, support more efficient and accurate remote control of the unmanned aerial vehicle and realize virtual and real digital twinning of ground virtual simulation and aerial real flight. An aerial all-round panorama system scheme supporting remote control operation of the unmanned aerial vehicle is provided; according to the aerial all-around panoramic system of the unmanned aerial vehicle, environment video pictures at different angles around the aircraft are acquired through the image acquisition system composed of the plurality of cameras, and the airborne embedded image splicing mainboard performs primary splicing on the acquired pictures and then integrally transmits the spliced pictures to a ground station through the image transmission equipment; and the ground station carries out cutting, correction and other processing on the complete image containing the images of the multiple cameras, then splicing is carried out to obtain a mapping file, and finally a complete panoramic image containing an aircraft shape model is formed, so that the flight situation of the unmanned aerial vehicle is visually displayed, and efficient and accurate remote control on the unmanned aerial vehicle is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles (UAVs), and more specifically, to an aerial omnidirectional panoramic system that supports remote control of UAVs. Background Art

[0002] An omnidirectional panoramic system, also known as a panoramic monitoring system or a panoramic imaging system, is an advanced automotive safety technology applied to vehicle driving. It uses multiple cameras and image processing technologies to provide drivers with an all-round view, thereby enhancing driving safety and convenience. The panoramic system usually consists of multiple wide-angle cameras. The images captured by these cameras are processed and fused in real time to form a panoramic image or a bird's-eye view, showing all areas around the vehicle. Through the omnidirectional panoramic system, the driver can see a real-time image of the panoramic view including the vehicle on the in-vehicle display screen, including the blind spots around the vehicle, to achieve functions such as assisting in parking and safety reminders, which is beneficial to improving road safety and reducing traffic accidents. At the same time, the omnidirectional panoramic system can better achieve the integration of people, vehicles, and the environment, significantly enhancing the driver's driving experience and comfort.

[0003] UAVs have been developing rapidly in recent years and have been applied more and more widely. Various sensor devices such as high-definition cameras, infrared sensors, and radars carried on UAVs enable UAVs to not only obtain the necessary information for performing complex and precise tasks, but also enable staff to accurately grasp and real-time monitor the flight state of UAVs. However, the current display on the ground station is mainly traditional flight tracks and status data. Although it can also directly display the image data collected by the camera, generally it still cannot comprehensively and intuitively display the flight situation of the UAV in the environment and cannot support more efficient and accurate remote control of the UAV. Summary of the Invention

[0004] The present invention provides an aerial omnidirectional panoramic system that supports remote control of UAVs to solve the problem that the existing UAV image system generally still cannot comprehensively and intuitively display the flight situation of the UAV in the environment and cannot support more efficient and accurate remote control of the UAV.

[0005] To achieve the above object, the present invention provides the following technical solutions: An aerial omnidirectional panoramic system that supports remote control of UAVs, comprising an image acquisition system, an image processing system, a storage system, a transmission system, and a display operation system.

[0006] Preferably, the image acquisition system includes multiple camera video streams, and each of the cameras can realize real-time acquisition of video image information; the camera installation angle needs to avoid field of view obstruction as much as possible, and the camera installation needs to change the aerodynamic shape of the aircraft as little as possible; the camera adopts a mipi camera or a panoramic camera that outputs imaging based on a multi-lens stitching mode, or a combination of the two, and the field of view of the camera should satisfy that the sum of the horizontal field of view angles of the camera combination is greater than 360 degrees, and the sum of the vertical field of view angles is greater than 180 degrees, thereby realizing the acquisition of all-round image information of the environment.

[0007] Preferably, the image processing system includes an airborne embedded image splicing mainboard and a ground station. The airborne embedded image splicing mainboard can collect real-time video streams from the camera, perform edge splicing calculations, perform initial splicing of the collected images through hardware decoding, compress and encode the spliced ​​images in H.265 format through hardware encoding, and transmit the encoded video images to the ground station through an image transmission device.

[0008] Preferably, the hardware functions of the device include preprocessing, image stitching and panoramic synthesis. The preprocessing step includes image decoding, image cropping, picture correction, perspective, chromatic aberration, denoising, and contrast enhancement. The image stitching step generates a mapping file for use in stitching a panoramic picture, specifically including image registration, image transformation, image stitching, and image fusion. The panoramic synthesis step combines the three-dimensional shape model of the aircraft to form a complete panoramic picture that includes the aircraft shape model.

[0009] Preferably, the image registration step includes using a feature point matching algorithm to accurately establish a geometric correspondence between images to ensure accurate alignment of the images; the image transformation step performs an accurate geometric transformation on the image based on the calculated transformation matrix to perfectly connect it with the adjacent image; the image stitching step processes the pixel values ​​of the overlapping area to ensure the continuity and consistency of the image; the image fusion step uses a fusion algorithm to further eliminate the image splicing gaps so that the spliced ​​image presents a natural, smooth and coherent visual effect.

[0010] Preferably, the storage system includes a storage device, including an onboard embedded image stitching mainboard part and a ground station part. The storage device stores the original data collected by all cameras and the stitched and fused panoramic video to facilitate subsequent processing and analysis.

[0011] Preferably, the transmission system includes an airborne image transmission system and a ground station image transmission system. The airborne image transmission system receives data from an airborne embedded image stitching mainboard and synchronizes the data to the ground station image transmission system, thereby realizing communication interaction between the airborne embedded image stitching mainboard and the ground station. The airborne image transmission system has a sufficient effective transmission distance, a small transmission delay, and sufficient video image transmission capability.

[0012] Preferably, the display operating system includes a ground station for ground - end display operation, which conducts image display and settings and interactions with the surround - view panoramic system; it displays real - time live imaging, supports the selection operation of different viewing angles, supports historical return visits, and plays historical panoramic videos based on the videos stored in the storage system by selecting the date and time point.

[0013] Principle and beneficial effects of this technical solution: (1) The aerial surround - view panoramic system set in the present invention can better display the flight posture of the UAV. By setting this aerial surround - view panoramic system, customized presentation of real - time images of the panoramic view including the UAV can be achieved, helping users to remotely control the UAV more efficiently and accurately, realizing the virtual - real digital twin of ground virtual simulation and aerial real - flight, and thus improving the control efficiency of the UAV.

[0014] (2) In the image acquisition system of the present invention, image stabilization technology is used in image encoding and decoding to ensure the stability of the video frame captured during the movement of the UAV, avoid video jitter caused by the aircraft's shaking, and improve the video quality.

[0015] (3) The image stitching main board set in the present invention uses the pull - stream algorithm to obtain multi - path real - time video stream information. For the obtained video stream information, hardware decoding technology is used to decode the video data and perform primary stitching to form a complete frame containing multi - path camera images. Then, through hardware encoding technology, each obtained complete frame is compressed and encoded in the H.265 format, which can ensure the time synchronization of multi - path video images and improve the transmission efficiency.

[0016] (4) The present invention supports multiple ground - station devices to work simultaneously, can present a richer display of flight postures and multi - view observation images, realize the virtual - real digital twin of ground virtual simulation and aerial real - flight, and better support the efficient and accurate remote control of the UAV. Description of the Drawings

[0017] Figure 1 It is the architecture diagram of the UAV aerial surround - view panoramic system; Figure 2 It is the front - side view of the camera installation of the UAV aerial surround - view panoramic system; Figure 3 It is the rear - side view of the camera installation of the UAV aerial surround - view panoramic system; Figure 4 It is the original image information collected by the 6 - channel airborne camera; Figure 5 It is the scene information of different angles presented from the first perspective of the UAV; Figure 6 It is the scene information of different angles presented from the third perspective of the UAV; DETAILED DESCRIPTION

[0018] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments: Example:

[0019] like Figures 1 to 6 As shown, the present invention provides an aerial panoramic view system supporting remote control of unmanned aerial vehicles, including an image acquisition system, an image processing system, a storage system, a transmission system and a display operation system.

[0020] like Figure 1 As shown, the image acquisition system includes multiple camera video streams, each camera can realize real-time acquisition of video image information, the camera installation angle should avoid field of view obstruction as much as possible, and the camera installation should minimize the impact on the aircraft aerodynamic layout. The camera uses a mipi camera or a panoramic camera that outputs imaging based on a multi-lens stitching mode, or a combination of the two. The camera's field of view should satisfy the requirement that the sum of the horizontal field of view of the camera combination is greater than 360 degrees, and the sum of the vertical field of view is greater than 180 degrees.

[0021] like Figure 1 As shown, the image processing system includes an airborne embedded image stitching mainboard and a ground station. The airborne embedded image stitching mainboard can collect real-time video streams from the camera, perform edge stitching calculations, and stitch the collected images for the first time through hardware decoding. The stitched images are then compressed and encoded in H.265 format through hardware encoding, and the encoded video images are transmitted to the ground station through an image transmission device.

[0022] like Figure 1 As shown, the ground station is provided with equipment hardware, and the equipment hardware functions include preprocessing, image stitching and panoramic synthesis. The preprocessing steps include image decoding, image cropping, picture correction, perspective, chromatic aberration, denoising, and contrast enhancement. The image stitching step generates a mapping file for stitching panoramic pictures, which specifically includes image registration, image transformation, image stitching, and image fusion. The panoramic synthesis step combines the three-dimensional shape model of the aircraft to form a complete panoramic picture including the aircraft shape model.

[0023] like Figure 1 As shown, the image registration step includes using a feature point matching algorithm to accurately establish the geometric correspondence between images to ensure accurate alignment of the images; the image transformation step performs precise geometric transformation on the image based on the calculated transformation matrix to perfectly connect it with the adjacent images; the image stitching step processes the pixel values ​​of the overlapping area to ensure the continuity and consistency of the image; the image fusion step uses a fusion algorithm to further eliminate the image stitching gaps so that the stitched image presents a natural, smooth and coherent visual effect.

[0024] likeFigure 1 As shown in the figure, the storage system includes storage devices, which are located in the airborne embedded image stitching motherboard part and the ground station part. The storage devices store all the raw data collected by the cameras and the stitched and fused panoramic videos, facilitating subsequent processing and analysis.

[0025] As Figure 1 shown, the transmission system includes an airborne video transmitter and a ground station video transmitter. The airborne video transmitter receives data from the airborne embedded image stitching motherboard and synchronizes it to the ground station video transmitter, realizing the communication and interaction between the airborne embedded image stitching motherboard and the ground station. It has sufficient effective transmission distance, small transmission delay, and sufficient video image transmission capacity.

[0026] As Figure 1 shown, the display operation system includes a ground station for ground - end display operation, which performs image display and settings and interactions for the surround - view panoramic system. It displays real - time live imaging, supports the selection operation of different viewing angles, supports historical playback, and plays historical panoramic videos based on the videos stored in the storage system by selecting the date and time point.

[0027] The specific usage method and function of this embodiment: The designed UAV air - borne surround - view panoramic system includes 6 - channel camera video streams. All 6 cameras are panoramic fisheye cameras. The image of each camera is stitched by 4 ordinary non - wide - angle lenses, reducing image distortion while achieving a large field of view.

[0028] The specific installation method of the cameras on the UAV is as Figure 2 、 Figure 3 shown. One camera is installed vertically upward to obtain a top - view angle; one camera is installed vertically downward to obtain a bottom - view angle; one camera is installed horizontally in each of the four directions of the front, rear, left, and right of the UAV, respectively along the forward direction of the aircraft (arranged at the nose), left direction (arranged at the left wing tip), rear direction (arranged at the tail), and right direction (arranged at the right wing tip). The included angle between the horizontally installed cameras is 90 degrees, forming a circumferential view angle.

[0029] Before the UAV aerial panoramic view system works, the parameters of the camera are first set, including resolution, frame rate, field of view angle, etc., to ensure the clarity and smoothness of the video. The image stitching main board accesses the camera to the service through a protocol, and uses the pulling stream algorithm to obtain multi-channel real-time video stream data. For the acquired data, the hardware decoding technology is used to decode the video data respectively, and the decoded pictures are initially stitched to form a complete picture containing multi-channel camera pictures. Then, through the hardware encoding technology, each complete picture obtained is compressed and encoded in the H.265 format to ensure the time synchronization of the multi-channel video pictures and improve the transmission efficiency. Finally, the compressed video pictures are transmitted to the ground station through the on-board video transmission device for subsequent operations. In particular, the image stabilization technology is used in the image encoding and decoding to ensure the stability of the video pictures during the movement of the UAV, avoid video jitter caused by the shaking of the aircraft, and improve the video quality.

[0030] The ground station receives the stitched pictures transmitted from the sky end through the ground station video transmission, and uses the hardware decoding of the ground station equipment to obtain the complete image. Then, by cropping the complete image, each channel of video picture is obtained. For different points, correction is carried out through the fisheye columnar unfolding and fisheye horizontal unfolding correction algorithms, and further preprocessing such as perspective, cropping, color difference, denoising, and contrast enhancement is carried out. Then, stitching processing (including image registration, image transformation, image stitching, and image fusion steps) is carried out to obtain the mapping file, determine the relative positions of different images in space, use the transformation model to map the video frames from different perspectives to a unified coordinate system, and combine with the three-dimensional external shape model of the aircraft to finally form a complete panoramic picture containing the external shape model of the aircraft, realizing the panoramic synthesis of the image. The ground station presents the UAV panoramic view image and sets the system parameters to realize the panoramic view display of the UAV flight motion scene and support the efficient and accurate remote control of the UAV.

[0031] Based on the designed UAV aerial panoramic view system, real-time imaging test verification is carried out. Figure 4 The original image information collected by 6 on-board cameras is given. Figure 5 The real-time scene information of the front, rear, left, right, up, and down of the UAV presented from the first perspective of the UAV and displayed on the ground station is given. Figure 6 The real-time scene information of the front, rear, left, right, up, and down of the UAV presented from the third perspective of the environment and displayed on the ground station is given. It can be seen from the figure that the UAV aerial panoramic view system of the present invention realizes the real-time live imaging of the aircraft from different perspectives, achieves the matching of the position, speed, attitude, time and scene of the aircraft, and well displays the situation information of the UAV in the environment.

[0032] The above are only embodiments of the present invention, and common general technical solutions and / or characteristics in the solutions are not described in detail herein. It should be noted that for those skilled in the art, without departing from the technical solutions of the present invention, several modifications and improvements can be made, which should also be regarded as the protection scope of the present invention, and these will not affect the implementation effect of the present invention and the practicability of the patent. The protection scope claimed in this application shall be subject to the content of its claims, and the specific implementation manners and the like recorded in the specification can be used to interpret the content of the claims.

Claims

1. An aerial panoramic system for supporting remote control of drones, characterized in that: It includes image acquisition system, image processing system, storage system, transmission system and display operating system.

2. The aerial panoramic system for supporting remote control of unmanned aerial vehicles according to claim 1, wherein: The image acquisition system includes multiple camera video streams, each of which can realize real-time acquisition of video image information; the camera installation angle should avoid field of view obstruction as much as possible, and the camera installation should not change the aerodynamic shape of the aircraft; The camera adopts a mipi camera or a panoramic camera that outputs imaging based on a multi-lens stitching mode, or a combination of the two. The field of view of the camera should satisfy that the sum of the horizontal field of view angles of the camera combination is greater than 360 degrees, and the sum of the vertical field of view angles is greater than 180 degrees, thereby realizing the collection of all-round image information of the environment.

3. The aerial panoramic system for supporting the remote control of drones according to claim 1, wherein: The image processing system includes an airborne embedded image splicing mainboard and a ground station. The airborne embedded image splicing mainboard can collect real-time video streams from cameras, perform edge splicing calculations, initially splice the collected images through hardware decoding, compress and encode the spliced ​​images in H.265 format through hardware encoding, and transmit the encoded video images to the ground station through image transmission equipment.

4. The aerial omnidirectional panoramic system for supporting the remote control of drones according to claim 3, characterized in that: The ground station is provided with equipment hardware, and the equipment hardware functions include preprocessing, image stitching and panoramic synthesis. The preprocessing step includes image decoding, image cropping, picture correction, perspective, chromatic aberration, denoising, and contrast enhancement. The image stitching step generates a mapping file for stitching a panoramic picture, specifically including image registration, image transformation, image stitching, and image fusion. The panoramic synthesis step combines the three-dimensional shape model of the aircraft to form a complete panoramic picture including the aircraft shape model.

5. The aerial panoramic system for supporting remote control of drones according to claim 4, characterized in that: The image registration step includes using a feature point matching algorithm to accurately establish a geometric correspondence between images to ensure accurate alignment of the images; the image transformation step performs an accurate geometric transformation on the image based on the calculated transformation matrix to perfectly connect it with the adjacent image; the image stitching step processes the pixel values ​​of the overlapping area to ensure the continuity and consistency of the image; the image fusion step uses a fusion algorithm to further eliminate the image splicing gaps so that the spliced ​​image presents a natural, smooth and coherent visual effect.

6. The aerial panoramic system for supporting remote control of drones according to claim 1, characterized in that: The storage system includes a storage device, including an airborne embedded image splicing mainboard part and a ground station part. The storage device stores the original data collected by all cameras and the spliced ​​and fused panoramic video to facilitate subsequent processing and analysis.

7. The aerial panoramic system for supporting remote control of drones according to claim 1, characterized in that: The transmission system includes an airborne image transmitter and a ground station image transmitter. The airborne image transmitter receives data from an airborne embedded image splicing mainboard and synchronizes it to the ground station image transmitter, thereby realizing communication interaction between the airborne embedded image splicing mainboard and the ground station. The airborne image transmitter has video image transmission capability.

8. An aerial panoramic system for supporting remote control of an unmanned aerial vehicle according to claim 1, characterized in that: The display operating system includes a ground station for ground-side display operation, which performs image display and setting and interaction with the surround view panoramic system; It displays real-time real-scene imaging, supports the selection of different viewing angles, supports historical review, and plays historical panoramic videos based on the videos stored in the storage system by selecting date and time points.

Images