Airborne panoramic viewing device and method

Abstract

The present invention relates to an airborne panoramic surround-view device and method. The device includes a switch control group, a lidar sensor group, a camera sensor group, and a panoramic view processing and display platform. The switch control group is used to control the opening and closing of the device, the switching of application scenarios, and the switching of display screens. The lidar sensor group is used to obtain the distance and azimuth data between obstacles and the target aircraft. The camera sensor group is used to collect real-time image data of the surrounding and bottom environments of the target aircraft. The panoramic view processing and display platform is used to automatically adjust the azimuth of the sensor group, add auxiliary lines according to the distance and azimuth data obtained by the lidar sensor group in combination with the inherent parameter data (such as the aircraft shell) in the panoramic view, perform obstacle detection in real time and display it in the panoramic view, and finally display the comprehensive panoramic view on the display. It has strong practicability and good application prospects.

Description

Technical Field

[0001] The invention relates to the technical field of airborne graphics and image processing, and in particular to an airborne panoramic viewing device and method. Background Art

[0002] Among the existing monitoring methods for large aircraft such as airplanes, most of them adopt the method of using a single camera to monitor a single-viewing angle. The monitoring personnel need to judge the monitoring area by themselves, which has poor consistency. In addition, the viewing angle of a single camera is limited and it is impossible to monitor a wide-angle environment. When monitoring a larger area, more cameras need to be configured. The monitoring effect benefits show low cost-effectiveness compared with the expenditure.

[0003] Due to the special structure of the aircraft, the pilot cannot intuitively observe the surrounding environment of the aircraft in all directions during the take-off phase, especially when driving in the warehouse and runway. There are visual blind spots, which leads to low personal initiative of the pilot when encountering emergencies, and there is a situation where the pilot cannot detect emergencies and respond to risks in time. This is one of the main causes of collision accidents. Therefore, when the aircraft is driving on the ground, it often requires multiple ground crew members to command and cooperate. In addition, when performing flight missions in extreme weather, sending outdoor ground crew members to conduct on-site command will cause the safety of the ground crew members to be unable to be guaranteed. Therefore, it is necessary to equip the aircraft with a complete set of panoramic viewing devices to reduce manpower input and improve the safety of surrounding environment monitoring.

[0004] Existing panoramic view devices are mostly used in miniaturized driving tools (such as cars and small drones). There are two types of panoramic view devices. One is a distributed panoramic view device. The composition method of this panoramic view device is relatively simple in terms of camera distribution. Most of them use cameras installed in the front, back, left, and right directions of the driving tool to perform image stitching to obtain a bird's-eye view of the target driving tool. Large aircraft are more complex in structure. There are many curves and protruding parts in their appearance, and they are huge in size. Compared with small driving tools such as cars, they require a wider coverage of camera images, and the distribution of cameras should also be more flexible. Image stitching is not just a simple horizontal and vertical fusion, but requires multi-angle image transformation. Another type of panoramic view device is an integrated panoramic view device, which closely combines multiple cameras to form a 360-degree annular or spherical device, which is installed on the top of the target driving tool or hung below the target driving tool to obtain a bird's-eye view based on the surrounding environment of the target driving tool. This method of composing a panoramic view device is mostly used to obtain external environmental conditions other than the target driving tool, but does not focus on the target driving tool itself. It is not suitable for large aircraft. In addition to the problem of insufficient camera coverage, the pixels obtained for distant images are relatively fuzzy, which is not conducive to the driver's observation of distant scenes.

[0005] In addition to the above deficiencies of the existing panoramic surround view devices, there are also the following remarkable features when installing a panoramic surround view device on a large aircraft: Besides the need for the driver to observe the peripheral environment of the target vehicle, there is also a need to observe the bottom environment of the area covered by the target vehicle itself. During the start-up phase of driving on the road surface, it is necessary to pay attention to whether there are obstacles in the bottom environment. Most large aircraft take off with runway assistance. Different from the lateral driving of cars and the vertical take-off of small aircraft such as drones, there is a need to observe the surrounding environment during the initial stage of the large aircraft's lift-off.

[0006] In many currently commercially available surround view devices, multiple camera sensors installed on the vehicle are used to collect image data of the road surface conditions around the target vehicle, and the images of each camera are combined into a panoramic view. In the process of calibrating the surround view camera, in the step of overlapping the images of the overlapping areas from adjacent cameras, the method used is to directly overlap the common elements in the individual images with each other to provide the required view. This method has problems such as visual blind spots and image distortion at the seams. On the other hand, due to the differences in the installation positions of multiple cameras, the cameras are affected differently by the environment, resulting in differences in the brightness and color of the images corresponding to the multiple cameras after surround view restoration, making the viewing experience of the spliced image extremely poor. Especially during the driving process of the target vehicle, the brightness difference of the images corresponding to different cameras will be more obvious.

[0007] Chinese Patent Application: CN111145362A discloses a virtual-real fusion display method for an airborne integrated vision system, which relates to a virtual-real fusion display method and system for an airborne integrated vision system. The proposed virtual-real fusion display method for an airborne integrated vision system, through the sub-pixel accuracy matching of real-time multi-modal video and synthetic vision map images, on the one hand, accurately indicates the precise positions of the runway and obstacles, and on the other hand, through image and graphics fusion, enhances the synthetic vision image in terms of color and texture. Further geometric correction and fusion of the three-dimensional graphics of the obstacles are carried out, and finally, the organic fusion display of multi-modal video and synthetic vision map images is achieved. The present invention can effectively improve the pilot's perception accuracy of the spatial position and shape of the airport runway and obstacles under low visibility conditions, enhance the situational awareness, reduce typical accidents such as controlled flight into terrain and runway incursions during flight approach and landing, and improve the safety of the aircraft.

[0008] Chinese patent application: CN211280826U discloses a vehicle-mounted radar and panoramic image enhancement system, including a radar and panoramic image system, a vehicle-mounted host system, a vehicle-mounted display screen and sound system, a radar probe and a panoramic image enhancement system and a radar probe. The vehicle-mounted host system is respectively linked to the vehicle-mounted display screen and sound system, the vehicle-mounted radar and panoramic image system, and the vehicle-mounted radar and panoramic image enhancement system. The enhancement system will analyze the CAN signal of B1 to determine the running speed value of the car at this time and the distance of the obstacle detected by the radar probe. When these two conditions are met by the MCU software setting at the same time, the vehicle-mounted radar and panoramic image enhancement system sends a wake-up signal to the radar and panoramic image system through the LIN bus, and the system sends the relevant information to the host, which is displayed on the display screen.

[0009] In view of the defects of the prior art, the present invention combines the images of fisheye camera, black and white camera, infrared camera, depth camera and telephoto camera through multi-camera technology, and the color camera is combined with the color camera to calculate the depth of field, realize background blur and refocus; the color camera is combined with the black and white camera to improve the quality of low light / night scene image shooting; the wide-angle lens is combined with the telephoto lens for optical zoom; the color camera is combined with the depth camera for three-dimensional reconstruction, so as to achieve clearer image quality, better adaptability to low light environment, prominent environmental stereoscopic sense and farther viewing distance, and obtain a better visual experience in different scenes. In addition, considering that the panoramic view device on large aircraft uses a relatively single device, only using a camera as an image acquisition device, and the driver's poor perception of distance, the present invention is also combined with a laser radar ranging sensor, and by adding auxiliary lines in the panoramic view view, the driver can control the safe distance and grasp the surrounding environment information of the target aircraft during the ground driving stage and the initial stage of takeoff. There is no report on the present invention's airborne panoramic view device and method. Summary of the invention

[0010] The purpose of the present invention is to provide an airborne panoramic viewing device and method in view of the deficiencies in the prior art.

[0011] To achieve the above object, the technical solution adopted by the present invention is:

[0012] In a first aspect, the present invention provides an airborne panoramic surround view device, which includes a switch control group, a lidar sensor group, a camera sensor group, and a panoramic view processing and display platform. The switch control group includes a power controller, an application scenario controller, and a screen display control module. The switch control group is used to control the device to turn on and off, switch application scenarios, and switch display screens. The lidar sensor group is used to obtain the distance and azimuth data between the obstacle and the target aircraft. The camera sensor group includes a first camera module and a second camera module, which are used to collect real-time image data of the surrounding and bottom environments of the target aircraft. The panoramic view processing and display platform includes an image display selection module, a sensor group azimuth adjustment module, a distance and azimuth data acquisition module, an image data acquisition module, a device synchronization module, an image processing module, a parameter memory, a central arithmetic unit, and a display. It is used to automatically adjust the azimuth of the sensor group. First, the images collected by the camera sensor groups with the same azimuth area are synchronously fused, adjusted, processed, etc. Then, the images processed by the camera sensor groups with different azimuth areas are stitched and fused into a panoramic view. According to the distance and azimuth data obtained by the lidar sensor group and combined with the inherent parameter data of the aircraft in the panoramic view, auxiliary lines are added, obstacle detection is performed in real time and presented in the panoramic view. Finally, the comprehensive panoramic view is displayed on the display.

[0013] Further, the application scenario controller includes a bright light scenario button, a low light scenario button, and a no light scenario button. The bright light scenario button controls the first camera module in the camera sensor group to turn on the fish-eye camera, depth camera, and telephoto camera, and the second camera module to turn on the fish-eye camera, black and white camera, depth camera, and telephoto camera. The low light scenario button controls the first camera module in the camera sensor group to turn on the fish-eye camera, black and white camera, depth camera, and telephoto camera, and the second camera module to turn on the fish-eye camera, black and white camera, depth camera, and telephoto camera. The no light scenario button controls the first camera module in the camera sensor group to turn on the fish-eye camera, infrared camera, and telephoto camera, and the second camera module to turn on the fish-eye camera, infrared camera, and telephoto camera.

[0014] Further, the screen display control module divides the touch screen display screen into a main interface and a secondary interface. The main interface displays the panoramic bird's-eye surround view and the bottom azimuth view of the target aircraft, and the secondary interface displays the panoramic views of the front, rear, left, right, and bottom azimuths of the target aircraft.

[0015] Further, the lidar sensor group is located at the nose, leading edge of the wing, and middle bottom part of the outer edge of the aircraft, and is used to obtain the obstacle distance and azimuth, and to obtain the distance between the target aircraft and the ground.

[0016] Further, the image display selection module is configured to obtain the signal input of the screen display control module in the switch control group, match the device numbers of the lidar sensors and camera sensors in the corresponding area according to the screen display control signal, and bundle the internal and external parameters of the corresponding camera sensors, the conversion matrix parameters of the lidar sensors, and the azimuth arrangement parameters of the sensor group, and transmit these parameters to the distance and azimuth data acquisition module and the image data acquisition module.

[0017] Further, the sensor group azimuth adjustment module is configured to obtain the switch signal of the switch control group and the rotation parameter and telescopic parameter in the parameter memory, control the position and azimuth of the sensor group. When it receives the opening signal of the switch control group, it opens the occlusion cover of the sensor group, extends the sensor group out of the groove to the specified position according to the telescopic parameter, and adjusts its up and down angle according to the rotation parameter. When it receives the closing signal of the switch control group, it retracts the sensor group into the groove and closes the groove with the occlusion cover.

[0018] Further, the distance and azimuth data acquisition module and the image data acquisition module respectively turn on the corresponding lidar sensor device in the lidar sensor group and the corresponding camera sensor device in the camera sensor group through the device numbers input by the image display selection module, and transmit the acquired distance and azimuth data and image data and the corresponding device parameters to the device synchronization module.

[0019] Further, the device synchronization module is configured to synchronize the threads for obtaining the distance and azimuth by the lidar sensor group and the threads for obtaining images by different camera sensors in the camera sensor group and output them to the image processing module.

[0020] Further, the image processing module is configured to perform image processing obtained by the camera sensors, and fuse the distance and azimuth data and image data obtained by the lidar sensors to synthesize a panoramic image and add auxiliary lines and mark the obstacle warning area; the parameter memory is used to store the internal and external parameters of the camera, the device numbers and position numbers of the sensors in each azimuth, the telescopic parameters and rotation parameters of each sensor group, the conversion matrix parameters for transforming the world coordinate system of the data points obtained by the lidar sensors to the image pixel coordinate system, the image overlay projection matrix parameters of multi-type cameras in each azimuth, the height parameter of the target aircraft from the ground before takeoff, the safe flight height parameter of the target aircraft, the obstacle recognition error distance parameter, the height error distance parameter for leaving the ground, and the pixel matrix parameters of the depth auxiliary line, horizontal distance auxiliary line, and aircraft contour auxiliary line in the preset image pixel coordinate system; the central arithmetic unit is used to implement the relevant operations in the image processing module; the display is used to display the processed comprehensive panoramic image view.

[0021] In a second aspect, the present invention provides a method for performing panoramic viewing using the device as described above, comprising the following steps:

[0022] (1) Arrange sensor groups to obtain and store calculation parameters:

[0023] There are two types of LiDAR sensor groups: one is a sensor group consisting of three LiDAR sensors and a camera sensor, and the other is a sensor group consisting of a single LiDAR sensor.

[0024] The camera sensor group is composed of a fisheye camera, an infrared camera, a black and white camera, a depth camera and a wide-angle camera. According to the different ways of matching the camera sensors, it can be divided into two categories: one is the camera sensor group composed of the camera sensors required in the first camera module, and the other is the camera sensor group composed of the camera sensors required in the second camera module. According to the matching and use of the lidar sensor, it can also be divided into two categories: one is the sensor group composed of 3 lidar sensors (same as lidar sensor classification 1), and the other is the sensor group composed of pure camera sensors.

[0025] The following sensor group arrangements are mainly based on three types of sensor groups: sensor group 1 in which a lidar sensor and a camera sensor are used together, sensor group 2 consisting of a single lidar sensor, and sensor group 3 consisting of only camera sensors.

[0026] When arranging the sensor groups, sensor group 1 is arranged at the nose, leading edge of the wing and fixed position at the bottom of the middle of the fuselage of the target aircraft. For aircraft with longer wings, multiple sensor groups are arranged at a distance of at least 2m and no more than 4m to obtain peripheral image data and obstacle distance data. Sensor group 2 is arranged at the bottom of the middle of the fuselage, perpendicular to the ground, to record the height change between the aircraft and the ground. Sensor group 3 is arranged 10cm inside the outermost edge of the aircraft, facing inward, and installed at the bottom of the aircraft, facing vertically downward, to obtain image data of the bottom of the aircraft.

[0027] In addition, it is required that the images acquired by the camera sensors in the sensor group should avoid the engine part. When a single camera sensor group cannot obtain comprehensive image data of a long part, multiple camera sensor groups can be arranged. It is recommended to arrange them at intervals of 2m. The images acquired by all camera sensor groups should cover the entire aircraft's close surrounding area and the aircraft's bottom cover area.

[0028] After arranging the sensor groups, use the conventional calibration method to lay a grid calibration board around the target aircraft, obtain the images of each camera in each camera sensor group, and synchronize the threads for obtaining sensor data by using thread locks and sensor number counting. For the same camera sensor, calculate the internal and external parameters of each camera in the camera group by mapping the coordinate points of the image coordinate system obtained by the camera to the coordinate points of the real-world coordinate system. For the same lidar sensor, calculate the transformation matrix parameters for mapping the coordinate points of the real-world coordinate system obtained by the lidar sensor to the image coordinate system displayed on the monitor by using the measured attitude information and position, as well as the coordinate point data of the calibration board laid in the corresponding azimuth. Map the coordinate points of the images in multiple coordinate systems obtained by all the camera sensors in a group of camera sensors to the same coordinate system, and calculate the image overlay projection matrix of the images obtained by each camera in the same image pixel coordinate system.

[0029] According to the position data of the corresponding key coordinate points directly measured in the real-world coordinate system, map the coordinate points of the real-world coordinate system to the image coordinate system displayed on the monitor, obtain their mapping relationship, and thus calculate the mapping parameters of the vertical depth auxiliary line, horizontal distance auxiliary line, and aircraft contour auxiliary line in the 6 views of panorama, front, rear, left, right, and bottom on the image pixel coordinate system displayed on the monitor.

[0030] (2) Select the application scenario (bright light / weak light / no light), and turn on the corresponding sensor device.

[0031] (3) Fuse multi-type camera images in the same azimuth:

[0032] For the same group of camera sensor groups, use the internal and external parameters of each camera sensor to transform the images obtained by the camera sensors into a bird's-eye view in a single perspective, and then use the overlay projection matrix to overlap multiple bird's-eye views among the fish-eye camera bird's-eye view, infrared camera bird's-eye view, black-and-white camera bird's-eye view, depth camera bird's-eye view, and telephoto camera bird's-eye view in the same group of camera groups into one bird's-eye view according to the scene requirements.

[0033] According to different application scenarios, the fusion forms of multiple camera bird's-eye views mainly include the following 3 types:

[0034] Form 1: Fish-eye + telephoto + depth.

[0035] Form 2: Fish-eye + telephoto + black-and-white + depth.

[0036] Form 3: Fish-eye + telephoto + infrared.

[0037] Method for fusing an image with a telephoto camera image: Multilevel filtering is performed using translation-invariant discrete wavelet transform to form high-frequency sub-images and low-frequency sub-images of the two images. Based on the corresponding high-frequency sub-images of the two images, high-frequency component fusion is performed to form high-frequency component fusion coefficients. Based on the corresponding low-frequency sub-images of the two images, low-frequency component fusion is performed to form low-frequency component fusion coefficients. An inverse translation-invariant discrete wavelet transform is performed based on the high-frequency components corresponding to the high-frequency component fusion coefficients and the low-frequency components corresponding to the low-frequency component fusion coefficients to generate a fused image, achieving the effect of optical zoom.

[0038] Method for fusing an image with a depth camera image: The depth region is obtained from the depth camera image using the pixel values of adjacent pixels, and then the position subscripts of the contour pixels in the region are obtained using the pixel value approximation method. The ratio of the value of the contour line subscript pixel to the maximum subscript is set as the weight value, and the weight value of the pixels in the non-contour region is set to 1 to obtain a new pixel weight matrix. The product of the values of the corresponding points of the pixel weight matrix and the original image pixel matrix is calculated and rounded to obtain the image after fusing the original image with the depth camera image. Compared with the original image, the sharpness of the object edge line in the image is increased, and the three-dimensional effect is more prominent.

[0039] Method for fusing an image with a black-and-white camera image: First, the luminance component and chrominance component of the fish-eye camera image are obtained. The luminance component and the black-and-white image are fused, and the average value is used to replace the numerical value of the luminance component pixel matrix and the corresponding point of the black-and-white image pixel matrix to obtain the luminance fusion component pixel matrix. The target image is obtained by combining the luminance fusion component and the chrominance component. Compared with the original image, the image brightness in the low-light scene can be significantly improved, and the noise can be reduced at the same time.

[0040] Method for fusing an image with an infrared camera image: In a lightless scene, the infrared fill light is turned on to obtain an infrared image.

[0041] (4) Synthesize a panoramic view:

[0042] For the fused bird's-eye view obtained in step (2), arrange it according to the image position information, fuse the overlapping regions, and then perform image brightness balance processing and color balance processing on the image.

[0043] For the overlapping area of the images, the fusion method adopted is to set a weight coefficient for each pixel point. This weight coefficient changes with the pixel value of the point and varies continuously with the coordinate distance of each pixel point to the boundary of the overlapping area. The main method for obtaining the weight coefficient is as follows: Take out the overlapping area in the projection image for grayscale and binaryzation processing. The binaryzation image removes noise through morphological methods, and then determines the outer boundary of the overlapping area by the values of adjacent pixel points. The coordinate values of the polygon contour are obtained by using the approximation method, and then the distance value of the pixel point to the outer boundary of the overlapping area can be calculated by using the coordinate of the target pixel point and the contour coordinate value of the nearest overlapping area. Then calculate the weight value of each pixel point in the overlapping area to the boundary area, set the weight value of the pixels not in the overlapping area, obtain a continuously changing weight matrix, and use the weight matrix to perform a multiplication operation on the pixel values of the overlapping area image to obtain a new pixel point matrix, that is, the pixel matrix on the fused overlapping area image, so as to obtain the image of the overlapping area.

[0044] For the brightness balance of multiple groups of images, a coefficient is calculated for each of the three RGB channels corresponding to the processed camera images in different orientations, and then multiplied by the original channel values respectively. The over-bright channel is dimmed, that is, the coefficient multiplied by it is less than 1, and the over-dark channel is brightened, that is, the coefficient multiplied by it is greater than 1. Finally, a picture with adjusted brightness is formed.

[0045] For the color balance of multiple groups of images, for the panoramic image after image brightness balance, the average value of the total values of the three RGB channels in the pixel matrix is obtained. For the values of the three RGB channels of each pixel point, the original value is multiplied by the average value and then divided by the average value of the channel in the pixel matrix, and the pixel point matrix of the panoramic image after color balance can be obtained.

[0046] (5) Addition of auxiliary lines:

[0047] Use the auxiliary line addition module to add horizontal distance auxiliary lines, vertical depth distance auxiliary lines and aircraft contour auxiliary lines. The horizontal distance auxiliary lines are represented by multiple groups of circles. The multiple groups of circles are in a ring shape, and the distance between the circles from the outside to the inside gradually decreases. The side lines of each group of circles are horizontal distance lines, and the distance between the circles represents the actual horizontal distance, which is calculated by projecting the length in the world coordinate system. The vertical depth auxiliary lines are represented by trapezoids, and there are two in front of and behind the target aircraft. Solid lines are drawn at the intersection of the trapezoids and the circles to achieve a stereoscopic vision effect. Among them, the vertical auxiliary lines are only added to the bird's-eye panoramic image, and the horizontal auxiliary lines are added to the panoramic, front, rear, left and right perspective images respectively. The aircraft contour auxiliary line is the projection of the outer edge contour of the entire aircraft on the image pixel coordinate system, which is used to distinguish the area of the panoramic view around the outside of the aircraft from the area of the panoramic view around the inside of the aircraft.

[0048] (6) Obstacle detection and warning:

[0049] Before the target aircraft leaves the ground, the laser radar sensor located at the front edge of the aircraft obtains the ground information at 50 meters, 100 meters, and 200 meters from the sensor in real time, and calculates the straight-line distance between the laser radar sensor and the obstacle through the laser radar sensor angle. If the distance is less than the obstacle recognition error distance recorded in the parameter memory when there is no obstacle, it is marked that there is an obstacle in its direction, and the display area of ​​its direction on the display flashes. Obstacles at 50 meters, 100 meters, and 200 meters are displayed in red, orange, and yellow regular triangle images on the panoramic view and the corresponding azimuth perspective image respectively. When the target aircraft leaves the ground and does not reach the preset safe flight altitude, that is, when the distance value obtained by the laser radar sensor at the bottom of the aircraft is greater than the preset aircraft ground driving altitude value and less than the preset safe flight altitude value, the 50-meter, 100-meter, and 200-meter obstacle boundaries are no longer determined by the preset aircraft ground driving altitude value, but by the bottom laser radar height ranging value. If a signal of the target aircraft landing gear retraction is detected, the obstacle warning module is turned off.

[0050] (7) Display comprehensive panoramic image:

[0051] The processed panoramic image is transmitted to the monitor at 20 to 30 frames per second for display, which can achieve a smooth panoramic view effect. Users can switch the different orientation images displayed on the main interface and the secondary interface on the monitor by touch. The switch of the camera and the display interface is controlled by the landing gear retraction and extension signal and the main power switch signal. When the aircraft landing gear is retracted, the camera sensor is turned off and the panoramic view system is turned off; when the aircraft landing gear is deployed, the camera sensor is turned on and the panoramic view system is started.

[0052] The advantages of the present invention are:

[0053] 1. The device of the present invention can be configured with a camera group composed of multiple groups of fisheye cameras, black and white cameras, infrared cameras, depth cameras and telephoto cameras, and obtain a higher quality panoramic view under the premise that the camera group can cover the entire monitoring area. Different camera combinations are applied in different scenes to obtain a clearer and more recognizable local or global view of the surrounding environment.

[0054] 2. In addition, the present invention also combines ranging devices such as laser radar. Through the information obtained by multiple sensors, various types of auxiliary lines are added, such as vertical depth line auxiliary lines, horizontal distance line auxiliary lines and aircraft contour auxiliary lines. The distance and azimuth information of obstacles can also be obtained through laser radar ranging, which is directly drawn in the panoramic image and displayed on the display, so that the location of obstacles and the safe distance can be judged more intuitively.

[0055] 3. The image display can be switched to make the observation of the specified range more flexible and detailed. When displaying the specified orientation image to the main interface, the image resolution of the main interface is improved, the picture is more detailed, and the image resolution of the secondary interface is reduced, mainly to maintain the operation of the obstacle detection module, which can effectively reduce energy consumption and the burden of the operator, and reduce the delay of image transmission.

[0056] In addition, the number of sensor groups can be flexibly matched. If you only need to view the panoramic view of a specified direction, you can only assemble the sensor group in the corresponding direction and display the panoramic view only in the main interface, which meets basic needs while reducing costs.

[0057] 4. The present invention can be used to accurately grasp the surrounding environment conditions of the aircraft during the ground driving stage and the initial stage of takeoff, while further obtaining vertical depth perception and horizontal distance perception, understanding the ground environment conditions of the aircraft's periphery and bottom areas, realizing obstacle warning, accurately controlling the distance between the aircraft and obstacles, determining the scope of the safe area, and reducing the probability of accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Attached Figure 1 It is a schematic block diagram of the structure of the airborne panoramic viewing device of the present invention.

[0059] Attached Figure 2A -B is a schematic flow chart of the airborne panoramic view method of the present invention.

[0060] Attached Figure 3 It is a simplified diagram of an example of a panoramic view image of an aircraft with auxiliary lines added in the present invention.

[0061] Attached Figure 4 It is a simplified diagram of an example of the installation position of the laser radar sensor group and the camera sensor group of the present invention.

[0062] Attached Figure 5 It is a schematic diagram of converting the pixel points of an image in the real world to the pixel points of an image displayed on a display according to the present invention.

[0063] Attached Figure 6 It is an exemplary simplified diagram of the front view angle and the top view angle of the camera sensor group of the present invention.

[0064] Attached Figure 7 It is an exemplary simplified diagram of the front view angle of the sensor group composed of a camera sensor group and a laser radar sensor according to the present invention.

[0065] Attached Figure 8 It is an exemplary schematic diagram of the top-down perspective of a sensor group composed of a camera sensor group and a laser radar sensor according to the present invention.

[0066] Attached Figure 9It is an exemplary schematic diagram of the side view angle of the sensor group formed by combining the camera sensor group and the lidar sensor in the present invention.

[0067] Appendix Figure 10 It is a schematic diagram of an example of laying a calibration board around the target aircraft during the calibration of the present invention.

[0068] Appendix Figure 11 It is an exemplary schematic diagram of the layout of the azimuth views displayed on the display of the present invention.

[0069] Appendix Figure 12 It is an exemplary schematic diagram of the display status of the display when obstacles are detected in the front, rear, and left three azimuths of the target aircraft of the present invention.

[0070] 1. First camera module;

[0071] 2. Second camera module;

[0072] 3. Group formed by combining 3 lidar sensors;

[0073] 301. 200-meter limit lidar sensor;

[0074] 302. 100-meter limit lidar sensor;

[0075] 303. 50-meter limit lidar sensor;

[0076] 4. Group formed by combining single lidar sensors;

[0077] 5. Black and white camera sensor;

[0078] 6. Fisheye camera sensor;

[0079] 7. Azimuth fine-tuning device for camera sensors;

[0080] 8. Rotating device;

[0081] 9. Telescopic device;

[0082] 10. Integrated housing for camera sensor group;

[0083] 11. Telephoto camera sensor;

[0084] 12. Depth camera sensor;

[0085] 13. Infrared supplementary light;

[0086] 14. Camera sensor group;

[0087] 15. Occlusion cover adjusting frame;

[0088] 16. Occlusion cover;

[0089] 17. Occlusion cover adjustment frame control device. Specific embodiments

[0090] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. In addition, it should be understood that after reading the content recorded in the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of this application.

[0091] Embodiment 1

[0092] This embodiment provides an apparatus and method for a panoramic view around an aircraft, as Figure 1 shown. The apparatus includes a switch control group, a lidar sensor group, a camera sensor group, and a panoramic view processing and display platform. The switch control group manually controls the device power on or off by the driver. When the device is turned on, the sensor groups are automatically adjusted to the specified positions and orientations. The lidar sensor group and the camera sensor group required are turned on through the application scenario controller and the screen display control module. The obstacle distance and azimuth data and the surrounding image data obtained by the lidar sensor group and the camera sensor group are transmitted to the panoramic view processing and display platform. First, the images collected by the camera sensor groups with the same azimuth area are synchronously fused, adjusted, processed, etc., and then the images processed by the camera sensor groups with different azimuth areas are stitched and fused into a panoramic view. Auxiliary lines are added to the panoramic view according to the distance and azimuth data and the inherent parameter data obtained by the lidar sensor group, and obstacle detection is performed in real time and presented in the panoramic view. Finally, the comprehensive panoramic view is displayed on the display.

[0093] Among them, the switch control group can select the bright light scene button, the low light scene button, and the no light scene button. The bright light scene button is used in an environment with good normal visible light, the low light scene button is used in an environment with weak visible light such as cloudy days, and the no light scene button is used in a no light environment such as night.

[0094] The bright light scene button controls the first camera module in the camera sensor group to turn on the fisheye camera, depth camera, and telephoto camera, and the second camera module to turn on the fisheye camera, black and white camera, depth camera, and telephoto camera. The low light scene button controls the first camera module in the camera sensor group to turn on the fisheye camera, black and white camera, depth camera, and telephoto camera, and the second camera module to turn on the fisheye camera, black and white camera, depth camera, and telephoto camera. The no light scene button controls the first camera module in the camera sensor group to turn on the fisheye camera, infrared camera, and telephoto camera, and the second camera module to turn on the fisheye camera, infrared camera, and telephoto camera.

[0095] The screen display control module divides the touch screen display screen into 1 main interface and 5 sub-interfaces. As Figure 11 shown, the main interface is located in the upper left corner of the screen and is used to display the panoramic bird's-eye view of the target aircraft. The sub-interfaces are located below and to the right of the main interface. Areas 2 to 6 in the figure respectively display the panoramic views of the front azimuth, rear azimuth, left azimuth, right azimuth, and bottom azimuth of the target aircraft. The driver can convert the sub-interface to the main interface for screen display by touching it.

[0096] The distribution positions of the lidar sensor group and the camera sensor group on the aircraft are as Figure 3 shown, where 1 represents the first camera module of the camera sensor group, 2 represents the second camera module of the camera sensor group, 3 represents the group formed by combining 3 lidar sensors in the camera sensor group, 4 represents the group formed by combining a single lidar sensor in the camera sensor group, and the positions of multiple symbol combinations indicate that the sensor groups represented by these symbols are in the same azimuth. The single-group multi-type camera sensor includes but is not limited to ultra-wide-angle cameras of 180° to 230°, fisheye cameras with more than 300,000 pixels and a minimum resolution of 640*480, infrared cameras, black and white cameras, depth cameras, and telephoto cameras. It is required that a group of multi-type cameras be placed closely and in a fixed azimuth. Open the fisheye camera in the camera group and adjust the rotation axis configured on it up and down according to the grid paper of the calibration board (the calibration board pattern and its laying method around the aircraft are as Figure 10 shown), so that the picture obtained by the camera can cover the inner and outer edges of the calibration board, and the covered picture can be effectively spliced with the images obtained by adjacent cameras after conversion. Finally, the pictures obtained by all camera sensors can cover the external and bottom environments of the aircraft at 360°. Record the rotation angle of the rotation axis and store it as the rotation parameter of the camera in this azimuth. This parameter can be directly called in subsequent use to automatically rotate the camera direction angle.

[0097] The front view and side view sketches of a single group of camera sensor groups are as follows Figure 6 shown, where 5 represents a black and white camera sensor; 6 represents a fish-eye camera sensor; 7 represents a camera sensor azimuth fine-tuning device, which is used to finely adjust the camera azimuth before installing the device so that the centers of the pictures obtained by the 4 cameras in the device are at one point; 8 represents a rotating device, which can rotate the sensor group up and down to a specified angle through point control; 9 represents a telescopic device, which is used to push out and retract the sensor group into the groove to prevent the device from affecting the normal flight of the target aircraft, 10 represents the integrated housing of the camera sensor group; 11 represents a long-focus camera sensor; 12 represents a depth camera sensor; 13 represents an infrared fill light.

[0098] The range of the surrounding environment image data obtained by the camera sensor is between 1 meter and 10 meters of the target aircraft. A lidar sensor group formed by combining 3 lidar sensors adjusts the azimuth angles of the lidar sensors so that they respectively point to the ground distances of 50 meters, 100 meters, and 200 meters from the sensor.

[0099] The front view, top view, and side view sketches of the sensor group device combined by the camera sensor group and the lidar sensor group are respectively as follows Figure 7 , Figure 8 , Figure 9 shown, where 14 represents the camera sensor group, 301 represents the 200-meter limit lidar sensor, 302 represents the 100-meter limit lidar sensor, 303 represents the 50-meter limit lidar sensor, 8 represents the rotating device, 15 represents the occlusion cover adjusting frame, 16 represents the occlusion cover, 17 represents the occlusion cover adjusting frame control device, and 9 represents the telescopic device. Among them, the occlusion cover is used to block the groove where it is located when the device retracts into the groove to prevent the airflow from being introduced during the flight of the target aircraft and affecting the flight. The occlusion cover adjusting frame control device can control the occlusion cover adjusting frame to open and close the occlusion cover.

[0100] In addition, Figure 3 Both the triangle and the square in the figure represent the sensor group. A virtual x-axis line is made with the center of the aircraft nose and the center of the tail, and a virtual y-axis line is made with the centers of the left and right wings of the aircraft to form a coordinate system. The direction of the third vertex outside the two points on the line parallel or perpendicular to the coordinate axis connected by the two points of the triangle represents the orientation of the sensor group. The outward direction represents the direction of the sensor towards the external environment outside the bottom environment of the aircraft, that is, the direction of the unshielded part of the aircraft in the bird's-eye view. The opposite inward direction represents the direction of the sensor towards the bottom environment of the aircraft, that is, the direction of the shielded part of the aircraft in the bird's-eye view. The square symbol represents that the sensor group is oriented vertically downward and perpendicular to the ground. The distribution of the cameras in the same straight line is only simply indicated and can be extended to multiple groups of the same orientation distribution arrangement.

[0101] The described panoramic view processing and reality platform can be built on the AGX Xavier computing platform, including an image display selection module, a distance and azimuth data acquisition module, an image data acquisition module, a device synchronization module, an image processing module, a parameter memory, a central arithmetic unit, and a display. Among them:

[0102] The image display selection module is used to obtain the signal input of the screen display control module in the switch control group, match the device numbers of the lidar sensor and the camera sensor in the corresponding area according to the screen display control signal, and bundle the corresponding internal and external parameters of the camera, the conversion matrix parameter for transforming the data points obtained by the lidar sensor from the world coordinate system to the image pixel coordinate system, and the position information in this azimuth, and transmit these parameters to the distance and azimuth data acquisition module and the image data acquisition module. The selectable display screens include the panoramic view, front, rear, left, right, and bottom 6 azimuths of the target aircraft.

[0103] Another possible implementation is to further subdivide the display azimuth. Mainly by marking the device numbers of the azimuth sensors, the corresponding sensors are selected to be turned on to synthesize the panoramic image of the specified azimuth. There is also another possible implementation, which is to combine the sensor devices at multiple different azimuth angles, and when synthesizing the panoramic image, the adjacent azimuth images are merged and the non-adjacent azimuth images are displayed separately for image display.

[0104] The device synchronization module is used to synchronize the threads for obtaining the distance and azimuth of the lidar sensor group and the threads for obtaining images of different camera sensors in the camera sensor group. The main strategy used by this module is to record the total number of sensors in the image display azimuth area specified by the image display selection module, add a counter, and the counter will increment by 1 when each sensor obtains data, record "this task is completed and enter the sleep pool to wait for the next task", and once the counter count reaches the recorded total number of sensors, it will trigger to wake up all threads to enter the next round of task loop.

[0105] The image processing module is used to perform image processing on the images obtained by the camera sensors, and combine the distance and azimuth data and image data obtained by the lidar sensors to synthesize a panoramic image and add auxiliary lines and mark the obstacle warning area. This module includes an image adjustment module, a multi-camera image fusion module in the same area, a panoramic image synthesis module, an auxiliary line addition module, and an obstacle warning module. Among them:

[0106] The image adjustment module, in one implementation, can be used to receive multi-type camera images of the same group input by the device synchronization module, transform these images using the internal and external camera parameters obtained through camera calibration to form a bird's-eye view floor plan, and output the transformed images to the multi-camera image fusion module in the same area. In another implementation, it can be used to receive the images synthesized by the panorama synthesis module, perform brightness balance adjustment and color balance adjustment on the images, and then transmit them to the auxiliary line adding module.

[0107] The method of camera calibration used in the image adjustment module includes the following steps:

[0108] Step 101: Initialization, allocate storage space for camera parameters and the corner points of all images;

[0109] Step 102: Read an image and perform corner detection using the Shi-Tomasi algorithm;

[0110] Step 103: Refine the corner coordinates and draw the detected corners;

[0111] Step 104: For the images on which the corners have been successfully extracted, store the coordinate values of the corners in the world coordinate system and the sub-pixel coordinate values in the image coordinate system. The conversion schematic diagram of points in multiple coordinate systems is as Figure 5 shown;

[0112] Step 105: Calibration, the calibration method used is the conventional calibration method. The schematic diagram of the calibration board laid around the target aircraft before calibration is as Figure 10 shown;

[0113] Step 106: Analyze the error of the calibration result. If the error is too large, re-perform the calibration process.

[0114] The method of brightness balance adjustment in the image adjustment module is to calculate a coefficient for each of the RGB three channels of the image after the fusion of camera groups in all different orientations. There are a total of 3N channels in the images transmitted back by N camera sensor groups. Since there will be color differences in each channel of different images, it is necessary to adjust the values of the over-bright or over-dark channels. We need to calculate 3N coefficients and multiply these 3N coefficients to these 3N channels respectively. The coefficient for adjusting the over-bright channel to make it darker is less than 1, and the coefficient for adjusting the over-dark channel to make it brighter is greater than 1. Finally, a picture with brightness adjustment is formed. The calculation is carried out through the following formula:

[0115]

[0116]

[0117]

[0118]

[0119] Among them, M1 is the pixel matrix of the image of the first camera among two adjacent cameras, M2 is the pixel matrix of the image of the second camera among two adjacent cameras, W represents the adjusted coefficient matrix after corresponding grayscale conversion and binarization, Lr is the average brightness ratio of the images of two adjacent cameras, Lr' is the average brightness ratio of the image after splicing the current two cameras, Lr'' is the average brightness ratio of the image after splicing other cameras except the current two cameras, L is the average brightness ratio of the spliced images in different orientations, L' is the brightness coefficient, P is the pixel point matrix of the original image, and P' is the pixel point matrix after brightness adjustment.

[0120] In the image adjustment module, the method for color balance adjustment is to calculate the average of the total values of the RGB three channels in the pixel matrix of the panoramic image after image brightness balance. Then, the original value of each channel is multiplied by the product of its total average value and the channel average value to obtain the RGB three-channel values of the image after color balance. The corresponding formula is as follows:

[0121]

[0122]

[0123] Among them, M R 、M B 、M G are the three-primary color matrices of the image after brightness adjustment, R, G, and B are the three-primary color values of a certain pixel point on the image, and R W 、G W 、B W are the three-primary color values of this pixel point after color adjustment.

[0124] For the multi-camera image fusion module of the same area, the same-orientation and same-group multi-type camera images transformed by the image adjustment module are overlapped on the image pixel coordinate system by using the multiply blend projection matrix in the parameter memory to be fused into a single single-orientation image.

[0125] The method for fusing the same-orientation and same-group multi-type camera images is as follows:

[0126] Fusion of the image and the telephoto camera image: Use translation-invariant discrete wavelet transform for multi-layer filtering to form the high-frequency sub-image and low-frequency sub-image of the two images. According to the corresponding high-frequency sub-images of the two images, perform high-frequency component fusion to form the high-frequency component fusion coefficient. According to the corresponding low-frequency sub-images of the two images, perform low-frequency component fusion to form the low-frequency component fusion coefficient. Perform inverse translation-invariant discrete wavelet transform according to the high-frequency component corresponding to the high-frequency component fusion coefficient and the low-frequency component corresponding to the low-frequency component fusion coefficient to generate the fused image, achieving the effect of optical zoom.

[0127] Image and depth camera image fusion: For the depth camera image, the depth area is obtained by using the pixel values of adjacent pixels, and then the contour subscript is obtained by using the approximation method. The ratio of the pixel value of the pixel point with the contour subscript to the maximum subscript value is set as the weight value, and the weight value of the pixel points in the non-contour area is set to 1, obtaining a new pixel point weight matrix. Multiply the values of the corresponding points of the pixel point weight matrix and the original image pixel matrix and round to obtain the image after the fusion of the original image and the depth camera image. Compared with the original image, the sharpness of the object edge line in the image is increased, and the three-dimensional sense is more prominent.

[0128] Image and black-and-white camera image fusion: First, obtain the luminance component and chrominance component of the fisheye camera image, fuse the luminance component and the black-and-white image, and use the average substitution of the pixel matrix values of the luminance component and the corresponding points of the black-and-white image pixel matrix to obtain the luminance fusion component pixel matrix. The target image is obtained by combining the luminance fusion component and the chrominance component. Compared with the original image, the image brightness in the low-light scene can be significantly improved, and the noise can be reduced.

[0129] The fusion with the infrared camera image adopts the form of virtual fusion. The actual application is to turn on the infrared fill light in the lightless scene to obtain the infrared image.

[0130] The panoramic image synthesis module is used to synthesize the images after the fusion of multiple cameras in the same azimuth into a panoramic image according to the position information of the sensors in this azimuth. The main steps of the method for fusing the overlapping areas of the images in this module are as follows:

[0131] Step 201: Take out the overlapping area in the images after the fusion of two adjacent camera sensor groups;

[0132] Step 202: Gray-scale and binarize the overlapping area image obtained in Step 201;

[0133] Step 203: Use morphological operations to remove noise from the image processed in Step 202. Specifically, slide the structuring element on the original image, and set the gray value of the image pixel point at the anchor position of the structuring element to the minimum value of the corresponding image area pixels in the area where the value of the structuring element is 1. It is expressed by the formula as follows:

[0134]

[0135] Among them, element is the structuring element, (x, y) is the position of the anchor point O, x' and y' are the position offsets of the pixels with the value of 1 in the structuring element relative to the anchor point O, src represents the original image, and dst represents the result image.

[0136] Step 204: Use the approximation method to detect the boundary outside the overlapping area of the image processed in Step 203;

[0137] Step 205: Calculate the weight of each pixel point in the overlapping area to the boundary area. Use the coordinates of the target pixel point and the contour coordinate values after the nearest overlapping area to solve the distances from the pixel point to the two boundaries of the overlapping area, and obtain the weight values of each pixel point in the overlapping area using the distance ratio;

[0138] Step 206: Set the weights of the pixels in the non-overlapping area to obtain a continuously varying weight matrix. Set the weights of all pixel points in the first camera image of two adjacent cameras that are not in the overlapping area to 1, and set the weights of all pixel points in the second camera image of two adjacent cameras that are not in the overlapping area to 0. Combine the weight value matrix of each pixel point in the overlapping area to obtain a continuously varying matrix W with a value range between 0 and 1;

[0139] Step 207: Use the weight matrix to fuse the images in the overlapping area. The pixel matrix of the fused image is:

[0140] I f = W * I1 + (1 - W) * I2

[0141] where W is the weight matrix corresponding to the pixel points, I f is the pixel matrix of the fused image, I1 is the pixel matrix of the first camera image of two adjacent cameras, and I2 is the pixel matrix of the first camera image of two adjacent cameras;

[0142] The auxiliary line adding module is used to directly add depth auxiliary lines, horizontal distance auxiliary lines, and aircraft contour auxiliary lines to the panoramic image adjusted by the image adjustment module according to the preset distance in the parameter memory. The schematic diagram is as Figure 4 shown. The horizontal distance auxiliary lines are represented by 3 groups of rings. The width of the rings gradually decreases from the outside to the inside. When drawing the ring diagram, take the center of the panoramic image as the center of the circle, map the upper left corner coordinates of the world coordinate calibration board to the image pixel coordinate system, and use the distance between the center coordinate point and the mapped calibration board coordinate point as the radius of the innermost circle, and draw a circle with a black-filled inner transparent border. The 2nd, 3rd, and 4th layer circles are expanded outward by 3m, 6m, and 10m respectively based on the radius of the previous layer circle, and the length mapped to the image pixel coordinate system is used as the radius of the 2nd, 3rd, and 4th layer circles to form 3 rings. The depth auxiliary lines are represented by 2 isosceles trapezoids in the front and back. The front trapezoid and the rear trapezoid are symmetric about the center of the circle. The lower bottom side length of the front trapezoid coincides with the upper side of the calibration board. The smallest base angle of the isosceles trapezoid is set to 45°. It is required that the four points of the trapezoid fall on the edges of the 1st and 4th layer circles respectively to draw the trapezoid. A solid line is drawn at the intersection of the trapezoid and the ring to achieve a stereoscopic vision effect. The height of the trapezoid and the length of the segment where the trapezoid intersects the ring represent the depth distance. The aircraft contour auxiliary line is the projection of the entire outer edge contour of the aircraft on the image pixel coordinate system.

[0143] Because there are many obstacles in the bottom environment, such as the power device, and the space it occupies cannot be ignored, it is necessary to add auxiliary lines of the aircraft contour to distinguish the panoramic view area outside the aircraft from the panoramic view area of the bottom of the aircraft (i.e., the bottom ground condition covered by the aircraft in the bird's-eye view).

[0144] The obstacle warning module is used to detect the obstacle information in front of the target aircraft and draw an obstacle warning identification map on the panoramic view. The main methods used in this module are as follows: Before the target aircraft takes off, the lidar sensor at the front edge of the aircraft continuously obtains the ground information at 50 meters, 100 meters, and 200 meters from the distance sensor. The straight-line distance between the lidar sensor and the obstacle is calculated through the lidar sensor angle. If the distance is less than the obstacle recognition error distance in the parameter memory when there is no obstacle recorded, it is marked that there is an obstacle in its azimuth, and the display area corresponding to the azimuth on the display is flashed. The obstacles at 50 meters, 100 meters, and 200 meters are marked with red, orange, and yellow equilateral triangle images on the panoramic view and the corresponding azimuth perspective images respectively. As Figure 12 shown, it means that obstacle information has been detected respectively between 50 meters and 100 meters in front of the target aircraft, within 50 meters behind, and within 100 meters to 200 meters on the left. When the target aircraft takes off and has not reached the preset safe flight altitude, that is, when the distance value obtained by the lidar sensor at the bottom of the aircraft is greater than the preset ground travel altitude value of the aircraft and less than the preset safe flight altitude value, it is determined that the obstacle boundaries at 50 meters, 100 meters, and 200 meters no longer use the preset ground travel altitude value of the aircraft, but use the bottom lidar height measurement value for calculation. If a signal indicating that the landing gear of the target aircraft is retracted is detected, the obstacle warning module is turned off.

[0145] The parameter memory is used to store the pre-calibrated internal and external parameters of the camera, the device numbers and position numbers of the sensors in each azimuth, the telescopic parameters and rotation parameters of each sensor group, the conversion matrix parameters for transforming the world coordinate system of the data points obtained by the lidar sensor to the image pixel coordinate system, the image overlay projection matrix parameters of the multi-type cameras in each azimuth, the height parameter of the target aircraft from the ground before takeoff, the safe flight altitude parameter of the target aircraft, the obstacle recognition error distance parameter, the takeoff height error distance parameter, and the pixel matrix parameters of the depth auxiliary line, horizontal distance auxiliary line, and aircraft contour auxiliary line in the preset image pixel coordinate system. The parameters stored in the parameter register are all adjustable parameters set manually and need to be set before the actual flight of the target aircraft.

[0146] The central computing unit is used to implement related operations in the image processing module, and can use high-computing power high-speed computing platforms such as AGX Xavier to process these operations.

[0147] The display is used to display the processed comprehensive panoramic image view. It can be a flat display or a head-mounted helmet display through image projection. The processed panoramic image is transmitted to the display at 20 to 30 frames per second for display, so that a smooth panoramic view effect can be achieved. The switch of the camera and the display is controlled by the retraction and extension signal of the landing gear, the flight altitude of the aircraft and the main power switch signal. When the aircraft landing gear is retracted and the flight altitude of the target aircraft detected by the bottom laser radar sensor reaches a safe flight altitude, the camera is turned off and the panoramic view image is no longer displayed; when the aircraft landing gear is deployed, the camera is turned on to restore the panoramic view image.

[0148] The present invention also proposes an embodiment of a process for implementing a panoramic view method for an aircraft, as shown in FIG2 , and the specific process is as follows:

[0149] Step 301: Turn on the system power supply. If the operation parameter acquisition operation has not been performed, first calculate the coordinate point conversion in multiple coordinate systems through the calibration board to obtain the corresponding parameters. If these parameters have been obtained, directly proceed to step 302;

[0150] Step 302: The driver determines the current environmental conditions and selects the corresponding device operation scene (bright light / weak light / no light) using the application scene controller;

[0151] Step 303: Obtain the aircraft landing gear signal and determine whether to turn on or off the device;

[0152] Step 304: Select the orientation screen to be displayed through the screen display control module, open different camera sensor devices in combination with the application scenario selected by the application scenario controller, synchronously acquire image data through the thread, and determine the obstacle detection calculation method on the 50-meter, 100-meter, and 200-meter boundaries through the laser radar sensor at the bottom of the fuselage to obtain distance and orientation data;

[0153] Step 305: Process the camera sensor images through the data processing module, and fuse the images acquired by the same group of multi-camera sensors through the same area multi-camera image fusion module;

[0154] Step 306: stitching the processed images in the specified direction of the images obtained in step 305 into a panoramic view through a panoramic image synthesis module;

[0155] Step 307: adjusting the brightness balance of the panoramic image synthesized in step 306 and adding auxiliary lines thereto through the image adjustment module and the auxiliary line adding module;

[0156] Step 308: Based on the obstacle distance and azimuth data obtained by the obstacle warning module, flash the display area corresponding to the azimuth on the display, and visually display it on the panoramic image obtained in Step 307 and the corresponding azimuth perspective image (the obstacles at 50 meters, 100 meters, and 200 meters are marked with red, orange, and yellow equilateral triangle images respectively);

[0157] Step 309: Display the comprehensive panoramic image.

[0158] In Step 301, when the camera is not calibrated, use the switch control module to turn on the device, and obtain the surrounding environment images of the target aircraft from multiple groups of multi-type camera sensors in the first and second camera modules. The images need to include the inner and outer edges of the calibration board, or at least one layer of the black and white grid pattern of the calibration board, and the images obtained by all camera sensors can cover the external and bottom environments of the aircraft by 360°. Set the length and width of the bird's-eye view, the length and width from the inner circle to the outer circle of the calibration board, the distance between the calibration board and the target aircraft, and the values of the 4 world coordinate points measured according to the length and width of the calibration board grid with the upper left corner of the bird's-eye view as the origin. Using the imaging principle of the pinhole camera model, obtain the internal parameters and distortion coefficients of each camera sensor. Through rigid body transformation operations such as adjusting the horizontal and vertical scaling ratios and the center coordinate points of the image, convert the distorted original pattern of the camera sensor from the world coordinate system to the camera coordinate system image. The transformed image is sequentially selected in the image physical coordinate system according to the preset 4 key coordinate points, so as to obtain the mapping matrix of the camera sensor. Combine the internal parameter matrix and distortion coefficient of the camera sensor for perspective projection to convert the image from the camera coordinate system to the image coordinate system. At this time, the bird's-eye view of a single camera sensor is obtained. Set the parameters for the severely distorted bird's-eye view, and store the internal and external parameters of the normal bird's-eye view in the camera. In addition, according to the preset coordinate points on the image, solve the image overlay projection matrix for converting multiple different types of camera sensors in the same group of camera clusters to the same image pixel coordinate system. For the conversion matrix parameters, depth auxiliary lines, horizontal distance auxiliary lines, and aircraft contour auxiliary lines of the data points obtained by the lidar sensor from the world coordinate system to the image pixel coordinate system in the pixel matrix of the preset image pixel coordinate system, calculate them based on the actual measured distance and the azimuth distance of multiple coordinate points relative to the calibration board. In addition, it is also necessary to set and measure the device numbers and positions of each azimuth sensor, the height of the target aircraft from the ground before takeoff, the safe flight height of the target aircraft, the obstacle recognition error distance, and the error distance of the height from the ground when leaving the ground, etc., and store them in the parameter memory. If the camera sensor has been calibrated, then Step 301 can directly use the real-time data obtained by the camera sensor and the lidar sensor to perform Step 302.

[0159] In step 302, the scene controller buttons and the touch display display area sub-interface can be switched at any time according to the actual driving situation.

[0160] In step 305, the fisheye camera image needs to be transformed for image distortion correction first, and then other camera images are transformed accordingly to obtain bird's-eye views of each camera image, and the bird's-eye views of multiple different types of cameras in the same camera group are overlapped to obtain a new bird's-eye view.

[0161] In step 306, when the images are stitched into a panoramic view, the bird's-eye view is sorted by sensor position and rotated according to a preset angle. The images of non-overlapping areas are mapped to the image pixel matrix of the corresponding position of the panoramic view as they are, and the images of overlapping areas are fused according to the weight matrix of pixel smoothing. The synthesized panoramic image is then subjected to brightness balance adjustment and color balance mediation.

[0162] The auxiliary lines in step 307 are added in the auxiliary line adding module in the image processing module. The module obtains the above-mentioned synthesized panoramic view of the aircraft, and adds depth auxiliary lines, horizontal distance auxiliary lines and aircraft contour auxiliary lines on the basis of the view. The horizontal distance auxiliary lines are drawn using a circular graph, and the depth auxiliary lines are drawn using two positive and negative trapezoidal graphs. The trapezoidal graph and the circular graph are used in combination to give it a three-dimensional visual experience. The aircraft contour auxiliary lines are scaled proportionally through the bird's-eye view of the original target aircraft in the world coordinate system, and only its outer edges are drawn.

[0163] In step 308, the obstacle warning is performed by detecting the obstacle information through the laser radar sensor group, and then using the laser radar sensor to obtain the conversion matrix of the data point world coordinate system to the image pixel coordinate system, and the numerical data is visualized. The obstacles are represented by triangular symbols and drawn in the panoramic view and on the corresponding azimuth perspective image, and the display area of ​​the corresponding azimuth on the display flashes for prompts.

[0164] The step 309 is to transmit the panoramic surround view image synthesized by the image processing module to the display for display, and form a real-time panoramic surround view image by using high frame rate image display.

[0165] The above is only a preferred embodiment of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and supplements without departing from the principle of the present invention. These improvements and supplements should also be regarded as the scope of protection of the present invention.

Claims

1. An airborne panoramic viewing device, characterized in that: The device includes a switch control group, a lidar sensor group, a camera sensor group, a panoramic view processing and display platform. The switch control group includes a power controller, an application scenario controller, and a screen display control module. The switch control group is used to control the device to turn on and off, switch application scenarios, and switch display screens; the lidar sensor group is used to obtain the distance and azimuth data between obstacles and the target aircraft; the camera sensor group includes a first camera module and a second camera module, and the camera sensor group is used to collect real-time image data of the environment around and under the target aircraft; the panoramic view processing and display platform includes an image display selection module, a sensor group azimuth adjustment module, a distance and azimuth data acquisition module, an image data acquisition module, a device synchronization module, an image processing module, a parameter memory, a central arithmetic unit, and a display. The panoramic view processing and display platform is used to automatically adjust the azimuth of the sensor group, first synchronously fuse, adjust, and process the images collected by the camera sensor groups with the same azimuth area, then splice and fuse the images processed by the camera sensor groups with different azimuth areas into a panoramic view, add auxiliary lines according to the distance and azimuth data obtained by the lidar sensor group combined with the inherent parameter data of the aircraft in the panoramic view, detect obstacles in real time and display them in the panoramic view, and finally display the comprehensive panoramic view on the display; The application scenario controller includes a bright light scenario button, a low light scenario button, and a no light scenario button. The bright light scenario button controls the first camera module in the camera sensor group to turn on the fisheye camera, depth camera, and telephoto camera, and the second camera module to turn on the fisheye camera, black and white camera, depth camera, and telephoto camera; The low light scenario button controls the first camera module in the camera sensor group to turn on the fisheye camera, black and white camera, depth camera, and telephoto camera, and the second camera module to turn on the fisheye camera, black and white camera, depth camera, and telephoto camera; The no light scenario button controls the first camera module in the camera sensor group to turn on the fisheye camera, infrared camera, and telephoto camera, and the second camera module to turn on the fisheye camera, infrared camera, and telephoto camera; The image display selection module is used to obtain the signal input of the screen display control module in the switch control group, match the device numbers of the lidar sensor and camera sensors in the corresponding area according to the screen display control signal, and bundle the internal and external parameters of the corresponding camera sensors, the transformation matrix parameters of the lidar sensor, and the azimuth layout parameters of the sensor group, and transmit these parameters to the distance and azimuth data acquisition module and the image data acquisition module; The sensor group azimuth adjustment module is used to obtain the switch signals of the switch control group and the rotation parameters and telescopic parameters in the parameter memory, and control the position and azimuth of the sensor group. When it receives the opening signal of the switch control group for acquisition, it opens the occlusion cover of the sensor group, extends the sensor group out of the groove to a specified position according to the telescopic parameter, and adjusts its up and down angle according to the rotation parameter. When it receives the closing signal of the switch control group for acquisition, it retracts the sensor group into the groove and closes the groove with the occlusion cover; The distance and azimuth data acquisition module and the image data acquisition module respectively turn on the corresponding lidar sensor devices in the lidar sensor group and the corresponding camera sensor devices in the camera sensor group according to the device numbers input by the image display selection module, and transmit the acquired distance and azimuth data and the corresponding device parameters to the device synchronization module; The device synchronization module is used to synchronize the threads for acquiring the distance and azimuth of the lidar sensor group and the threads for different camera sensors in the camera sensor group to acquire images, and transmit the data to the image processing module.

2. The device according to claim 1, wherein The screen display control module divides the touch screen display screen into a main interface and a secondary interface. The main interface displays the panoramic bird's-eye view and the bottom azimuth view of the target aircraft, and the secondary interface displays the panoramic views of the front, rear, left, right, and bottom azimuths of the target aircraft.

3. The device according to claim 2, wherein The lidar sensor group is located at the nose of the outer edge of the aircraft, the leading edge of the wing, and the middle bottom part of the fuselage. The lidar sensor group is used to obtain the distance and azimuth of obstacles and the distance between the target aircraft and the ground.

4. The device according to claim 1, characterized in that, The image processing module is used for processing the images acquired by the camera sensors, and for fusing the distance and azimuth data and the image data acquired by the lidar sensors to synthesize a panoramic image and add auxiliary lines and mark the obstacle warning area; the parameter memory is used to store the internal and external parameters of the camera, the device numbers and position numbers of the sensors in each azimuth, the telescopic parameters and rotation parameters of each sensor group, the conversion matrix parameters for transforming the data point world coordinate system acquired by the lidar sensor to the image pixel coordinate system, the image overlay projection matrix parameters of the multi-type cameras in each azimuth, the height parameter of the target aircraft from the ground before takeoff, the safe flight height parameter of the target aircraft, the obstacle recognition error distance parameter, the error distance parameter for leaving the ground height, and the pixel matrix parameters of the depth auxiliary line, the horizontal distance auxiliary line, and the aircraft contour auxiliary line in the preset image pixel coordinate system; the central arithmetic unit is used to implement the relevant operations in the image processing module; The display is used to display the processed comprehensive panoramic image view.

5. A method for panoramic surround view using the device according to any one of claims 1-4, characterized in that, It includes the following steps: (1)Arrange the sensor group, obtain and store the operation parameters; (2)Select the application scenario (bright light / weak light / no light), and turn on the corresponding sensor device; (3)Fuse the images of the cameras in the same azimuth; (4)Synthesize the panoramic view; (5)Add auxiliary lines; (6)Detect and warn of obstacles; (7)Display the comprehensive panoramic image.

Images