Aircraft taxiing anti-collision assistance method based on binocular vision

Abstract

In an embodiment of the present disclosure, a method for aircraft taxiing anti-collision assistance based on binocular vision is provided. A binocular camera is installed through a window inside the aircraft cabin to obtain an external disparity map, which is converted into a depth map, and the missing information thereof is post-processed. Finally, it is visually displayed to assist the pilot in observing the distance information of the external scene of the aircraft during the ground taxiing stage.

Description

Technical Field

[0001] The present invention relates to the technical field of avionics equipment, and particularly to an aircraft taxiing anti-collision assistance method based on binocular vision. Background Art

[0002] In the prior art, millimeter-wave radars are installed at four preset positions on the aircraft. During the ground taxiing process of the aircraft, collisions between the wings and obstacles often occur. Since the cockpit or the towing vehicle can only observe the front area, there are visual blind spots on both sides of the fuselage and the wings, and it is difficult for the human eye to accurately distinguish the distance between the obstacle at the front end of the wing and the wing edge. Moreover, when the wing is designed, no position is reserved for installing a ranging device. The ranging device needs to be located inside the cabin and obtain external scene information through the porthole. Under this condition, the ultrasonic radar cannot be applied inside the porthole, while the lidar has a high cost and a short equipment life. In particular, when the distance between two aircraft is relatively close, there is interference between multiple device radars, and when facing a far-away small object, the small object may be missed due to the laser resolution problem. When the dot matrix distance is too far, the laser resolution is likely to miss small objects, such as iron wires and railings. For the monocular ranging scheme, although the natural images collected by the camera have a low cost and are portable to install, it is difficult to accurately measure the distance between the external object and the aircraft.

[0003] In the prior art, when using binocular cameras, there is a lack of structured light compensation, and the generated disparity map is only sensitive to the object edges, which is insufficient to accurately describe the external environment information when facing an airport runway environment where there may be large obstacles lacking texture structures. Summary of the Invention

[0004] In view of this, the embodiments of the present disclosure provide an aircraft taxiing anti-collision assistance method based on binocular vision, which can identify and warn of the obstacles in front of the wing without changing the external structure of the aircraft, and assist the aircraft in ground taxiing.

[0005] An aircraft taxiing anti-collision assistance method based on binocular vision, wherein binocular cameras are respectively installed in the nose position and the two passenger cabins on the front side of the adjacent wing, and the binocular cameras are arranged horizontally. The method includes:

[0006] Obtaining the image data collected by each camera and the current external point cloud data, and determining the depth information of each binocular camera;

[0007] Performing filtering and filling on the depth information of each binocular camera horizontally to obtain a filled and filtered depth information matrix;

[0008] Visualizing the depth information matrix, and removing the point cloud information below the ground and higher than twice the aircraft height, where the aircraft height corresponds to the aircraft model;

[0009] Draw a top-down obstacle display diagram, perform maximum filtering on the 3D point cloud information along the direction perpendicular to the ground, take the camera as the center, consider the area behind the obstacle occlusion area as the obstacle area and assign values to obtain the top-down obstacle information, where the area in collision with the aircraft is marked with the first color and the safe area is marked with the second color.

[0010] Beneficial effects:

[0011] It can be used to assist the driver during the aircraft taxiing phase. The driver can briefly understand the external object information of the aircraft through the top-down obstacle display diagram and can more intuitively see the external object information through the real-scene obstacle warning diagram, making up for the problem that there are blind spots in the driver's position for observation and it is difficult for the naked eye to accurately judge the distance.

[0012] Installed inside the aircraft, it does not affect the external structure of the aircraft, does not affect the safety of the aircraft, and does not require additional testing and inspection. Using binocular cameras, directly obtain external natural images through the porthole, with lower hardware costs, not affected by multiple layers of glass, and no mutual interference between multiple device radars. Description of the drawings

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0014] Figure 1 is the top-down obstacle display diagram;

[0015] Figure 2 is the gradient schematic diagram with multiple colors for the first color. Detailed implementation manners

[0016] The following will describe the embodiments of the present disclosure in detail with reference to the drawings.

[0017] The following illustrates the implementation manners of the present disclosure through specific specific examples. Those skilled in the art can easily understand other advantages and effects of the present disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present disclosure without creative efforts belong to the scope of protection of the present disclosure.

[0018] It should be noted that the following description relates to various aspects of embodiments within the scope of the appended claims. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art should understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement a device and / or practice a method. Additionally, this device can be implemented and this method can be practiced using other structures and / or functionality in addition to one or more of the aspects described herein.

[0019] For the aircraft taxiing anti-collision assistance method based on binocular vision of the present invention, binocular cameras are respectively installed in the nose position and the two passenger cabins on the front side of the adjacent wing. The binocular cameras are arranged horizontally. The method includes:

[0020] S101: Obtain the image data collected by each camera and the external point cloud data at the current moment, and determine the depth information of each binocular camera. Specifically,

[0021] Arrangement mode of the binocular cameras: Deploy the binocular ranging cameras at the portholes in front of the aircraft wing, one on each side, with the dual cameras of the cameras arranged horizontally and the lenses perpendicular to the glass and facing outwards. Deploy one in the aircraft cockpit in the direction of the aircraft's travel for obtaining the data directly ahead. Obtain the image data collected by all cameras and the external point cloud data at this moment. The display terminal receives the data packet, which includes: Output of the binocular cameras: original disparity information, RGB information collected by the cameras, and camera parameter information;

[0022] Determining the depth information of each binocular camera includes,

[0023] Obtain the original disparity information output by each frame of the binocular camera, the RGB information collected by the camera, and the camera parameter information, that is, each frame of the camera gives: original disparity information, RGB information collected by the camera, and camera parameter information;

[0024] Since there is more noise at the image edges, it is necessary to discard the information of three pixels at the boundary, process the original disparity information using the camera parameters, and discard the information of three pixels at the boundary; The processing method is "convert the disparity information into depth information, discard the information of three pixels at the boundary, and discard a circle with a width of 3 pixels on the outside of the collected data, that is, discard the peripheral boundary information with a width of three pixels;

[0025] Convert the disparity information into depth information, satisfying:

[0026] , where f is the focal length of the camera, Baseline is the camera baseline value, Dsp represents the disparity value at the image coordinate point (m, n); Z represents the vertical distance from the pixel point at coordinate (m, n) in the image data to the plane where the camera lens is located;

[0027] S102: Filter and fill the depth information of each binocular camera horizontally to obtain the depth information matrix after filling and filtering. Specifically, since it is difficult to effectively identify the depth information of large textureless areas in the disparity map, the pixel point values for which the distance cannot be accurately judged are 0, and it is only sensitive to the boundary information. Therefore, the depth information coordinate points are concentrated on the boundaries of external objects, and the currently obtained depth information needs to be further processed. The specific processing method is as follows:

[0028] First, filter and fill the depth information horizontally, that is, filter and fill the depth information of each binocular camera horizontally, including:

[0029] Obtain the image data of each frame of the binocular camera, determine multiple depth values in the image data, and all the depth information forms a depth information matrix;

[0030] Traverse the depth information matrix horizontally to calculate the horizontal change rate value of the null points, with horizontal priority. The horizontal change rate value satisfies:

[0031] , where is the depth value of the first non-zero point to the left of the point (m,n), is its ordinate value; is the depth value of the first non-zero point to the right, is the ordinate value;

[0032] Set a threshold to judge whether the horizontal change rate value of the current point coordinate is greater than the threshold. If so, do not fill; if not, fill the depth value of this point. For example, the threshold is , when the change rate do not fill, when the change rate fill the depth value of this point, and the filled value satisfies:

[0033] ;

[0034] After horizontal filling, fill in the diagonal downward direction, diagonal upward direction, and then vertically;

[0035] Fill in the remaining null points and iterate multiple times to obtain the depth information after filling and filtering , forming a filled digital matrix;

[0036] S103: Visualize the depth information matrix and remove the point cloud information below the ground and above a preset multiple of the aircraft altitude (the preset multiple refers to the "point cloud information" that is 2 times higher than the aircraft altitude. The aircraft altitude corresponds to the aircraft model. Specifically, visualizing the depth information matrix includes:

[0037] According to the depth information, convert it into a color space representation (for example, convert it to the RGB color space), red, green, and blue, to form an initial visible image, and use colors to represent the distance values of the corresponding pixel points from the aircraft. Generate a color representation of the depth map based on the distance values;

[0038] Overlay it on the original image to display a real-scene obstacle warning map. Among them, the objects that may collide are filled or marked with a first color. The first color has the gradient characteristics of multiple colors. As the distance value gets farther, the color gradually changes to green.

[0039] Determine the effective visible range of the camera. For example, it changes from 1.5 meters to 30 meters and can cover the area in front of the wing. Overlay the initial visible image with the photo taken by the camera, that is, the original image, to display a real-scene obstacle warning map. Among them, the objects that may collide are filled or marked with a first color. The first color has the gradient characteristics of multiple colors. As Figure 2 shown, as the distance value gets farther, the color gradually changes from dark red to green;

[0040] S104: Draw a top-down obstacle display map. Filter the three-dimensional point cloud information along the direction perpendicular to the ground with the maximum value. Take the camera as the center, consider the area behind the obstacle occlusion area as the obstacle area and assign values to obtain the top-down obstacle information. Among them, the area that collides with the aircraft is marked with a first color, and the safe area is marked with a second color. Specifically,

[0041] Calculate the coordinates of the filled digital matrix in the real space, including:

[0042] Obtain the filled depth information, and calculate the x-axis and y-axis coordinate information of this point in the camera coordinate system according to the formula:

[0043]

[0044]

[0045] Among them, f is the camera focal length; img width is the width of the image collected by the camera, img Heighthis the height of the image collected by the camera. Z(m, n) is the depth or distance of the coordinate (m, n) in the filled matrix, representing the actual distance value of this point from the plane. X, Y, and Z establish a three-dimensional coordinate system with the lens. The direction the lens points to is the Z-axis, representing the vertical distance of the pixel point of the image coordinate (m, n) from the plane where the camera lens is located. The lens points above the plane for the Y-axis, which is perpendicular to the X-axis and Z-axis. The X-axis points to the right of the imaging plane, obtaining the point cloud data in the camera coordinate system.

[0046] Convert the camera coordinate system to the plane coordinate system. Specifically:

[0047] Obtain the point cloud map in the camera coordinate system Backward, according to the camera installation position, convert it to the plane coordinate system. The camera installation position is , the angles between the camera pointing direction and the x-axis, y-axis, and z-axis in the camera coordinate system are respectively , and the obtained rotation transformation matrix is:

[0048]

[0049] After calculation, obtain the point cloud matrix in the plane coordinate system :

[0050]

[0051] Remove the point cloud information below the ground and above twice the height of the plane (invalid data, collisions that cannot occur to the plane in the real environment, just the data collected), which is the point cloud information of the objects that may collide with the fuselage and wings during the plane's taxiing.

[0052] Merge: Draw a top-down obstacle display map. Filter the point cloud information along the Z-axis direction with a maximum filter. Since the point cloud information is only the surface of the obstacle, with the camera as the center, consider the area behind the blocked area by the obstacle as the obstacle area and assign values. Then obtain the top-down obstacle information, where the area that will collide with the plane is marked in red, as Figure 2 shown. The area that is close or above the ground surface but will not collide with the wings and poses a danger is marked with the first color (yellow), and the safe area is marked with the second color (green). Then obtain the top-down obstacle display map as Figure 1 shown;

[0053] Top-down obstacle display map, Figure 1 The top-down obstacle display map in

[0054] As described above, it is only the specific implementation manner of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed by the present disclosure should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.

Claims

1. An aircraft taxiing anti-collision assistance method based on binocular vision, characterized in that, Binocular cameras are respectively installed in the nose position and in the two passenger cabins on both sides adjacent to the front side of the wing. The binocular cameras are arranged horizontally. The method includes: Obtain the image data collected by each camera and the external point cloud data at the current moment, and determine the depth information of each binocular camera. Among them, determining the depth information of each binocular camera includes obtaining the original disparity information output by each frame of the binocular camera, the RGB information collected by the camera, and the camera parameter information; processing the original disparity information with the camera parameters, and discarding the peripheral boundary information with a width of three pixels, and converting the disparity information into depth information, satisfying: , where f is the focal length of the camera, Baseline is the camera baseline value, Dsp represents the disparity value at the image coordinate point (m, n); Z represents the vertical distance from the pixel point at the coordinate (m, n) in the image data to the plane where the camera lens is located; Perform filtering and filling on the depth information of each binocular camera horizontally to obtain a filled and filtered depth information matrix. Among them, performing filtering and filling on the depth information of each binocular camera horizontally includes obtaining the image data of each frame of the binocular camera, determining multiple depth values in the image data, and all depth information forms a depth information matrix. Horizontally traverse the depth information matrix in the horizontal direction to calculate the horizontal change rate value of the null points. The horizontal change rate value satisfies: , where is the depth value of the first non-zero point to the left of the point (m, n), is its ordinate value; is the depth value of the first non-zero point to the right, is its ordinate value. Set a threshold to determine whether the change rate value of the current point coordinates along the traversal direction is greater than the threshold. If so, do not perform filling. If not, fill the depth value of this point. After horizontal filling, fill along the diagonally downward direction and the diagonally upward direction in sequence, and then fill vertically to fill the remaining empty points. After multiple iterations, the depth information after filling and filtering is obtained. , forming a digital matrix after filling; Visualize the depth information matrix, and remove the point cloud information below the ground and higher than a preset multiple of the aircraft height. The setting of the preset multiple corresponds to the aircraft height and the aircraft model. Among them, visualizing the depth information matrix includes converting it into a color space representation according to the depth information; forming an initial viewable image, using colors to represent the distance value of the corresponding pixel points from the aircraft, and generating a depth map color representation according to the distance value; superimposing it on the original image to display a real-scene obstacle warning map, where the objects that may collide are filled or marked with a first color. The first color has a gradient characteristic of multiple colors, and as the distance value gets farther, the color gradually changes to green; Draw a top-down obstacle display map, filter the point cloud information along the Z-axis direction with the maximum value, take the camera as the center, regard the area behind the obstacle occlusion area as the obstacle area and assign values to obtain the top-down obstacle information. Among them, the area in collision with the aircraft is marked with the first color, and the safe area is marked with the second color.

2. The method according to claim 1, wherein The threshold value is , when the change rate , no filling is performed. When the change rate , the depth value of this point is filled, and the filled value satisfies: 。 3. The method according to claim 1, characterized in that, Calculate the coordinates of the filled digital matrix in the real space, including: Obtain the filled depth information, and calculate the x-axis and y-axis coordinate information of this point in the camera coordinate system according to the formula: Among them, f is the camera focal length; img width is the width of the image captured by the camera, img Heighth is the height of the image captured by the camera, Z(m, n) is the depth or distance of the coordinate (m, n) in the filled matrix, expressed as the true distance value of this point from the plane. X, Y, and Z establish a three-dimensional coordinate system with the lens. The direction of the lens pointing is the Z-axis, indicating the vertical distance of the pixel point of the image coordinate (m, n) from the plane where the camera lens is located. The lens points above the plane for the Y-axis, perpendicular to the X-axis and Z-axis, and points to the right of the imaging plane for the X-axis, obtaining the point cloud data of the camera coordinate system.

Images