Autonomous landing guidance method for aircraft in unknown environment based on Radon transform

Abstract

The present application relates to an autonomous landing guidance method for an aircraft based on Radon transform in an unknown environment. The method includes: obtaining semantic segmentation information, depth information and slope information of a monocular RGB image; gridding the monocular RGB image, calculating a regional evaluation index based on the semantic segmentation information, depth information and slope information in the grid, normalizing the regional evaluation index and mapping it to the grid position to obtain a two-dimensional grid scene information map; converting the two-dimensional grid scene information map into a binary image according to a threshold, performing connectivity detection on the binary image to obtain multiple candidate regions; performing discrete Radon transform on each candidate region to screen the optimal landing area that meets the landing requirements of the aircraft; constructing a local coordinate system on the optimal landing area, solving the relative position and posture between the local coordinate system and the camera coordinate system, and guiding the aircraft to complete autonomous landing. The use of this method can improve the efficiency and reliability of autonomous landing guidance of aircraft in unknown scenes.

Description

Technical Field

[0001] The present application relates to the field of monocular vision measurement technology, and in particular to an autonomous landing guidance method for an aircraft based on Radon transform in an unknown environment. Background Art

[0002] With the development of intelligent unmanned aerial vehicles, aircraft have shown great potential and application prospects in different fields, among which the safety and reliability of aircraft are crucial. When faced with emergencies such as mechanical failure, severe weather conditions or strong interference, the aircraft needs to land in time to protect ground personnel and airborne equipment to the greatest extent and reduce economic losses. Therefore, autonomous landing guidance of aircraft has become a research hotspot in this field, and higher requirements are placed on the intelligence level of aircraft. The relative posture measurement between the aircraft and the landing area is one of the core technologies for realizing autonomous landing guidance. Current autonomous landing technology for aircraft usually relies on global positioning systems, inertial navigation sensors, optoelectronic sensors, radars and other equipment, and is applied to known scenarios such as fixed airports. There is still a lack of effective technical solutions for unknown scenarios.

[0003] Unknown scenes are common in emergency situations and do not contain prior information such as feature points, descriptors, reference maps, cooperative signs and three-dimensional models that can be used to measure the relative pose of the aircraft for autonomous landing guidance. Therefore, traditional means such as global positioning systems, optoelectronic sensors and radars no longer meet the mission requirements. Airborne monocular cameras are widely used in the process of visual pose measurement for autonomous landing guidance of aircraft due to their low cost, simple structure and strong anti-interference ability, which can meet the needs of emergency autonomous landing missions in unknown scenes. In known scenes with cooperative signs or target three-dimensional information, airborne monocular cameras extract the image features of the target area to obtain the relative pose relationship between the aircraft and the landing area, providing a reference for the control system. However, these methods are difficult to achieve autonomous landing of aircraft in unknown scenes without prior information. With the development of monocular camera scene understanding and perception technology, autonomous landing guidance technology for aircraft based on monocular visual pose measurement has been realized in scenes with weak or no prior information. In existing research, the autonomous landing guidance methods for aircraft in unknown scenes based on vision are mainly divided into direct image detection methods and scene three-dimensional reconstruction methods. Direct image detection methods use information such as texture and geometry in the target scene to learn semantic features to determine the optimal landing area. However, this type of method focuses on the two-dimensional information of the scene, which is not enough to judge complex three-dimensional terrain information and is difficult to provide reliable relative pose information. Compared with direct detection methods, scene three-dimensional reconstruction strategies are more widely used. Although the landing environment is complex and has great differences, for the task of searching for a suitable landing area, the reconstructed point cloud data can further provide high-precision three-dimensional terrain information. However, since the three-dimensional reconstruction task has high requirements on the image viewing angle and number, and the dense point cloud file increases the computational complexity, it is often difficult to measure in real time in emergency autonomous landing of aircraft. In the existing practice of autonomous landing of aircraft in unknown scenes, researchers search for safe landing points and guide the aircraft to land through terrain conditions, but the research objects of these works are often rotorcraft. Compared with rotorcraft that can land in areas with smaller areas and lower terrain requirements, fixed-wing aircraft have more stringent requirements on the landing area. Under the condition of meeting the width requirements of the aircraft itself, higher requirements are placed on the measurement accuracy and robustness of the length of the landing area and ground conditions. Neither of the two existing strategies can accurately and robustly achieve the real-time autonomous landing vision guidance task of an aircraft in an unknown scene. Summary of the invention

[0004] Based on this, it is necessary to provide an autonomous landing guidance method for aircraft based on Radon transform in an unknown environment to address the above technical problems.

[0005] An autonomous landing guidance method for an aircraft based on Radon transform in an unknown environment, the method comprising:

[0006] Use the pre-trained multi-task neural network to obtain the semantic segmentation information, depth information and slope information of the monocular RGB image collected by the aircraft monocular camera;

[0007] The monocular RGB image is gridded, a regional evaluation index is calculated according to semantic segmentation information, depth information and slope information in the grid, and the regional evaluation index is normalized and mapped to the grid position to obtain a two-dimensional grid scene information map;

[0008] The two-dimensional grid scene information graph is converted into a binary image according to a preset suitable region scoring threshold, and connectivity detection is performed on the binary image to obtain a plurality of candidate regions;

[0009] Perform discrete Radon transform on each candidate area, and select the optimal landing area that meets the landing requirements of the aircraft by combining the area length, width and pixel information of the two-dimensional grid scene information map;

[0010] A two-dimensional key point set is constructed in the optimal landing area, and the three-dimensional key point set in the camera system is restored in combination with the depth information. The center of gravity of the three-dimensional key point set is used as the origin of the local coordinate system and the local coordinate system is fitted. The relative posture between the local coordinate system and the camera coordinate system is solved according to the correspondence between the two-dimensional key point set and the three-dimensional key point set in the local coordinate system, and the aircraft is guided to complete autonomous landing according to the relative posture.

[0011] An autonomous landing guidance device for an aircraft based on Radon transform in an unknown environment, the device comprising:

[0012] A 3D information extraction module is used to obtain semantic segmentation information, depth information, and slope information of a monocular RGB image captured by an aircraft monocular camera using a pre-trained multi-task neural network;

[0013] A scene information graph construction module is used to grid the monocular RGB image, calculate the regional evaluation index according to the semantic segmentation information, depth information and slope information in the grid, normalize the regional evaluation index and map it to the grid position to obtain a two-dimensional grid scene information graph;

[0014] A candidate region acquisition module is used to convert the two-dimensional grid scene information graph into a binary image according to a preset suitable region scoring threshold, and perform connectivity detection on the binary image to obtain multiple candidate regions;

[0015] The optimal landing area screening module is used to perform discrete Radon transform on each candidate area, and screen the optimal landing area that meets the landing requirements of the aircraft by combining the area length, width and pixel information of the two-dimensional grid scene information map;

[0016] The autonomous landing guidance module is used to construct a two-dimensional key point set in the optimal landing area, and restore the three-dimensional key point set under the camera system in combination with the depth information. The center of gravity of the three-dimensional key point set is used as the origin of the local coordinate system and the local coordinate system is fitted. The relative posture between the local coordinate system and the camera coordinate system is solved according to the correspondence between the two-dimensional key point set and the three-dimensional key point set in the local coordinate system, and the aircraft is guided to complete autonomous landing according to the relative posture.

[0017] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0018] Use the pre-trained multi-task neural network to obtain the semantic segmentation information, depth information and slope information of the monocular RGB image collected by the aircraft monocular camera;

[0019] The monocular RGB image is gridded, a regional evaluation index is calculated according to semantic segmentation information, depth information and slope information in the grid, and the regional evaluation index is normalized and mapped to the grid position to obtain a two-dimensional grid scene information map;

[0020] The two-dimensional grid scene information graph is converted into a binary image according to a preset suitable region scoring threshold, and connectivity detection is performed on the binary image to obtain a plurality of candidate regions;

[0021] Perform discrete Radon transform on each candidate area, and select the optimal landing area that meets the landing requirements of the aircraft by combining the area length, width and pixel information of the two-dimensional grid scene information map;

[0022] A two-dimensional key point set is constructed in the optimal landing area, and the three-dimensional key point set in the camera system is restored in combination with the depth information. The center of gravity of the three-dimensional key point set is used as the origin of the local coordinate system and the local coordinate system is fitted. The relative posture between the local coordinate system and the camera coordinate system is solved according to the correspondence between the two-dimensional key point set and the three-dimensional key point set in the local coordinate system, and the aircraft is guided to complete autonomous landing according to the relative posture.

[0023] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0024] Use the pre-trained multi-task neural network to obtain the semantic segmentation information, depth information and slope information of the monocular RGB image collected by the aircraft monocular camera;

[0025] The monocular RGB image is gridded, a regional evaluation index is calculated according to semantic segmentation information, depth information and slope information in the grid, and the regional evaluation index is normalized and mapped to the grid position to obtain a two-dimensional grid scene information map;

[0026] The two-dimensional grid scene information graph is converted into a binary image according to a preset suitable region scoring threshold, and connectivity detection is performed on the binary image to obtain a plurality of candidate regions;

[0027] Perform discrete Radon transform on each candidate area, and select the optimal landing area that meets the landing requirements of the aircraft by combining the area length, width and pixel information of the two-dimensional grid scene information map;

[0028] A two-dimensional key point set is constructed in the optimal landing area, and the three-dimensional key point set in the camera system is restored in combination with the depth information. The center of gravity of the three-dimensional key point set is used as the origin of the local coordinate system and the local coordinate system is fitted. The relative posture between the local coordinate system and the camera coordinate system is solved according to the correspondence between the two-dimensional key point set and the three-dimensional key point set in the local coordinate system, and the aircraft is guided to complete autonomous landing according to the relative posture.

[0029] The above-mentioned autonomous landing guidance method for aircraft based on Radon transform in unknown environment extracts semantic segmentation, depth and slope information from monocular RGB images through pre-trained multi-task neural network, comprehensively characterizes scene characteristics, provides reliable data support for suitable landing area evaluation, generates two-dimensional grid scene information map based on semantic, flatness and slope information, realizes multi-dimensional comprehensive expression of complex scenes, enhances the flexibility and accuracy of suitable landing area search, uses discrete Radon transform to extract linear features of candidate areas, and combines the length, width and pixel information of the area to accurately screen out the optimal suitable landing area that meets the landing requirements of the aircraft, combines the depth information to restore the three-dimensional key point set, calculates the relative pose of the camera coordinate system and the local coordinate system, solves the scale problem of monocular vision, and realizes high-precision relative pose measurement. The embodiment of the present invention significantly improves the efficiency and reliability of autonomous landing guidance of aircraft in unknown scenes, and meets the needs of aircraft of different sizes to achieve safe and autonomous landing in complex unknown environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is an application scenario diagram of an aircraft autonomous landing guidance method based on Radon transform in an unknown environment in an embodiment;

[0031] Figure 2 A two-dimensional scene grid information diagram in one embodiment;

[0032] Figure 3Schematic diagram of a process of selecting an appropriate region for a drop-in region in an embodiment, wherein (a) is a schematic diagram of selecting a candidate region, (b) is a schematic diagram of Radon transform of a candidate region, (c) is a schematic diagram of a projection value, and (d) is a schematic diagram of a direction of an appropriate region for a drop-in region;

[0033] Figure 4 A schematic diagram of establishing a regional parameterized representation and a local coordinate system in one embodiment;

[0034] Figure 5 A schematic diagram of a flow chart of a monocular vision measurement method for autonomous landing guidance of an aircraft in any unknown environment in an embodiment;

[0035] Figure 6 A schematic diagram of monocular vision measurement for autonomous landing guidance of an aircraft in one embodiment;

[0036] Figure 7 It is a structural block diagram of an aircraft autonomous landing guidance device based on Radon transform in an unknown environment in one embodiment;

[0037] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0039] In one embodiment, Figure 1 As shown, a method for autonomous landing guidance of an aircraft based on Radon transform in an unknown environment is provided, comprising the following steps:

[0040] Step 102, using a pre-trained multi-task neural network to obtain semantic segmentation information, depth information, and slope information of a monocular RGB image captured by a monocular camera of an aircraft.

[0041] The semantic segmentation information, depth information and slope information can be extracted from the monocular RGB image of the aircraft through a neural network, wherein the semantic segmentation information can distinguish different terrain categories (such as roads, grass, obstacles, etc.), thereby quickly identifying possible suitable landing areas, and the depth information provides basic data for subsequent regional flatness assessment, helping to identify terrain changes and height differences, and the slope information can quantify the degree of regional inclination and exclude areas that are too steep or dangerous. Step 102 can be implemented through the records in the Chinese invention patent CN118623886A, and the specific implementation method will not be repeated here.

[0042] Step 104, gridding the monocular RGB image, calculating the regional evaluation index according to the semantic segmentation information, depth information and slope information in the grid, normalizing the regional evaluation index and mapping it to the grid position, and obtaining a two-dimensional grid scene information map.

[0043] Grid processing discretizes the scene and introduces a unified analysis framework, which facilitates the fusion and processing of the multi-dimensional information of the scene. The two-dimensional grid scene information map uniformly represents the multi-dimensional information (semantics, flatness, slope) in the form of an image, overcoming the problem of insufficient feature description in the traditional suitable landslide area assessment method. The present invention proposes a new information representation form, making the search for suitable landslide areas more flexible, comprehensive and reliable.

[0044] Step 106, converting the two-dimensional grid scene information graph into a binary image according to a preset suitable region scoring threshold, performing connectivity detection on the binary image, and obtaining a plurality of candidate regions.

[0045] Potential suitable areas can be quickly screened through preset scoring thresholds, greatly reducing the scope that needs to be analyzed. Connectivity detection can eliminate scattered and invalid areas, further improving calculation efficiency.

[0046] Step 108, performing discrete Radon transform on each candidate area, combining the area length, width and pixel information of the two-dimensional grid scene information map to select the optimal landing area that meets the landing requirements of the aircraft.

[0047] The linear features extracted by discrete Radon transform can effectively identify the directionality and length of the region, which is suitable for searching runways or flat areas. The length, width and pixel distribution of the region are combined to further comprehensively evaluate the regional suitability, and the optimal suitable landing area that meets the landing requirements is flexibly and accurately found, which improves the reliability of the selection of the suitable landing area.

[0048] Step 110, construct a two-dimensional key point set in the optimal landing area, and restore the three-dimensional key point set in the camera system in combination with the depth information, use the center of gravity of the three-dimensional key point set as the origin of the local coordinate system and fit the local coordinate system, solve the relative posture between the local coordinate system and the camera coordinate system according to the correspondence between the two-dimensional key point set and the three-dimensional key point set in the local coordinate system, and guide the aircraft to complete autonomous landing according to the relative posture.

[0049] The scale problem of monocular vision is solved through three-dimensional key point recovery and relative pose estimation, and high-precision pose measurement is achieved.

[0050] In the above-mentioned autonomous landing guidance method for aircraft based on Radon transform in unknown environment, semantic segmentation, depth and slope information in monocular RGB images are extracted through pre-trained multi-task neural network, scene characteristics are fully characterized, reliable data support is provided for suitable landing area evaluation, and a two-dimensional grid scene information map is generated based on semantic, flatness and slope information to achieve multi-dimensional comprehensive expression of complex scenes, enhance the flexibility and accuracy of suitable landing area search, use discrete Radon transform to extract linear features of candidate areas, and combine the length, width and pixel information of the area to accurately screen out the optimal suitable landing area that meets the landing requirements of the aircraft, combine the depth information to restore the three-dimensional key point set, calculate the relative pose of the camera coordinate system and the local coordinate system, solve the scale problem of monocular vision, and achieve high-precision relative pose measurement. The embodiment of the present invention significantly improves the efficiency and reliability of autonomous landing guidance of aircraft in unknown scenes, and meets the needs of aircraft of different sizes to achieve safe and autonomous landing in complex unknown environments.

[0051] In one embodiment, calculating the regional evaluation index based on the semantic segmentation information, depth information and slope information in the grid includes: respectively calculating the grid semantic segmentation information, grid flatness information and grid slope information based on the semantic segmentation information, depth information and slope information in the grid; and calculating the regional evaluation index of the grid based on the grid semantic segmentation information, grid flatness information and grid slope information.

[0052] In one embodiment, the grid semantic segmentation information, grid flatness information and grid slope information are respectively calculated according to the semantic segmentation information, depth information and slope information in the grid, including: obtaining the grid semantic segmentation information according to the proportion of pixels in the suitable drop area category in the grid area; obtaining the grid flatness information according to the variation range of the depth in the grid; and obtaining the grid slope information according to the variation range of the slope in the grid.

[0053] In one embodiment, the regional evaluation index is:

[0054] ;

[0055] in, is the regional evaluation index, is the grid semantic segmentation information, is the grid flatness information, is the grid slope information, , and is the weight coefficient.

[0056] In one embodiment, normalizing the regional evaluation index and mapping it to the grid position to obtain a two-dimensional grid scene information graph includes: normalizing the regional evaluation index and mapping it to the grid position to obtain a two-dimensional grid scene information graph as follows:

[0057] ;

[0058] in, It is a two-dimensional grid scene information map. is the height of the monocular RGB image, is the width of the monocular RGB image, is the gridding factor.

[0059] like Figure 2 As shown, a two-dimensional scene grid information map is provided. Based on the semantic segmentation information, depth information and normal information output by the multi-task neural network, the present invention designs a two-dimensional grid scene information map for searching and evaluating suitable drop areas in unknown scenes, such as Figure 2 As shown in Figure 1, it contains the semantics, flatness, and slope of the target area. The gridding coefficient is set to , that is, for the size Original image according to The scale is meshed to obtain a size of Two-dimensional grid diagram of , each grid size is The semantic segmentation information, depth information and slope information in each grid are used to generate regional evaluation indicators. ,

[0060] (1)

[0061] The grid semantic segmentation information is represented as , whose formula is defined as

[0062] (2)

[0063] in For images In the classification label, based on semantic segmentation, the present invention defines targets such as roads and grass as landing area categories , define objects such as trees, vehicles, and buildings as non-landing area categories. The corresponding grid The categories of pixels, is the total number of pixels in the landing area. In addition, the grid flatness information is determined by the relative depth change amplitude, which is used to evaluate the flatness of the landing area. Defined as

[0064] (3)

[0065] in The corresponding grid The relative depth value of the pixel, is the average relative depth within the grid. Considering the flatness evaluation between grids, the length and width of the flatness calculation are increased by The purpose is to evaluate the flatness information of adjacent grids. Similarly, the slope metric is measured by the slope change in the grid area. The definition is as follows

[0066] (4)

[0067] is the normal vector value corresponding to each pixel point, is the vertical ground direction in the world coordinate system, and the present invention sets it as axis, is the vector modulus operation, is the inner product of the vector. Grid slope information Defined as

[0068] (5)

[0069] The corresponding grid The slope value of each pixel, is the average value of the slope. So the two-dimensional grid map Defined as

[0070] (6)

[0071] adjust The coefficient of variation of each indicator in , through the normalization operation The optimal two-dimensional grid scene information image can be generated to provide a basis for searching suitable landing areas.

[0072] In one embodiment, a discrete Radon transform is performed on each candidate area, and the optimal suitable landing area that meets the landing requirements of the aircraft is screened in combination with the area length, width and pixel information of the two-dimensional grid scene information map, including: a discrete Radon transform is performed on each candidate area to obtain the projection distribution information of the pixels in the candidate area in different directions, the longest line segment is determined according to the projection distribution information, and the area with the longest line segment in the candidate area is taken as the suitable landing area; the suitable landing area that passes the connectivity test and meets the aircraft landing safety width and the minimum landing length is selected as the candidate suitable landing area, and the optimal suitable landing area is selected according to the pixel mean of the two-dimensional grid scene information map in the candidate suitable landing area.

[0073] like Figure 3As shown, a schematic diagram of the process of selecting a suitable landing area is provided, wherein (a) is a schematic diagram of candidate area selection, (b) is a schematic diagram of Radon transform of candidate areas, (c) is a schematic diagram of projection values, and (d) is a schematic diagram of the direction of a suitable landing area. Based on scene analysis, for fixed-wing aircraft landing missions, the longer the suitable landing area is, the more suitable it is for the aircraft to land while meeting the minimum width requirement. In order to be able to quickly and accurately locate the suitable landing area in the scene analysis results, the present invention proposes a suitable landing area search method based on Radon transform, obtains candidate areas in a two-dimensional grid scene information image, and performs Radon transform on each area to obtain a set of suitable landing areas. The two-dimensional grid scene information image is , according to the preset suitable drop area scoring threshold Convert the two-dimensional grid scene information map into a binary image. Figure 3 As shown, the two-pass algorithm is used to obtain the binary image candidate regions, forming a candidate region set, denoted as The present invention is in each candidate area Discrete Radon transform algorithm is performed in. Candidate region Add-in Rectangle as discrete Radon transform input image .

[0074] Under the condition that the width requirement of fixed-wing aircraft is met, it is necessary to Find the longest area that meets the landing requirements. Discrete Radon transform is to input the candidate area image The straight line where the middle suitable drop area is located Projection. Straight line Defined as:

[0075] (7)

[0076] Represents a straight line direction angle, Represents a straight line Distance from the origin. Projection value Represents a straight line along The present invention performs Radon transformation by adding the angle and distance Discretization, choose Different angles :

[0077] (8)

[0078] distance The range is:

[0079] (9)

[0080] in is the length of the image diagonal:

[0081] (10)

[0082] After calculating the projection values ​​for all angles and distances, find the one with the strongest projection value. The angle corresponding to the point and the corresponding distance According to the maximum value, we can get The slope and intercept of the strongest directional line in the image coordinate system:

[0083] (11)

[0084] In the In the candidate area The area where the straight line is located is the set of suitable drop areas and is recorded as It is worth noting that from the candidate region The suitable drop area obtained in The total is recorded as . Connectivity detection is performed again in all suitable landing areas, and the pixel mean of the two-dimensional grid scene information map in the area is solved as a reference for selecting suitable landing areas. In order to facilitate experimental analysis and testing, the safe landing width of fixed-wing aircraft is met. Based on the screening, compliance with the minimum landing length The optimal drop area is obtained by comparing the pixel mean of the two-dimensional grid scene information map in the area. .

[0085] In one embodiment, taking the centroid of the three-dimensional key point set as the origin of the local coordinate system and fitting the local coordinate system includes: obtaining the origin of the local coordinate system according to the centroid of the three-dimensional key point set, and fitting the local coordinate system using the principal component analysis method, wherein the first axis is the main direction of the adaptive region, the second axis is the direction perpendicular to the adaptive region, and the third axis is the direction perpendicular to the first axis and the second axis.

[0086] like Figure 4 As shown in the figure, a schematic diagram of regional parameterized representation and establishment of local coordinate system is provided. Sparse key points are used to regionalize the optimal landing area. On the basis of parameterized representation of the landing area, the relative pose between frames of the monocular image sequence measured by the airborne inertial unit is combined to restore the absolute scale information, thereby realizing high-precision relative pose measurement between the aircraft and the landing area.

[0087] Specifically, after completing the landing area selection, it is necessary to solve the relative posture relationship between the aircraft and the landing area to provide guidance information for autonomous landing guidance. Determine the minimum landing width of the aircraft Based on the Radon transform results, the region is searched The longest rectangle is taken as the optimal drop area .like Figure 4 As shown in the figure, the green area is the optimal landing area The present invention uses key points to parameterize the optimal drop area. Establish a two-dimensional key point set with fixed internal intervals , which contains 2D boundary points . Using the depth information, the three-dimensional key point set under the camera system can be restored , which contains 3D key points under the camera system . Set up a sparse keypoint set The center of gravity is the origin of the local coordinate system , which uses the PCA (principal component analysis) method to fit the local coordinate system, where Axis is the axis vector of the fitting area, which is the main direction of key point fitting and the direction of the road; The axis vertical suitable drop area is upward; Axis perpendicular to Plane. Set the key points Transformed to the local coordinate system, recorded as , and finally establish a two-dimensional key point set and the set of 3D key points in the local coordinate system The corresponding relationship between the local coordinate system of the landing area and the camera coordinate system is realized by solving the PnP problem. Solve and complete the visual measurement of autonomous landing guidance for aircraft.

[0088] Since the depth output by the multi-task neural network is relative depth information, the relative pose parameters between the aircraft and the landing area lack scale information. The present invention uses a multi-view geometry method to simulate the inertial navigation to obtain the relative pose between frames and restore the scene scale information. In practical applications, the scene scale information can be obtained with the help of an aircraft altimeter, a laser rangefinder or a multi-view geometry method based on an inertial navigation system. , the final pose parameters are The landing mission of a rotorcraft has lower requirements on the area size and terrain, and because it can take off and land vertically, the path planning is simpler than that of a fixed-wing aircraft, so the method adopted by the present invention is fully applicable.

[0089] In a specific embodiment, Figure 5As shown in the figure, a flow chart of a monocular vision measurement method for autonomous landing guidance of an aircraft in an arbitrary unknown environment is provided, including comprehensive analysis of three-dimensional scenes, construction of two-dimensional grid scene information graphs, suitable landing area search based on Radon transform, and relative pose estimation, wherein:

[0090] (1) Comprehensive analysis of 3D scenes: Using a multi-task network, the semantic segmentation results, depth information results, and slope information results are simultaneously predicted based on the monocular RGB image. The segmentation information provides length and width results, the depth information is used to measure the flatness of the terrain and restore 3D key points, and the slope information is used to evaluate the regional slope.

[0091] (2) Construction of a two-dimensional grid scene information map: After obtaining the semantic segmentation results, depth information results, and slope information in the image in step (1), the two-dimensional grid scene information map of the image is constructed using the measurement information.

[0092] (3) Search for suitable landing area based on Radon transform: Based on steps (1) and (2), a two-pass algorithm is used to obtain the connected areas in the information graph. A Radon transform is performed in each connected area to obtain the optimal landing direction. The optimal suitable landing area is selected based on the characteristics of the aircraft itself and the corresponding two-dimensional grid scene information.

[0093] (4) Regional parameterization and relative pose estimation: Based on steps (1) to (3), the optimal landing area is obtained, and a two-dimensional key point set is established in the optimal landing area. The three-dimensional key point set in the camera system can be restored using depth information. The PCA method is used to fit the local coordinate system, and the three-dimensional key point set in the camera coordinate system is converted to the three-dimensional key point set in the local coordinate system. The correspondence between the two-dimensional key point set of the image and the three-dimensional key point set in the local coordinate system is established to achieve the relative pose solution between the optimal landing area and the aircraft.

[0094] like Figure 6 As shown in the figure, a schematic diagram of monocular vision measurement for autonomous landing guidance of an aircraft is provided. The specific measurement method is: RGB images in an unknown environment are collected by a monocular camera, and semantic segmentation, depth estimation and slope regression are performed using a multi-task neural network to construct a two-dimensional grid scene information map; then, the optimal suitable landing area is searched based on Radon transform in the candidate area; finally, the suitable landing area is parameterized by key points, and the absolute scale information is restored by combining the airborne inertial unit to realize the relative posture measurement between the suitable landing area and the aircraft, providing reliable measurement information for autonomous landing guidance.

[0095] It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0096] In one embodiment, Figure 7 As shown, an autonomous landing guidance device for an aircraft based on Radon transform in an unknown environment is provided, comprising:

[0097] The three-dimensional information extraction module 702 is used to obtain semantic segmentation information, depth information and slope information of the monocular RGB image collected by the monocular camera of the aircraft using a pre-trained multi-task neural network;

[0098] The scene information map construction module 704 is used to grid the monocular RGB image, calculate the regional evaluation index according to the semantic segmentation information, depth information and slope information in the grid, normalize the regional evaluation index and map it to the grid position to obtain a two-dimensional grid scene information map;

[0099] The candidate region acquisition module 706 is used to convert the two-dimensional grid scene information map into a binary image according to a preset suitable region scoring threshold, perform connectivity detection on the binary image, and obtain multiple candidate regions;

[0100] The optimal landing area screening module 708 is used to perform discrete Radon transform on each candidate area, and screen the optimal landing area that meets the landing requirements of the aircraft by combining the area length, width and pixel information of the two-dimensional grid scene information map;

[0101] The autonomous landing guidance module 710 is used to construct a two-dimensional key point set in the optimal landing area, and restore the three-dimensional key point set under the camera system in combination with the depth information, use the center of gravity of the three-dimensional key point set as the origin of the local coordinate system and fit the local coordinate system, solve the relative posture between the local coordinate system and the camera coordinate system according to the correspondence between the two-dimensional key point set and the three-dimensional key point set in the local coordinate system, and guide the aircraft to complete autonomous landing according to the relative posture.

[0102] For the specific limitations of the aircraft autonomous landing guidance device based on Radon transform in an unknown environment, please refer to the limitations of the aircraft autonomous landing guidance method based on Radon transform in an unknown environment mentioned above, which will not be repeated here. Each module in the above-mentioned aircraft autonomous landing guidance device based on Radon transform in an unknown environment can be fully or partially implemented by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0103] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an autonomous landing guidance method for an aircraft based on Radon transform in an unknown environment is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0104] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0105] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0106] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0107] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0108] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0109] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for autonomous landing guidance of an aircraft based on Radon transform in an unknown environment, characterized in that: The method comprises: Use the pre-trained multi-task neural network to obtain the semantic segmentation information, depth information and slope information of the monocular RGB image collected by the aircraft monocular camera; The monocular RGB image is gridded, a regional evaluation index is calculated according to semantic segmentation information, depth information and slope information in the grid, and the regional evaluation index is normalized and mapped to the grid position to obtain a two-dimensional grid scene information map; The two-dimensional grid scene information graph is converted into a binary image according to a preset suitable region scoring threshold, and connectivity detection is performed on the binary image to obtain a plurality of candidate regions; Perform discrete Radon transform on each candidate area, and select the optimal landing area that meets the landing requirements of the aircraft by combining the area length, width and pixel information of the two-dimensional grid scene information map; A two-dimensional key point set is constructed in the optimal landing area, and the three-dimensional key point set in the camera system is restored in combination with the depth information. The center of gravity of the three-dimensional key point set is used as the origin of the local coordinate system and the local coordinate system is fitted. The relative posture between the local coordinate system and the camera coordinate system is solved according to the correspondence between the two-dimensional key point set and the three-dimensional key point set in the local coordinate system, and the aircraft is guided to complete autonomous landing according to the relative posture; The regional evaluation indicators are normalized and mapped to the grid positions, and the two-dimensional grid scene information graph is obtained, including: After normalizing the regional evaluation index, it is mapped to the grid position, and the two-dimensional grid scene information graph is obtained as follows: ； in, It is a two-dimensional grid scene information map. is the height of the monocular RGB image, is the width of the monocular RGB image, is the gridding factor.

2. The method according to claim 1, characterized in that The regional evaluation indicators calculated based on the semantic segmentation information, depth information and slope information in the grid include: Calculate mesh semantic segmentation information, mesh flatness information and mesh slope information respectively according to the semantic segmentation information, depth information and slope information in the mesh; The regional evaluation index of the grid is calculated based on the grid semantic segmentation information, grid flatness information and grid slope information.

3. The method according to claim 2, characterized in that According to the semantic segmentation information, depth information and slope information in the grid, the grid semantic segmentation information, grid flatness information and grid slope information are calculated respectively, including: The grid semantic segmentation information is obtained according to the proportion of pixels in the suitable drop area category in the grid area; The grid flatness information is obtained according to the variation range of the depth in the grid; The grid slope information is obtained according to the variation of the slope within the grid.

4. The method according to claim 2, characterized in that: The regional evaluation indicators are: in, is the regional evaluation index, is the grid semantic segmentation information, is the grid flatness information, is the grid slope information, , and is the weight coefficient.

5. The method according to claim 1, characterized in that The discrete Radon transform is performed on each candidate area, and the optimal landing area that meets the landing requirements of the aircraft is selected by combining the area length, width and pixel information of the two-dimensional grid scene information map: Perform discrete Radon transform on each candidate area to obtain the projection distribution information of the pixels in the candidate area in different directions, determine the longest line segment according to the projection distribution information, and take the area with the longest line segment in the candidate area as the suitable area; The suitable landing areas that pass the connectivity test and meet the aircraft landing safety width and minimum landing length are selected as candidate suitable landing areas, and the optimal suitable landing area is selected according to the pixel mean of the two-dimensional grid scene information map in the candidate suitable landing area.

6. The method according to claim 1, characterized in that Taking the centroid of the three-dimensional key point set as the origin of the local coordinate system and fitting the local coordinate system includes: The origin of the local coordinate system is obtained according to the centroid of the three-dimensional key point set, and the local coordinate system is fitted using the principal component analysis method, where the first axis is the main direction of the suitable landing area, the second axis is the direction perpendicular to the suitable landing area, and the third axis is the direction perpendicular to the first axis and the second axis.

7. The method according to claim 1, characterized in that The multi-task neural network includes a backbone network, a kernel generation module, a feature encoding module and a regression module; The backbone network is used to extract features from the input monocular RGB image to obtain single-level features and high-resolution features; The kernel generation module is used to generate kernels adapted to different tasks according to the single-level features, and obtain a classification kernel, a segmentation kernel, a depth kernel and a slope kernel; The feature encoding module is used to encode the high-resolution features to obtain a segmentation encoding map, a depth encoding map and a slope encoding map; The regression module is used to convolve the segmentation coding map, the depth coding map and the slope coding map according to the segmentation kernel, the depth kernel and the slope kernel respectively to obtain semantic segmentation information, depth information and slope information.

8. The method according to claim 1, characterized in that The steps for training a multi-task neural network include: The multi-task neural network is trained according to the loss function and the training samples to obtain a trained multi-task neural network; the loss function includes a category loss function, a segmentation loss function, a depth estimation loss function and a slope estimation loss function.

Images