An aircraft forced landing assisting method and device, electronic equipment and storage medium

Abstract

This application provides an emergency landing assistance method, device, electronic device, and storage medium for an aircraft. The method includes acquiring flight environment images collected by airborne visual sensors available on the aircraft at the current moment, and acquiring airport environment images collected by ground visual sensors at the airport where the aircraft is to land at the current moment; inputting the flight environment images into the airborne image feature extraction branch to obtain a global feature map and aligning it with the field of view of the ground visual sensors based on the pose of the airborne visual sensors; inputting the airport environment images into the airport image feature extraction branch to obtain local feature maps; inputting the aligned global feature map and local feature map into a multi-branch fusion network, adaptively allocating fusion weights, and performing multi-level weighted fusion to generate a fused feature map; inputting the fused feature map into a decoding module to identify and locate the target object, thereby generating emergency landing visual assistance information for use by the crew.

Description

Technical Field

[0001] This application relates to the field of aviation technology, and more specifically, to an aircraft emergency landing assistance method, device, electronic equipment, and storage medium. Background Technology

[0002] During flight, airborne visual sensors are core components for crew members to assess the flight environment and identify airport runways and surrounding obstacles. Their core function is to acquire real-time images of the scene below and in front of the aircraft, providing crucial visual support for emergency flight scenarios such as forced landings. However, due to complex factors such as extreme weather, mechanical wear, and electromagnetic interference, airborne visual sensors are prone to partial failure. This manifests as damage to some cameras, distorted images, or signal interruptions, directly leading to the loss of critical visual data and an inability to fully and clearly present the airport runway and its surrounding environment. When airborne visual sensors partially fail and cannot be repaired in a timely manner, if an emergency landing is required, the crew will lose vital visual references, making it difficult to accurately determine the true location, boundary range, and distribution of surrounding obstacles on the runway, significantly increasing the risk of an emergency landing and potentially causing a safety accident.

[0003] Traditional guidance methods rely solely on radar coordinate data, lacking intuitive information support and failing to accurately reconstruct the true location of the runway and its surrounding environment. This makes them ill-suited for emergency landings in complex airport environments such as runway obstruction and severe weather. Summary of the Invention

[0004] The purpose of this application is to provide an aircraft emergency landing assistance method, device, electronic device, and storage medium to solve the technical problems of unintuitive information support and poor data reliability in existing aircraft emergency landing guidance methods.

[0005] In a first aspect, the present invention provides an emergency landing assistance method for an aircraft, the method comprising responding to an emergency landing assistance request, acquiring flight environment images collected by airborne visual sensors available on the aircraft at the current moment, and acquiring airport environment images collected by ground visual sensors at the airport where the aircraft is to land at the current moment. The flight environment image is input into the airborne image feature extraction branch to obtain a global feature map and align it with the field of view of the ground vision sensor based on the pose of the airborne vision sensor. Input the airport environment image into the airport image feature extraction branch to obtain the local feature map; The aligned global feature map and local feature map are input into a multi-branch fusion network, fusion weights are adaptively assigned, and multi-level weighted fusion is performed to generate a fused feature map. The fused feature map is input into the decoding module to identify and locate the target object. Combined with the real-time flight status of the aircraft, visual aid information for emergency landing is generated for the crew. The target object includes at least the runway and obstacles.

[0006] In an optional implementation, the global feature map is aligned in the following manner: For each ground vision sensor, determine the first coordinates of the four corner points of the field of view of that ground vision sensor in the three-dimensional world coordinate system corresponding to the airport; For each airborne vision sensor, an extrinsic parameter matrix is ​​determined based on the pose of the airborne vision sensor at the current moment, so as to transform the first coordinates of the four corner points of the ground vision sensor and obtain the second coordinates of the four corner points in the camera coordinate system. For each airborne vision sensor, based on the intrinsic parameter matrix of that airborne vision sensor, the second coordinates are transformed to obtain the third coordinates of the four corner points in the pixel coordinate system; The dynamic region of interest for the current frame is determined based on the minimum bounding rectangle corresponding to the third coordinate. The dynamic region of interest is input into the region of interest alignment module, which then extracts and interpolates it in the global feature map to obtain the aligned global feature map.

[0007] In an optional implementation, the airborne image feature extraction branch includes a local feature coding unit, a spatial location embedding unit, a serialization and neighborhood enhancement unit, and a long-distance dependency modeling unit, wherein, The local feature encoding unit receives the flight environment image, performs two-dimensional convolution plus silicon linear unit operation, and outputs an initial local feature map; The spatial location embedding unit receives the initial local feature map, adds row and column coordinate information to the initial local feature map, and outputs a feature map carrying spatial location information. The serialization and neighborhood enhancement unit receives a feature map carrying spatial location information, rearranges it into a one-dimensional sequence, performs a one-dimensional convolution operation on the sequence, and outputs enhanced sequence features. The long-distance dependency modeling unit receives enhanced sequence features, performs state space scanning modeling along the row and column directions respectively, and adds the two scan results to output a global feature map representing the macro layout of the runway and the distribution of long-distance obstacles.

[0008] In an optional implementation, the airport image feature extraction branch includes a detail feature extraction unit and a feature matching unit, wherein, The detail feature extraction unit receives airport environmental images, performs multi-layer stacked two-dimensional convolution and linear operations, and outputs a detail feature map containing runway marking edges, surface textures and obstacle outlines; The feature matching unit receives the detailed feature map, adjusts its spatial size and number of channels, and outputs a local feature map with the same spatial size and number of channels as the global feature map.

[0009] In an optional implementation, the multi-branch fusion network includes a feature construction unit, a context vector generation unit, a weight dynamic generation unit, and a weighted fusion unit, wherein, The feature construction unit receives the aligned global feature map and local feature map, concatenates them according to the channel direction, and outputs the first type of feature, the second type of feature and the third type of feature respectively after convolution operation; wherein, the first type of feature is the aligned global feature map, the second type of feature is the local feature map, and the third type of feature is the purified cross-modal fusion feature map; The context vector generation unit receives the first type of features and the second type of features, performs global average pooling on them respectively, obtains two statistical vectors respectively, concatenates them according to the channel direction, and outputs a joint scene context vector after convolution. The weight dynamic generation unit receives the joint scene context vector, inputs it into the fully connected layer for linear mapping, and processes it through a normalized exponential function to output three dynamically fused weights. The weighted fusion unit receives the first type of features, the second type of features, the third type of features, and three dynamic fusion weights. It performs a weighted summation based on the three dynamic fusion weights to output a fused feature map.

[0010] In an optional implementation, the context vector generation unit is specifically used to perform the following steps: Receive the first type of features and the second type of features; Perform global average pooling on the first type of features to obtain a first statistical vector that retains only channel statistics; Perform global average pooling on the second type of features to obtain a second statistical vector that retains only channel statistics. The first statistical vector and the second statistical vector are concatenated along the channel direction to obtain the concatenated vector; Perform convolution operations on the concatenated vectors to fuse information and suppress noise at the channel dimension, and output a joint scene context vector.

[0011] In an optional implementation, the weighted fusion unit is specifically used to perform the following operations: It receives the first type of feature, the second type of feature, the third type of feature, and three dynamic fusion weights; The three dynamic fusion weights are extended to the same spatial dimension as the first type of feature, the second type of feature, and the third type of feature, respectively; The first type of features are weighted pixel by pixel based on the expanded first dynamic fusion weight; The second type of features are weighted pixel-by-pixel based on the extended second dynamic fusion weight; The third type of features are weighted pixel-by-pixel based on the extended third dynamic fusion weight; The three pixel-by-pixel weighted results are summed element-by-element to output a fused feature map.

[0012] Secondly, the present invention provides an aircraft emergency landing assistance device, the device comprising: The response module is used to respond to emergency landing assistance requests, obtain flight environment images collected by the onboard visual sensors available on the aircraft at the current moment, and obtain airport environment images collected by the ground visual sensors at the airport where the aircraft is to land at the current moment. The feature extraction module is used to input flight environment images into the airborne image feature extraction branch, obtain a global feature map, and align it with the field of view of the ground vision sensor based on the pose of the airborne vision sensor. Input the airport environment image into the airport image feature extraction branch to obtain the local feature map; The fusion module is used to input the aligned global feature map and local feature map into the multi-branch fusion network, adaptively allocate fusion weights, and perform multi-level weighted fusion to generate a fused feature map. The output module is used to input the fused feature map into the decoding module to identify and locate the target object. Combined with the real-time flight status of the aircraft, it generates visual assistance information for emergency landing for the crew. The target object includes at least the runway and obstacles.

[0013] Thirdly, the present invention provides an electronic device, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of any of the emergency landing assistance methods for an aircraft as described in the foregoing embodiments.

[0014] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the emergency landing assistance method for any of the aforementioned embodiments of an aircraft.

[0015] This application provides an aircraft emergency landing assistance method, device, electronic device, and storage medium. The method includes responding to an emergency landing assistance request, acquiring flight environment images collected by airborne visual sensors available on the aircraft at the current moment, and acquiring airport environment images collected by ground visual sensors at the airport where the aircraft is to land at the current moment; inputting the flight environment images into an airborne image feature extraction branch to obtain a global feature map and aligning it with the field of view of the ground visual sensors based on the pose of the airborne visual sensors; inputting the airport environment images into an airport image feature extraction branch to obtain a local feature map; inputting the aligned global feature map and local feature map into a multi-branch fusion network, adaptively allocating fusion weights, and performing multi-level weighted fusion to generate a fused feature map; inputting the fused feature map into a decoding module to identify and locate the target object, and in conjunction with the real-time flight status of the aircraft, generating emergency landing visual assistance information for the crew, wherein the target object includes at least the runway and obstacles. By combining airborne and ground-based visual data, the full coverage advantage of the airport's internal visual sensors is fully utilized, effectively compensating for the limited field of view after partial failure of airborne visual sensors. It can adapt to complex airport environments such as runway obstruction and severe weather, significantly improving the accuracy of environmental judgment during emergency landing, reducing the decision-making difficulty and emergency landing risk for the crew, and ensuring flight safety. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic diagram of the structure of an aircraft emergency landing assistance system provided in an embodiment of this application; Figure 2 A flowchart illustrating an aircraft emergency landing assistance method provided in this application embodiment; Figure 3 A structural diagram of an aircraft emergency landing assistance architecture provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of an aircraft emergency landing assistance device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0019] Example 1 Figure 1 This is a schematic diagram of the structure of an aircraft emergency landing assistance system provided in an embodiment of this application. Figure 1 As shown, an embodiment of this application provides an aircraft emergency landing assistance system, including an airborne terminal and a ground terminal, wherein the airborne terminal and the ground terminal establish a stable communication connection.

[0020] Specifically, the airborne terminal includes at least an airborne vision sensor module, an airborne image processing module, and a neural network fusion processing module.

[0021] The ground-based system includes at least an airport internal visual sensor module and a ground image processing module.

[0022] The airborne vision sensor module can include multiple sets of vision sensors distributed throughout the aircraft's fuselage, such as the wings, nose, fuselage shell, and tail, to collect information about the external environment of the aircraft.

[0023] The airborne image processing module receives image information from the airborne vision sensor module and performs standardized preprocessing (including noise reduction, image stitching, and sharpness enhancement) on images acquired by the aircraft's external sensors. This module has a built-in long-range airborne image feature extraction branch that extracts features from these images, ultimately extracting preliminary global feature information such as runway boundaries and obstacle distribution.

[0024] The airport's internal visual sensor module consists of multiple sets of visual sensors distributed in various areas of the airport, including both sides of the runway, around the control tower, and around the boarding gates, to collect information about the airport's internal environment.

[0025] The ground image processing module receives image information from the airport's internal visual sensor modules, performs standardized preprocessing on the images acquired by the sensors, and integrates them to form internal airport visual information. This module includes a built-in airport image feature extraction branch, which extracts key local feature details such as runway center coordinates and obstacle dimensions, and transmits this data to the airborne terminal to support visual information fusion.

[0026] The neural network fusion processing module is deployed on the airborne end, receiving feature information from both the airborne image processing module and the ground image processing module. By setting up a multi-scale, multi-branch fusion network with the airborne visual sensor as the primary viewpoint and the airport ground sensor as the secondary viewpoint, the two types of visual data are fused to reconstruct the complete information of the actual runway location, runway boundary contour, and surrounding environment at the time of the aircraft's emergency landing.

[0027] The multi-branch fusion network here is based on the Mamba module and includes feature construction unit, context vector generation unit, weight dynamic generation unit, and weighted fusion unit.

[0028] Furthermore, this application provides an emergency landing assistance method for an aircraft, applicable to an aircraft emergency landing assistance system, the method specifically including: S1. Respond to the emergency landing assistance request, acquire the flight environment images collected by the onboard visual sensors available on the aircraft at the current moment, and acquire the airport environment images collected by the ground visual sensors at the airport where the aircraft is to land at the current moment.

[0029] The emergency landing assistance request can be generated by the pilot when the aircraft is subjected to extreme weather, mechanical wear, electromagnetic interference, or other conditions, or it can be generated automatically by the system monitoring the effectiveness of the onboard sensors.

[0030] At this point, some of the aircraft's onboard sensors malfunctioned, necessitating an emergency landing.

[0031] S2. Input the flight environment image into the airborne image feature extraction branch, obtain the global feature map, and align it with the field of view of the ground vision sensor based on the pose of the airborne vision sensor.

[0032] Figure 3 This is a structural diagram of an aircraft emergency landing assistance architecture provided in an embodiment of this application. Figure 3 As shown, in step S2, the airborne image feature extraction branch includes a local feature coding unit, a spatial location embedding unit, a serialization and neighborhood enhancement unit, and a long-distance dependency modeling unit, wherein, The local feature encoding unit receives the flight environment image, performs 2D convolution plus silicon linear unit operations, and outputs an initial local feature map. The spatial location embedding unit receives the initial local feature map, adds row and column coordinate information to it, and outputs a feature map carrying spatial location information. The serialization and neighborhood enhancement unit receives the feature map carrying spatial location information, rearranges it into a one-dimensional sequence, and performs a one-dimensional convolution operation on the sequence, outputting an enhanced sequence feature. The long-distance dependency modeling unit receives the enhanced sequence feature, performs state space scanning modeling along the row and column directions respectively, and adds the two scan results to output a global feature map representing the macroscopic layout of the runway and the distribution of long-distance obstacles.

[0033] In one specific embodiment, the local feature encoding unit may include Conv2D and SiLU to complete the initial local feature extraction and channel expansion, thereby reducing the computational complexity of subsequent operations.

[0034] The spatial location embedding unit assigns row and column coordinate information to the feature map through 2D-Place-Compute (2D position encoding), which can avoid the problem of loss of two-dimensional spatial topology after Rearrange and provide spatial context for subsequent global scanning of Mamba.

[0035] The serialization and neighborhood enhancement unit can convert 2D feature maps into a sequence format through Rearrange and feed it into Depthwise Conv1D for local neighborhood feature enhancement, thereby strengthening the weak features of distant targets.

[0036] The long-distance dependency modeling unit uses the 2D Cross-Scan Mamba module to perform bidirectional row and column scanning of the sequence, capturing long-distance target dependencies across regions and diagonally in the airport scene, and outputting a global feature sequence.

[0037] The airborne image feature extraction branch also includes a ROI dynamic alignment unit, which can align the global feature map in the following ways: For each ground-based vision sensor, determine the first coordinates of the four corner points of its field of view in the 3D world coordinate system corresponding to the airport. For each airborne vision sensor, determine the extrinsic parameter matrix based on its current pose to transform the first coordinates of the four corner points of the ground-based vision sensor, obtaining the second coordinates of the four corner points in the camera coordinate system. For each airborne vision sensor, transform the second coordinates based on its intrinsic parameter matrix to obtain the third coordinates of the four corner points in the pixel coordinate system. Determine the dynamic region of interest (ROI) of the current frame based on the minimum bounding rectangle corresponding to the third coordinates.

[0038] The dynamic region of interest is input into the region of interest alignment module, which then extracts and interpolates it in the global feature map to obtain the aligned global feature map.

[0039] Here, the four corner points of the airport sensor images can be pre-calibrated in the world coordinate system. Then, the onboard camera in flight can be dynamically calibrated to obtain the transformation matrix between the onboard camera images and the world coordinate system. This transfers the four corner points of the airport sensor images to the onboard image coordinate system, completing ROI alignment. Specifically, the ROI-Dynamic-align subunit, combined with the real-time onboard pose, can accurately align the regions in the global features corresponding to the ground sensor's field of view, outputting the aligned global feature FMamba, achieving complete spatial coordinate and scale matching with local ground features.

[0040] S3. Input the airport environment image into the airport image feature extraction branch to obtain the local feature map.

[0041] The airport image feature extraction branch here includes a detail feature extraction unit and a feature matching unit.

[0042] The detail feature extraction unit receives airport environmental images, performs multi-layered stacked two-dimensional convolution and linear operations, and outputs a detail feature map containing runway marking edges, surface textures, and obstacle outlines. The feature matching unit receives the detail feature map, adjusts its spatial size and number of channels, and outputs a local feature map with the same spatial size and number of channels as the global feature map.

[0043] Similarly, the detail feature extraction unit can be composed of Conv2D plus SiLU multi-layer stacking to extract fine-grained features such as local edges, textures, and target contours.

[0044] Here, it is necessary to maintain the same downsampling rate as the airborne branch and output local features FConv that match the size and number of channels of FMamba, in order to prepare for subsequent fusion.

[0045] S4. Input the aligned global feature map and local feature map into the multi-branch fusion network, adaptively allocate fusion weights, and perform multi-level weighted fusion to generate a fused feature map.

[0046] The multi-branch fusion network includes a feature construction unit, a context vector generation unit, a weight dynamic generation unit, and a weighted fusion unit.

[0047] The feature construction unit receives the aligned global feature map and local feature map, concatenates them according to the channel direction, and outputs the first type of feature, the second type of feature and the third type of feature respectively after convolution operation; wherein, the first type of feature is the aligned global feature map, the second type of feature is the local feature map, and the third type of feature is the purified cross-modal fusion feature map.

[0048] The context vector generation unit receives the first type of features and the second type of features, performs global average pooling on each, obtains two statistical vectors respectively, concatenates them according to the channel direction, and outputs a joint scene context vector after convolution.

[0049] The dynamic weight generation unit receives the joint scene context vector, inputs it into the fully connected layer for linear mapping, and processes it through a normalized exponential function to output three dynamically fused weights. Specifically, the context vector generation unit performs the following steps: The system receives two types of features. Global average pooling is performed on the first type of features to obtain a first statistical vector that retains only channel statistics. Global average pooling is also performed on the second type of features to obtain a second statistical vector that retains only channel statistics. The first and second statistical vectors are concatenated along the channel direction to obtain a concatenated vector. A convolution operation is performed on the concatenated vector to fuse channel-dimensional information and suppress noise, outputting a joint scene context vector.

[0050] The weighted fusion unit receives three types of features (first, second, and third) and three dynamic fusion weights. It then performs a weighted sum based on these weights to output a fused feature map. Specifically, the weighted fusion unit can perform the following operations: The system receives three types of features: a first type, a second type, and a third type, along with three dynamic fusion weights. It then expands these weights to the same spatial dimension as the first, second, and third types of features, respectively. Based on the expanded first dynamic fusion weight, it performs pixel-by-pixel weighting on the first type of feature. Based on the expanded second dynamic fusion weight, it performs pixel-by-pixel weighting on the second type of feature. Finally, it performs pixel-by-pixel weighting on the third type of feature. The three pixel-by-pixel weighting results are then summed element-by-element to output the fused feature map.

[0051] Understandably, a multi-branch fusion network can have two weighted fusion nodes. The first node is a two-branch fusion, which improves scene adaptability by assigning semantic weights to the original global and local feature maps. The second node is a three-branch fusion, which further dynamically fuses the original global and local feature maps and the fused feature map output from the first node, thereby enhancing expressive power.

[0052] S5. Input the fused feature map into the decoding module to identify and locate the target object. Combine the real-time flight status of the aircraft to generate visual assistance information for emergency landing for the crew. The target object includes at least the runway and obstacles.

[0053] In a specific embodiment, the detailed implementation process of step S5 may include: A decoder structure based on U-Net or FPN (Feature Pyramid Network) is adopted to gradually restore spatial resolution through upsampling and skip connections, while preserving deep semantic features.

[0054] Upsampling can be achieved through transposed convolution or bilinear interpolation. Skip connections can fuse high-resolution features from the corresponding layer in the encoding stage with those in the decoding layer, improving the localization accuracy of small targets (such as runway markers and obstacles).

[0055] The decoder branches out into two parallel outputs: a semantic segmentation head and an object detection head. The semantic segmentation head outputs pixel-level masks for the runway and obstacles. The object detection head outputs bounding boxes for obstacles, making it better suited for large obstacles (such as vehicles and buildings).

[0056] Based on the decoded high-resolution feature map, the texture and edge features (such as straight lines and rectangular outlines) of the runway can be extracted through pre-trained models (such as ResNet, EfficientNet, etc.), thereby obtaining the boundary line of the runway. Other forms can also be used, which are not limited here.

[0057] For obstacle detection, optical flow or temporal prediction models can be used to determine the motion state and distinguish between static obstacles (such as light towers and trees) and dynamic obstacles (such as other aircraft and vehicles). At the same time, a threshold (such as IoU>0.5) can be set to filter out low-confidence detection results.

[0058] Next, the aircraft's real-time pose (latitude and longitude, altitude, pitch / roll angle, etc. provided by GPS and IMU) and motion status (airspeed, rate of descent, wind direction, etc.) can be acquired, as well as the camera's intrinsic parameters (focal length, distortion) and extrinsic parameters (mounting angle) can be calibrated to map the image coordinates (2D) of the identified target to the 3D world coordinate system through perspective transformation.

[0059] It can also dynamically adjust the region of interest (ROI) based on the aircraft's altitude and viewing angle to eliminate redundant information outside the field of view.

[0060] Finally, augmented reality (AR) overlay technology can be used to highlight runway boundaries in real-time video on the aircraft's control interface, overlaying virtual centerlines (such as green arrows) to indicate the optimal approach path. Obstacles can be marked with red semi-transparent boxes and distance labels, such as "Obstacle: 200 meters to the left front".

[0061] It can also calculate a forced landing feasibility score based on runway length, obstacle distribution, and aircraft performance (such as braking distance), and display it using color coding (red / yellow / green). It can also trigger critical alerts, such as "obstacle too close on the right."

[0062] Through the above steps, the system can provide the crew with intuitive and real-time emergency landing decision support.

[0063] The emergency landing assistance method for aircraft provided in this application fully leverages the full coverage advantage of airport internal visual sensors through the collaboration of airborne and ground visual data. It effectively compensates for the limited field of view after partial failure of airborne visual sensors, and can adapt to complex airport environments such as runway obstruction and severe weather. It significantly improves the accuracy of environmental judgment during emergency landing, reduces the decision-making difficulty and emergency landing risk for the crew, and ensures flight safety.

[0064] Example 2 In one embodiment of this application, a coordinate transformation step is provided.

[0065] First, for each airport, a three-dimensional world coordinate system can be established in the following way: You can use the airport's fixed landmarks as a reference, and select permanent fixed objects such as airport runway markings, terminal corners, and fixed light poles as reference points.

[0066] Record the position of each ground sensor and the three-dimensional coordinates of its field of view corner in the world coordinate system. Specifically, the installation position of the ground sensor is denoted as... The four corner points of the field of view (near and far boundaries) are denoted as follows: .

[0067] Determine the intrinsic parameter matrix for each airport camera, the intrinsic parameter matrix including at least the focal length. Principal point coordinates .

[0068] In one specific embodiment, the system can perform the following steps in real time: Step 1: Obtain the real-time pose (external parameters) of the airborne camera.

[0069] Each frame of aerial image requires the real-time pose of the onboard camera, i.e., the extrinsic parameter matrix. ,Include: Translation vector , used to indicate the position of the airborne camera in the world coordinate system; Rotation matrix It can directly obtain real-time latitude, longitude, altitude, and attitude angles from the flight controller and GPS / IMU module, and convert them into pose in the world coordinate system.

[0070] Step 2: Transform the world coordinate system to the camera coordinate system.

[0071] Using an extrinsic parameter matrix, the world coordinates of the ground sensor's field of view corner points are transformed to the camera coordinate system.

[0072] Step 3: Transform the camera coordinate system to the image pixel coordinate system.

[0073] Using intrinsic parameter matrix Projecting camera coordinates into image pixel coordinates: This includes the pixel coordinates of corner points on the airborne image as follows: Depth value in camera coordinate system .

[0074] Step 4: Generate a dynamic ROI rectangle.

[0075] The minimum bounding rectangle is calculated from the pixel coordinates obtained from the four projections and used as the ROI for the current frame. Finally, the dynamically updated ROI coordinates (x1, y1, x2, y2) are obtained.

[0076] Step 5: ROI-Align feature alignment.

[0077] The calculated dynamic ROI is fed into the ROI-Align unit, and the region corresponding to the field of view of the ground sensor is cut out from the airborne global feature map to achieve pixel-level spatial alignment.

[0078] Example 3 In one embodiment of this application, a method for dynamically adjusting the weights of three branches is provided. This dynamic weight adjustment is achieved through the synergistic effect of a Linear layer and a Softmax function. The core principle is to allow the network to adaptively allocate the weights (Gs1, Gs2, Gs3) of the three branches based on the real-time status of the input features, ensuring optimal complementarity between the global context and local details. The entire process is detailed below: 1. Input basis for weight adjustment (global context vector) The dynamic adjustment of weights is guided by global context information, ensuring that the weight allocation closely matches the actual feature distribution of the current airport scene. The input to the Linear layer is not the original feature map, but rather a scene-level statistical vector after global average pooling. The specific preparation process is as follows: Global average pooling (AvgPool2d) is performed on the airborne global features FMamba to compress the 2D feature map of [B,C,H,W] into a vector of [B,C,1,1] (denoted as Vm). This vector condenses the overall statistical information of the airborne global features (such as the layout of a large-scale airport scene and the distribution of distant targets). The same global average pooling is applied to the local ground feature FConv to obtain a local global vector [B,C,1,1] (denoted as Vc), which condenses the overall statistical information of the local ground features (such as local target texture and edge density). Channel concatenation (or fusion) of Vm and Vc yields a joint global context vector (denoted as V) of [B, 2C, 1, 1]. This vector contains both global and local scene information, providing a basis for subsequent weight adjustments and preventing weight allocation from deviating from the actual scene. Here, concatenation followed by 1×1 convolutional purification is used instead of direct addition, allowing the network to automatically learn the fused weights of Vm and Vc, ensuring that the global context vector accurately reflects the features of the current frame.

[0079] The role of global average pooling is to filter out local noise, extract scene-level global information, and prevent the Linear layer from being disturbed by local details, which could lead to an imbalance in weight distribution (such as mistakenly assigning high weights to local features of the background region).

[0080] 2. Use a Linear layer to map global information to weight logits. The Linear layer (fully connected layer) is the core mapping module for dynamic weight adjustment. Its function is to map the high-dimensional global context vector V to low-dimensional logits (unnormalized weights) corresponding to the number of branches. The specific implementation is as follows: Define the input dimension of the Linear layer as 2C (consistent with the dimension of the squeezed vector) and the output dimension as 3 (corresponding to the three weights Gs1, Gs2, Gs3 of the three branches), that is, the parameters of the Linear layer are [3,2C] (output dimension × input dimension). First, the joint global context vector V is squeezed to convert it from [B,2C,1,1] into a two-dimensional vector of [B,2C] to adapt to the input format of the Linear layer; The squeezed vector is fed into a Linear layer, and the logits matrix is ​​calculated through a linear transformation. The logits matrix has a shape of [B,3], where each row corresponds to the three unnormalized weights of a sample (corresponding to the FMamba, FConv, and FConcat branches, respectively). The formula for calculating the logits matrix can be expressed as: logits = V × W + b; Where W is the weight matrix of the Linear layer ([2C,3]), b is the bias term ([1,3]), V is the squeezed global context vector ([B,2C]), and logits is the output unnormalized weights ([B,3]).

[0081] The parameters (W and b) of the Linear layer are updated adaptively as the network is trained. The core learning objective is to dynamically adjust the output of logits based on the changes in the global context vector V, so that the weight allocation can match the feature complementarity requirements of the current frame (e.g., increase the weight of FMamba for distant target frames; increase the weight of FConv for close-range clear target frames).

[0082] 3. The Softmax layer implements weight normalization constraints. The logits output by the Linear layer are unnormalized values ​​(can be positive or negative, with no range constraints), and cannot be used directly as weights. The core function of the Softmax layer is to normalize the logits, as implemented below: Perform a Softmax operation on each row of the logits matrix (three logits for each sample) to calculate the proportion of each logits to the sum of all logits in that row; The mathematical expression for the Softmax operation is as follows: For each sample's three logits(z1, z2, z3), the corresponding weights (Gs1, Gs2, Gs3) are calculated as follows: ; in, This demonstrates element-wise exponentiation.

[0083] After normalization, the three weights of each sample satisfy 0≤Gs≤1, and Gs1+Gs2+Gs3=1, ensuring the rationality of weight allocation.

[0084] In particular, the exponential operation of Softmax amplifies the differences between logits, enabling the network to more clearly highlight the more important branches in the current frame (e.g., if the logits of a certain branch are significantly higher than those of other branches, its normalized weight will be close to 1, while the weights of other branches will be close to 0), thus improving the targeting of weight adjustments.

[0085] 4. Dynamically adjust the fusion ratio of the three-branch features. The three weights (Gs1, Gs2, Gs3) obtained through Linear+Softmax will directly apply to the features of the three branches to achieve dynamic weighted fusion. The specific application process is as follows: The normalized weights ([B,3]) are expanded to [B,3,1,1] to fit the dimension of the feature map [B,C,H,W] (to facilitate pixel-by-pixel weighting). Features are extracted from three branches: FMamba (global features), FConv (local features), and FConcat (purified and fused features), with each branch having dimensions of [B,C,H,W]. The final fusion feature is obtained by weighted summation according to the weights, which can be expressed as: F_fused=Gs1×FMamba+Gs2×FConv+Gs3×FConcat; During network training, the parameters (W and b) of the Linear layer can be updated through backpropagation, so that the weight allocation can minimize the prediction error of the fused features and gradually realize the dynamic adjustment of weights to adapt to the scene.

[0086] The dynamic weight adjustment process provided in this application employs global context guidance, Linear mapping, and Softmax normalization. The Linear layer is responsible for learning the mapping relationship from global information to weights, while the Softmax layer is responsible for constraining the rationality of the weights. Together, they achieve dynamic adaptive weighting. Compared with fixed weight fusion in existing technologies, this method can automatically adjust the weight ratio of the three branches based on changes in airborne pose and differences in airport scenarios (such as long-distance / short-distance targets, and changes in illumination), avoiding conflicts between global and local features.

[0087] Example 4 Figure 4This is a schematic diagram of the structure of an aircraft emergency landing assistance device provided in an embodiment of this application. Figure 4 As shown, an embodiment of this application provides an aircraft emergency landing assistance device 40, the device comprising: The response module 410 is used to respond to a forced landing assistance request, acquire flight environment images collected by the onboard visual sensors available on the aircraft at the current moment, and acquire airport environment images collected by the ground visual sensors at the airport where the aircraft is to land at the current moment. The feature extraction module 420 is used to input the flight environment image into the airborne image feature extraction branch, obtain the global feature map, and align it with the field of view of the ground vision sensor based on the pose of the airborne vision sensor. Input the airport environment image into the airport image feature extraction branch to obtain the local feature map; The fusion module 430 is used to input the aligned global feature map and local feature map into the multi-branch fusion network, adaptively allocate fusion weights, and perform multi-level weighted fusion to generate a fused feature map. The output module 440 is used to input the fused feature map into the decoding module to identify and locate the target object, and to generate emergency landing visual assistance information for the crew by combining the real-time flight status of the aircraft. The target object includes at least the runway and obstacles.

[0088] In a preferred embodiment, the global feature map is aligned in the following manner: For each ground vision sensor, determine the first coordinates of the four corner points of the field of view of that ground vision sensor in the three-dimensional world coordinate system corresponding to the airport; For each airborne vision sensor, an extrinsic parameter matrix is ​​determined based on the pose of the airborne vision sensor at the current moment, so as to transform the first coordinates of the four corner points of the ground vision sensor and obtain the second coordinates of the four corner points in the camera coordinate system. For each airborne vision sensor, based on the intrinsic parameter matrix of that airborne vision sensor, the second coordinates are transformed to obtain the third coordinates of the four corner points in the pixel coordinate system; The dynamic region of interest for the current frame is determined based on the minimum bounding rectangle corresponding to the third coordinate. The dynamic region of interest is input into the region of interest alignment module, which then extracts and interpolates it in the global feature map to obtain the aligned global feature map.

[0089] In a preferred embodiment, the airborne image feature extraction branch includes a local feature encoding unit, a spatial location embedding unit, a serialization and neighborhood enhancement unit, and a long-distance dependency modeling unit, wherein, The local feature encoding unit receives the flight environment image, performs two-dimensional convolution plus silicon linear unit operation, and outputs an initial local feature map; The spatial location embedding unit receives the initial local feature map, adds row and column coordinate information to the initial local feature map, and outputs a feature map carrying spatial location information. The serialization and neighborhood enhancement unit receives a feature map carrying spatial location information, rearranges it into a one-dimensional sequence, performs a one-dimensional convolution operation on the sequence, and outputs enhanced sequence features. The long-distance dependency modeling unit receives enhanced sequence features, performs state space scanning modeling along the row and column directions respectively, and adds the two scan results to output a global feature map representing the macro layout of the runway and the distribution of long-distance obstacles.

[0090] In a preferred embodiment, the airport image feature extraction branch includes a detail feature extraction unit and a feature matching unit, wherein, The detail feature extraction unit receives airport environmental images, performs multi-layer stacked two-dimensional convolution and linear operations, and outputs a detail feature map containing runway marking edges, surface textures and obstacle outlines; The feature matching unit receives the detailed feature map, adjusts its spatial size and number of channels, and outputs a local feature map with the same spatial size and number of channels as the global feature map.

[0091] In a preferred embodiment, the multi-branch fusion network includes a feature construction unit, a context vector generation unit, a weight dynamic generation unit, and a weighted fusion unit, wherein, The feature construction unit receives the aligned global feature map and local feature map, concatenates them according to the channel direction, and outputs the first type of feature, the second type of feature and the third type of feature respectively after convolution operation; wherein, the first type of feature is the aligned global feature map, the second type of feature is the local feature map, and the third type of feature is the purified cross-modal fusion feature map; The context vector generation unit receives the first type of features and the second type of features, performs global average pooling on them respectively, obtains two statistical vectors respectively, concatenates them according to the channel direction, and outputs a joint scene context vector after convolution. The weight dynamic generation unit receives the joint scene context vector, inputs it into the fully connected layer for linear mapping, and processes it through a normalized exponential function to output three dynamically fused weights. The weighted fusion unit receives the first type of features, the second type of features, the third type of features, and three dynamic fusion weights. It performs a weighted summation based on the three dynamic fusion weights to output a fused feature map.

[0092] In a preferred embodiment, the context vector generation unit is specifically used to perform the following steps: Receive the first type of features and the second type of features; Perform global average pooling on the first type of features to obtain a first statistical vector that retains only channel statistics; Perform global average pooling on the second type of features to obtain a second statistical vector that retains only channel statistics. The first statistical vector and the second statistical vector are concatenated along the channel direction to obtain the concatenated vector; Perform convolution operations on the concatenated vectors to fuse information and suppress noise at the channel dimension, and output a joint scene context vector.

[0093] In a preferred embodiment, the weighted fusion unit is specifically used to perform the following operations: It receives the first type of feature, the second type of feature, the third type of feature, and three dynamic fusion weights; The three dynamic fusion weights are extended to the same spatial dimension as the first type of feature, the second type of feature, and the third type of feature, respectively; The first type of features are weighted pixel by pixel based on the expanded first dynamic fusion weight; The second type of features are weighted pixel-by-pixel based on the extended second dynamic fusion weight; The third type of features are weighted pixel-by-pixel based on the extended third dynamic fusion weight; The three pixel-by-pixel weighted results are summed element-by-element to output a fused feature map.

[0094] Example 4 Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 500 includes a processor 510, a memory 520, and a bus 530.

[0095] The memory 520 stores machine-readable instructions that can be executed by the processor 510. When the electronic device 500 is running, the processor 510 and the memory 520 communicate via the bus 530. When the machine-readable instructions are executed by the processor 510, the steps of an aircraft emergency landing assistance method as described in the above method embodiment can be executed. For specific implementation details, please refer to the method embodiment, which will not be repeated here.

[0096] Example 5 This application also provides a computer-readable storage medium storing a computer program. When the computer program is run by a processor, it can execute the steps of an aircraft emergency landing assistance method as described in the above method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0097] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0098] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0099] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0100] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0101] It should be noted that if the function is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0103] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for assisting in the forced landing of an aircraft, characterized in that, The method includes: In response to a forced landing assistance request, acquire flight environment images collected by onboard visual sensors available on the aircraft at the current moment, and acquire airport environment images collected by ground visual sensors at the airport where the aircraft is to land at the current moment. The flight environment image is input into the airborne image feature extraction branch to obtain a global feature map and, based on the pose of the airborne visual sensor, it is aligned with the field of view of the ground visual sensor. The airport environment image is input into the airport image feature extraction branch to obtain a local feature map; The aligned global feature map and the local feature map are input into a multi-branch fusion network, fusion weights are adaptively allocated, and multi-level weighted fusion is performed to generate a fused feature map. The fused feature map is input into the decoding module to identify and locate the target object. Combined with the real-time flight status of the aircraft, visual assistance information for emergency landing is generated for the crew. The target object includes at least the runway and obstacles.

2. The method of claim 1, wherein, Align the global feature maps using the following method: For each ground vision sensor, determine the first coordinates of the four corner points of the field of view of the ground vision sensor in the three-dimensional world coordinate system corresponding to the airport; For each airborne vision sensor, an extrinsic parameter matrix is ​​determined based on the pose of the airborne vision sensor at the current moment, so as to transform the first coordinates of the four corner points of the ground vision sensor and obtain the second coordinates of the four corner points in the camera coordinate system. For each airborne vision sensor, based on the intrinsic parameter matrix of that airborne vision sensor, the second coordinates are transformed to obtain the third coordinates of the four corner points in the pixel coordinate system; The dynamic region of interest for the current frame is determined based on the minimum bounding rectangle corresponding to the third coordinate. The dynamic region of interest is input into the region of interest alignment module, which then extracts and interpolates it in the global feature map to obtain the aligned global feature map.

3. The method of claim 1, wherein, The airborne image feature extraction branch includes a local feature coding unit, a spatial location embedding unit, a serialization and neighborhood enhancement unit, and a long-distance dependency modeling unit. in, The local feature encoding unit receives the flight environment image, performs two-dimensional convolution plus silicon linear unit operation, and outputs an initial local feature map; The spatial location embedding unit receives the initial local feature map, adds row and column coordinate information to the initial local feature map, and outputs a feature map carrying spatial location information. The serialization and neighborhood enhancement unit receives the feature map carrying spatial location information, rearranges it into a one-dimensional sequence, performs a one-dimensional convolution operation on the sequence, and outputs enhanced sequence features. The long-distance dependency modeling unit receives the enhanced sequence features, performs state space scanning modeling along the row and column directions respectively, adds the two scan results, and outputs a global feature map representing the macro layout of the runway and the distribution of long-distance obstacles.

4. The method of claim 1, wherein, The airport image feature extraction branch includes a detail feature extraction unit and a feature matching unit. in, The detail feature extraction unit receives the airport environment image, performs multi-layer stacked two-dimensional convolution and linear operations, and outputs a detail feature map containing runway marking edges, surface textures and obstacle outlines; The feature matching unit receives the detailed feature map, adjusts its spatial size and number of channels, and outputs a local feature map with the same spatial size and number of channels as the global feature map.

5. The method of claim 1, wherein, The multi-branch fusion network includes a feature construction unit, a context vector generation unit, a weight dynamic generation unit, and a weighted fusion unit, wherein... The feature construction unit receives the aligned global feature map and the local feature map, concatenates them according to the channel direction, and outputs the first type of feature, the second type of feature and the third type of feature respectively after convolution operation; wherein, the first type of feature is the aligned global feature map, the second type of feature is the local feature map, and the third type of feature is the purified cross-modal fusion feature map; The context vector generation unit receives the first type of features and the second type of features, performs global average pooling processing on each, obtains two statistical vectors respectively, concatenates them according to the channel direction, and outputs a joint scene context vector after convolution. The weight dynamic generation unit receives the joint scene context vector, inputs it into the fully connected layer for linear mapping, and processes it through a normalized exponential function to output three dynamic fusion weights. The weighted fusion unit receives the first type of feature, the second type of feature, the third type of feature, and the three dynamic fusion weights, and performs a weighted sum based on the three dynamic fusion weights to output a fused feature map.

6. The method of claim 5, wherein, The context vector generation unit is specifically used to perform the following steps: Receive the first type of feature and the second type of feature; Perform global average pooling on the first type of features to obtain a first statistical vector that retains only channel statistics. Perform global average pooling on the second type of features to obtain a second statistical vector that retains only channel statistics; The first statistical vector and the second statistical vector are concatenated along the channel direction to obtain the concatenated vector; Perform convolution operations on the concatenated vector to fuse information and suppress noise at the channel dimension, and output a joint scene context vector.

7. The method of claim 5, wherein, The weighted fusion unit is specifically used to perform the following operations: Receive the first type of feature, the second type of feature, the third type of feature, and the three dynamic fusion weights; The three dynamic fusion weights are respectively extended to the same spatial dimension as the first type of feature, the second type of feature, and the third type of feature; The first type of features are weighted pixel by pixel based on the expanded first dynamic fusion weight; The second type of features are weighted pixel-by-pixel based on the extended second dynamic fusion weight; The third type of features are weighted pixel-by-pixel based on the extended third dynamic fusion weight; The three pixel-by-pixel weighted results are summed element-by-element to output a fused feature map.

8. A forced landing assisting device for an aircraft, characterised in that, The device includes: The response module is used to respond to emergency landing assistance requests, obtain flight environment images collected by the onboard visual sensors available on the aircraft at the current moment, and obtain airport environment images collected by the ground visual sensors at the airport where the aircraft is to land at the current moment. The feature extraction module is used to input the flight environment image into the airborne image feature extraction branch, obtain a global feature map, and align it with the field of view of the ground vision sensor based on the pose of the airborne vision sensor. The airport environment image is input into the airport image feature extraction branch to obtain a local feature map; The fusion module is used to input the aligned global feature map and the local feature map into a multi-branch fusion network, adaptively allocate fusion weights, and perform multi-level weighted fusion to generate a fused feature map. The output module is used to input the fused feature map into the decoding module to identify and locate the target object, so as to generate emergency landing visual assistance information for the crew to use in combination with the real-time flight status of the aircraft. The target object includes at least the runway and obstacles.

9. An electronic device, comprising: include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the emergency landing assistance method for an aircraft as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the emergency landing assistance method for an aircraft as described in any one of claims 1 to 7.

Images