Camera repositioning method for approaching landing of fixed-wing aircraft

Abstract

The invention discloses a camera repositioning method for approaching landing of a fixed-wing aircraft, and relates to the technical field of data processing, and the method comprises the steps: collecting an approaching landing image of the fixed-wing aircraft through an airborne foresight camera, and obtaining the pose information of the aircraft; converting the position and attitude information of the fixed-wing aircraft into first camera pose information through a coordinate conversion module; on this basis, constructing a camera relocation network, inputting the image and the corresponding camera pose information into the network for feature extraction, and outputting a fusion feature map; and performing camera relocation according to the fused feature map, and finally outputting camera pose information, namely the relocated camera position and pose. The invention solves the technical problem of inaccurate positioning caused by signal loss in the approaching landing stage of the existing navigation method, and achieves the technical effect of providing accurate aircraft positioning when the signal is limited through the airborne foresight camera and the repositioning network.

Description

Technical Field

[0001] The invention relates to the technical field of data processing, and in particular to a camera repositioning method for approach and landing of a fixed-wing aircraft. Background Art

[0002] Fixed-wing aircraft usually rely on satellite navigation systems (GPS) and instrument landing systems (ILS) for positioning during the approach and landing phase. However, these traditional navigation methods have obvious defects in complex environments. For example, satellite signals are easily lost or reduced in accuracy due to bad weather, terrain obstruction, or interference from tall buildings around the airport, resulting in inaccurate estimation of the aircraft's position and attitude. In addition, the instrument landing system relies on ground signals and may not be able to provide stable positioning information due to equipment failure or environmental factors. Summary of the invention

[0003] The present application provides a camera repositioning method for fixed-wing aircraft approach and landing, which is used to solve the technical problem that the existing navigation method may cause inaccurate positioning due to signal loss during the approach and landing phase.

[0004] The present application provides a camera relocalization method for fixed-wing aircraft approach and landing, the method comprising: using an onboard forward-looking camera to collect images of the fixed-wing aircraft approach and landing; obtaining aircraft posture information of the fixed-wing aircraft, the aircraft posture information comprising position information and attitude information of the aircraft; converting the aircraft posture information through a coordinate conversion module; converting the position and attitude information of the fixed-wing aircraft through a coordinate conversion module to obtain camera posture information, the camera posture information comprising position information and attitude information of the forward-looking camera in a world coordinate system; constructing a camera relocalization network, inputting the image and the corresponding camera posture information into the camera relocalization network for feature extraction, outputting a fused feature map, performing camera relocalization according to the fused feature map, and outputting camera posture information, wherein the camera posture information is the relocated camera posture information.

[0005] One or more technical solutions provided in this application have at least the following technical effects or advantages: The camera repositioning method for fixed-wing aircraft approach and landing provided by the present application relates to the field of data processing technology. Approach and landing images are collected by an onboard forward-looking camera, the position and attitude information of the fixed-wing aircraft is obtained, and the camera posture information is obtained by a coordinate conversion module. A camera repositioning network is constructed, and image and posture information are input for feature extraction, a fused feature map is output, and repositioning is performed according to the feature map. Finally, the repositioned camera posture information is output, which solves the technical problem that inaccurate positioning may be caused by signal loss in the approach and landing stage of the existing navigation method, and realizes the technical effect of providing accurate aircraft positioning when the signal is limited by using the onboard forward-looking camera and the repositioning network. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0007] Figure 1 A schematic flow chart of a camera repositioning method for fixed-wing aircraft approach and landing provided in an embodiment of the present application; Figure 2 A schematic diagram of a process for generating second camera pose information in a camera repositioning method for a fixed-wing aircraft approach and landing provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a global interaction module of a feature interaction module in a camera repositioning method for fixed-wing aircraft approach and landing provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of a local interaction module of a feature interaction module in a camera repositioning method for a fixed-wing aircraft approach and landing provided in an embodiment of the present application; Figure 5 A schematic diagram of a coordinate system in a camera repositioning method for a fixed-wing aircraft approach and landing provided in an embodiment of the present application; Figure 6 A structural diagram of a proximity relocation network LandNet in a camera relocation method for fixed-wing aircraft approach and landing provided in an embodiment of the present application; Figure 7 A schematic diagram of a bottleneck block in a CNN branch in a camera relocalization method for fixed-wing aircraft approach and landing provided in an embodiment of the present application. DETAILED DESCRIPTION

[0008] The present application provides a camera repositioning method for fixed-wing aircraft approach and landing, which is used to solve the technical problem that the existing navigation method may cause inaccurate positioning due to signal loss during the approach and landing phase.

[0009] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0010] It should be noted that the terms "first", "second", etc. in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices.

[0011] Embodiment 1, as Figure 1 As shown, the present application provides a camera repositioning method for a fixed-wing aircraft approach and landing, the method comprising: P10: Using the onboard forward-looking camera to collect images of fixed-wing aircraft approaching and landing.

[0012] Specifically, the airborne forward-looking camera is first used to collect images during the approach and landing process of the fixed-wing aircraft. The airborne forward-looking camera is a sensor installed in front of the aircraft, which is mainly used to capture visual information of the environment in front of the aircraft. These environmental image data are crucial for the navigation of the aircraft during the approach and landing phase, because they can provide real-time environmental information to the pilot or the automatic flight system to help with tasks such as path planning, obstacle detection and landing point confirmation.

[0013] The approach and landing phase is the most risky part of the flight process. The onboard forward-looking camera captures the area that the aircraft passes through during the flight, and the real-time image information captured reflects key factors such as the runway, ground facilities, and weather conditions. By analyzing these images, the system can update the aircraft's position and attitude information in real time, so that the aircraft can land safely according to the predetermined route.

[0014] In order to ensure that the collected images have sufficient quality and accuracy, the onboard forward-looking cameras are usually equipped with high-resolution imaging sensors, such as CMOS or CCD sensors, which can provide clear images under different lighting conditions. In addition, the forward-looking cameras usually use optical zoom or high dynamic range (HDR) capabilities to cope with complex scenes that may be encountered during flight, such as strong lighting contrast or night flight.

[0015] P20: Acquire the aircraft posture information of the fixed-wing aircraft, wherein the aircraft posture information includes the position information and attitude information of the aircraft.

[0016] Optionally, obtain the position and attitude information of the fixed-wing aircraft. The position and attitude information consists of two key parts: the position information and attitude information of the aircraft. The position information is usually expressed in three-dimensional coordinates (X, Y, Z), where X, Y, and Z correspond to the position information of the aircraft in the world coordinate system. The attitude information describes the direction and posture of the aircraft relative to itself, usually expressed in three angles (roll angle, pitch angle, yaw angle), which are used to define the spatial orientation of the aircraft relative to the ground and the flight path.

[0017] Exemplarily, the acquisition of aircraft posture information can rely on multiple sensors and navigation systems. The most commonly used systems include the Global Positioning System (GPS), Inertial Measurement Unit (IMU) and barometric altimeter. The GPS system can provide the real-time position of the aircraft around the world, and obtain the longitude, latitude and altitude of the aircraft through the positioning of satellite signals. The IMU measures the acceleration and angular velocity of the aircraft through accelerometers and gyroscopes, thereby calculating the attitude changes of the aircraft, especially the roll angle, pitch angle and yaw angle. The barometric altimeter estimates the altitude information of the aircraft based on the changes in atmospheric pressure. It is usually used in conjunction with GPS to provide a more accurate flight altitude.

[0018] The aircraft posture information is the basis for coordinate transformation and camera repositioning in subsequent steps. It can provide the current position and attitude of the aircraft, and combined with the image data collected by the onboard forward-looking camera, through feature extraction and fusion, it can further help the repositioning system accurately estimate the attitude and position of the aircraft, thereby improving the safety and accuracy of landing.

[0019] P30: Figure 5 As shown, the aircraft posture information is converted by a coordinate conversion module to obtain first camera posture information, and the first camera posture information includes the position information and posture information of the forward-looking camera.

[0020] Furthermore, step P30 of the embodiment of the present application further includes: P31: Define multiple coordinate systems, including an Earth-centered Earth-fixed coordinate system, a navigation coordinate system, a carrier coordinate system, and a camera coordinate system; P32: Perform a unified coordinate transformation on the multiple coordinate systems, output a coordinate transformation matrix, analyze the aircraft posture information according to the coordinate transformation matrix, and obtain the first camera posture information.

[0021] The expression for performing unified coordinate transformation on the multiple coordinate systems is: ;in, To convert the geographic location into three-dimensional coordinates in the camera coordinate system, Including converting the longitude, latitude and altitude in the WGS84 coordinate system into the three-dimensional coordinates of the Earth-centered Earth-fixed coordinate system ECEF, and converting the coordinates of the Earth-centered Earth-fixed coordinate system ECEF into the world coordinate system ; Including the navigation coordinate system To the carrier coordinate system The conversion and definition of the carrier coordinate system based on the camera installation relationship To the camera coordinate system conversion.

[0022] It should be understood that the aircraft posture needs to undergo a series of coordinate transformations through the coordinate transformation module to generate the camera posture information. Specifically, this process can be divided into two main stages: coordinate system definition and transformation and coordinate unified transformation.

[0023] First, multiple coordinate systems need to be defined to ensure that all spatial data can be processed uniformly. The multiple coordinate systems include: Earth-centered Earth-fixed coordinate system (ECEF), which is a global coordinate system with the center of mass of the earth as the origin, suitable for data processing in geographic information systems such as the Global Positioning System (GPS). Navigation coordinate system (NED), a coordinate system corresponding to the local geographic coordinate system of the current position of the aircraft, especially providing relative position information of the aircraft during flight. Body coordinate system (Body), that is, the coordinate system of the aircraft itself, is usually aligned with the physical structure of the aircraft, such as the "front-right-up" direction of the fuselage. Camera coordinate system (Camera), a coordinate system related to camera hardware, used to represent the camera's viewing angle and the direction of capturing images.

[0024] Next, these coordinate systems are uniformly transformed. The goal of coordinate transformation is to transform the position and attitude information of the fixed-wing aircraft into the camera coordinate system. The transformation process can rely on the relationship between multiple coordinate systems, expressed as a series of transformation matrices. Each transformation matrix represents the relative position and attitude relationship between different coordinate systems, ensuring that the aircraft posture information is transformed from the global coordinate system to the camera coordinate system.

[0025] For example, the geodetic coordinate system WGS84 is converted to the Earth-centered Earth-fixed coordinate system ECEF. A point on the runway plane, with longitude, latitude and altitude, is projected from the Earth's ellipsoid flattening to the next point in the Earth-centered Earth-fixed coordinate system. , the conversion relationship is as follows: ; in, Represents the coordinate point in the Earth-centered Earth-fixed coordinate system, the converted three-dimensional coordinates, is the normal radius of the earth (i.e. the radius of the earth's surface at a certain latitude), and its value changes with latitude. Altitude is the height of a point on the ground. Lon is longitude and Lat is latitude. is the square of the Earth's eccentricity e, is the radius of the ellipsoid at a point on the ground, and T is the transpose.

[0026] Convert the Earth-centered Earth-fixed coordinate system ECEF to the world coordinate system For a point in the Earth-centered Earth-fixed coordinate system , convert it to the ENU coordinate system with runway point A as the origin, and use this coordinate system as the world coordinate system. The conversion relationship is as follows: ; Navigation coordinate system Convert to carrier coordinate system The carrier coordinate system can be obtained by rotating the navigation coordinate system three times around the Z axis, Y axis and X axis. , the conversion relationship is as follows: ; Carrier coordinate system Convert to camera coordinate system The aircraft body and the camera are rigidly connected, and the transformation model between the two coordinate systems includes the rotation matrix and translation matrix , the coordinate transformation relationship from the carrier coordinate system to the camera coordinate system is: ; Through the above transformation, the position and attitude of the camera in the world coordinate system can be obtained. The image data captured by the forward-looking camera can be further used to complete the camera's posture regression, thereby providing accurate visual data support for the aircraft's autonomous navigation and approach and landing.

[0027] P40: Figure 6 As shown, a camera relocalization network is constructed, and the camera relocalization network includes an initial feature extraction module, a branch feature extraction module, a feature interaction module and a pose regressor; the image and the camera pose information are input into the camera relocalization network for feature extraction, a fused feature map is output, the camera is relocalized according to the fused feature map, and second camera pose information is output, wherein the camera pose information is the relocated camera pose information.

[0028] Further, such as Figure 2 As shown, step P40 of the embodiment of the present application also includes: P41: According to the initial feature extraction module, an initial feature map is obtained; P43: The initial feature map is input into the branch feature extraction module for enhanced feature extraction to obtain a local feature map and a global feature map; P44: The local feature map and the global feature map are input into the feature interaction module for feature interaction to output the fused feature map; P45: The pose regressor performs regression analysis on the fused feature map to output the pose information of the second camera, including the position information and pose information after repositioning.

[0029] It should be understood that the camera pose prediction is performed by constructing a camera relocalization network. The process mainly includes inputting an image and first camera pose information, extracting and fusing features through a series of network modules, and finally outputting second camera pose information, i.e., the relocalized camera pose information.

[0030] First, a camera relocalization network is constructed. The network consists of multiple modules, including an initial feature extraction module, a branch feature extraction module, a feature interaction module, and a pose regressor. The role of these modules is to extract, interact, and fuse features from the input image and camera pose information in order to more accurately predict the camera pose.

[0031] Specifically, the main task of the initial feature extraction module is to perform preliminary processing on the input image and extract basic image features from it. Usually, the initial feature extraction module uses two layers Convolution obtains the initial feature map. These initially extracted features provide the basis for subsequent feature extraction and pose estimation.

[0032] The branch feature extraction module is used to perform deeper feature processing and enhancement on the initial feature map. This module consists of two branches, which are responsible for extracting local features and global features in the image. Local features usually focus on the details in the image. The extraction of local and global features helps to capture information of different scales, thereby improving the accuracy of camera relocalization.

[0033] Then, the local feature map and global feature map after branch feature extraction are sent to the feature interaction module. The function of this module is to fuse local features with global features. Through feature interaction, the system can better understand the overall situation and details of the image, thereby enhancing the accuracy of pose prediction.

[0034] Finally, the fused feature map is fed into the pose regressor for regression analysis. The pose regressor is a multi-layer perceptron (MLP) or a fully connected network whose task is to output the pose information of the camera based on the fused feature map. Specifically, the pose regressor predicts the camera's position information (including coordinates in the X, Y, and Z directions) and attitude information (including roll, pitch, and yaw angles). Through this regression process, the spatial information in the image is converted into an accurate camera pose, thereby achieving camera relocation.

[0035] Throughout the process, the accuracy of the feature extraction module and the pose regressor is crucial because they directly affect the accuracy of camera relocalization. The feature extraction method based on deep learning can automatically learn useful spatial features from the data, which not only reduces the workload of manually designing features, but also can handle complex environmental changes such as lighting changes and perspective changes.

[0036] Furthermore, step P43 of the embodiment of the present application further includes: P43-1: Input the initial feature map into the branch feature extraction module for enhanced feature extraction to obtain a local feature map and a global feature map, wherein the branch feature extraction module includes a CNN branch and a Transformer branch; P43-2: wherein, Figure 7 As shown, the CNN branch includes four CNN convolutional structural blocks, and the convolution size of each convolutional structural block is different. The Transformer branch includes four Transformer structural blocks, and the segmentation granularity of each Transformer structural block is different. P43-3: According to the CNN branch, the initial feature map is subjected to enhanced feature extraction, and a local feature map is output; P43-4: According to the Transformer branch, the initial feature map is subjected to enhanced feature extraction, and a global feature map is output.

[0037] In a possible embodiment of the present application, enhanced feature extraction is performed on the initial feature map through a branch feature extraction module to obtain a local feature map and a global feature map.

[0038] First, the initial feature map is further processed by inputting it into the branch feature extraction module. The branch feature extraction module consists of a CNN branch and a Transformer branch. The CNN branch is mainly used to extract local features of the image, while the Transformer branch is used to extract global features of the image. The purpose of this is to capture different levels of feature information from the image through two different network structures, thereby improving the ability to understand complex scenes in the image.

[0039] Specifically, the CNN branch includes four CNN convolutional blocks, each of which has a different convolution kernel size, designed to capture local features of different scales in the image. The change in the size of the convolution kernel enables each convolutional block to extract features at different scales, thereby enhancing the ability to recognize local details. For example, a small convolution kernel can capture small features in the image (such as edges, corners, etc.), while a large convolution kernel helps capture a larger range of local structures (such as textures, object shapes, etc.). The detailed structure of the CNN branch is shown in Table 1: Table 1 Detailed structure of CNN branches On the other hand, the Transformer branch consists of four Transformer blocks, each with a different granularity. The granularity of the Transformer structure usually refers to the input image being divided into multiple non-overlapping image blocks, each of which is processed by the Transformer's multi-head attention mechanism. Different granularities enable the Transformer to capture the global features of the image at different scales and levels of detail. Especially when dealing with long-distance or complex image content, the Transformer architecture exhibits stronger global dependency modeling capabilities than traditional CNNs.

[0040] Exemplarily, according to the CNN branch, the initial feature map will be processed by multiple layers of convolution to extract local features. Through the convolution operation, the network can automatically extract key information from the image, such as the edge, shape, texture, etc. of the object. The convolution layer scans the image through a sliding window and extracts the features of each local area to form a local feature map. In this way, CNN can effectively capture low-level and mid-level features in the image, providing the necessary detailed information for subsequent posture and position prediction.

[0041] At the same time, the Transformer branch performs global feature extraction on the initial feature map. The Transformer's self-attention mechanism allows the model to not only focus on local areas when processing images, but also capture the contextual information of the entire image. By dividing the image into multiple blocks and calculating the relationship between them, the Transformer can understand the mutual influence between the various parts of the image and generate a global feature map.

[0042] For example, when the image captured by the front-view camera is input into the Transformer branch, the image embedding operation is first performed, that is, the input image Expand into a series of non-overlapping image blocks .in, represents the input image size, It represents the number of non-overlapping blocks, and P represents the resolution of the image block. Represents the dimension after the image embedding operation. After the image embedding operation, the input Mapped to three matrices through linear transformation: query matrix , key matrix Sum Matrix , the linear transformation is shown in the formula: ; Among them, the linear mapping matrix , and The value of needs to be determined through learning. Get the query matrix , key matrix Sum Matrix Then, on this basis, the multi-head attention mechanism is calculated, and the calculation process is shown in the formula: ;in, represents the output of the multi-head attention mechanism, represents the query matrix, K represents the key matrix, represents the transpose of the key matrix, is the dimension of the key, Represents the K vector dimension.

[0043] A global feature map is obtained by calculation, which is crucial for the overall understanding of the scene. Especially when dealing with complex backgrounds or long-distance dependencies, Transformer shows stronger capabilities than traditional CNN.

[0044] By combining these two network architectures, we can make full use of local and global information, thereby providing richer and more accurate input features for subsequent feature fusion and pose regression.

[0045] Furthermore, step P44 of the embodiment of the present application also includes: P44-1: Input the local feature map and the global feature map into the feature interaction module for feature interaction, such as Figure 2 , Figure 3As shown, the feature interaction module includes a global interaction module and a local interaction module; P44-2: the global interaction module is used to upsample the local feature map output by the CNN branch, perform feature concatenation, dimensionality reduction and activation with the global feature map output by the Transformer branch, and output the optimized global feature map; P44-3: the local interaction module is used to downsample the global feature map output by the Transformer branch, perform feature concatenation, dimensionality reduction and activation with the local feature map output by the CNN branch, and output the optimized local feature map; P44-4: according to the optimized global feature map and the optimized local feature map, a fused feature map is output.

[0046] Optionally, the local feature map and the global feature map are further fused and optimized through the feature interaction module to provide more accurate input features for subsequent pose regression. This process mainly includes the collaborative work of the global interaction module and the local interaction module, and the functions of these two modules are to optimize the feature map and perform information fusion.

[0047] First, the local feature map and global feature map obtained from the CNN branch and Transformer branch respectively are input into the feature interaction module.

[0048] In the global interaction module, the local feature map output by the CNN branch is first upsampled. The upsampling operation is usually used to increase the resolution of the feature map so that it can match the global feature map. Next, the local feature map is feature concatenated with the global feature map output by the Transformer branch, that is, these two different types of feature maps are combined in the channel dimension to form a new comprehensive feature map. The concatenated feature map will also be reduced in dimension, and the number of channels will be reduced through convolution or fully connected layers to retain important feature information. Finally, after being processed by the activation function, the optimized global feature map is output. The role of the activation function is to enable the model to capture more complex features by introducing nonlinear transformations.

[0049] Similar to the global interaction module, the local interaction module downsamples the global feature map output by the Transformer branch. The downsampling operation is usually used to reduce the resolution of the feature map so that it can match the local feature map. Then, the optimized local feature map and the global feature map are feature concatenated. In this way, the information of the local feature map and the global feature map is complementary, which is convenient for subsequent feature processing. The concatenated feature map will undergo dimensionality reduction to further reduce redundant information, and will be activated to output the optimized local feature map. In this way, local information is strengthened and effectively combined with global information.

[0050] Finally, based on the optimized global feature map and the optimized local feature map, the two are fused to output the final fused feature map. This fused feature map contains the global information and local details in the image and is the input for the next step of pose regression. Through this multi-level and multi-dimensional information fusion, the spatial structure in the image can be understood more accurately, thereby improving the accuracy of camera pose relocation.

[0051] Furthermore, step P45 of the embodiment of the present application also includes: P45-1: wherein the pose regressor is a multi-layer perceptron, including two fully connected layers and an activation function; P45-2: inputting the fused feature map into the pose regressor, concatenating the obtained three-dimensional position feature vector and three-dimensional posture feature vector, and outputting a six-dimensional pose feature vector; P45-3: outputting the second camera pose information based on the six-dimensional pose feature vector.

[0052] Specifically, the input fusion feature map is regressed and analyzed by the pose regressor to predict the pose information of the camera. Specifically, this process mainly includes the extraction of three-dimensional position and pose features, feature splicing, and the final pose output.

[0053] The pose regressor is a multi-layer perceptron (MLP) composed of a fully connected layer and an activation function. The fully connected layer is used to convert the input feature map into the required pose information. The fully connected layer is a common network layer in deep learning, which maps the input data to the output space by weighting and biasing, and then generates the final prediction result. In this step, the role of the fully connected layer is to convert the fused feature map after the feature interaction module and optimization into a specific pose vector. The activation function is usually applied after each fully connected layer to introduce nonlinearity so that the model can capture complex patterns. Common activation functions include ReLU, Sigmoid or Tanh, where they help the neural network learn and adjust complex mapping relationships and enhance the expressiveness of the model.

[0054] First, the optimized fusion feature map is input into the pose regressor for processing. The pose regressor extracts the 3D position feature vector and the 3D pose feature vector from it. The 3D position feature vector contains the X, Y, and Z coordinate information of the camera in the air, indicating the position of the camera in the world coordinate system; while the 3D pose feature vector contains the roll angle, pitch angle, and yaw angle of the camera, describing the direction and spatial attitude of the camera. After being processed by the regressor, these two vectors represent the position information and attitude information of the camera respectively. Next, these two feature vectors are spliced ​​to form a six-dimensional pose feature vector, which contains both the 3D position and 3D attitude information of the camera. The function of feature splicing is to combine the position and attitude information to form a complete feature vector containing all necessary pose information.

[0055] Finally, the second camera pose information is output based on the spliced ​​six-dimensional pose feature vector. This output is the final predicted pose of the camera, which includes the camera's three-dimensional position and three-dimensional attitude, that is, the camera's accurate coordinates and orientation. Through this regression process, the camera's pose during the approach and landing phase can be accurately predicted, providing key data for subsequent navigation and control.

[0056] Furthermore, the camera relocalization network in the embodiment of the present application is obtained by introducing a loss function for supervised learning training, and the loss function includes: ;in, Used to measure the distance error between the true position and the predicted position. is the dynamic weight factor of the position error, To dynamically adjust the weight of the position error, Used to measure the normalized difference between the true pose and the predicted pose, is the dynamic weight factor of the attitude error, To dynamically adjust the weight of the posture error; , is the actual camera position information, is the predicted camera position information, is the Euclidean norm used to calculate the Euclidean distance between two position vectors; , is the actual camera posture information, is the predicted camera pose information, Normalization of the predicted camera pose information.

[0057] It should be understood that in this step, the loss function is introduced for supervised learning training to optimize the performance of the camera relocalization network. The loss function is an indispensable part of supervised learning, and its purpose is to measure the difference between the predicted value and the true value, and to update the weight of the model by the difference, thereby improving the accuracy of the model. In an embodiment of the present application, the loss function includes two main components: position error and attitude error.

[0058] Among them, the position error ( ) calculated the true position of the aircraft ( ) and the predicted position ( ), the Euclidean distance between them, the attitude error ( ) is used to calculate the true attitude of the aircraft ( ) and predicted pose ( ), since the posture is usually expressed in the form of unit quaternion, the validity of the rotation transformation is ensured. The total loss function is composed of the weighted sum of the position error and the posture error. Introducing dynamic weight factors (such as and ) is to balance the position change rate and attitude change rate during training to ensure that both position and attitude errors can be effectively optimized.

[0059] By introducing this loss function with dynamic adjustment, it is possible to comprehensively consider the different rates of change of position and attitude during the aircraft's approach and landing, optimize the accuracy of camera relocalization, ensure the adaptability of the loss function, and help the camera relocalization network better fit the actual flight path and attitude changes.

[0060] For example, as shown in Table 2, the accuracy of the LandNet method disclosed in the present invention is compared with that of the existing camera relocalization method: Table 2 Comparison of the solution accuracy of LandNet and existing camera relocalization methods As shown in Table 2, the camera repositioning method disclosed in the present invention has the best position errors in the three directions of X, Y, and Z. The average error in the X direction is 6.4 meters, and the average error in the Y direction is 4.1 meters. The average error in the height direction is 0.67 meters, and the position accuracy meets the requirements for aircraft landing; in terms of angle error, the LandNet disclosed in the present invention achieves a roll angle error of 0.011 degrees, a pitch angle error of 0.003 degrees, and a yaw angle error of 0.011 degrees. The attitude accuracy also meets the navigation requirements of the aircraft during the approach and landing process.

[0061] In summary, the embodiments of the present application have at least the following technical effects: This application uses an onboard forward-looking camera to capture images of a fixed-wing aircraft approaching and landing, and obtains the posture information (including position and attitude) of the aircraft. Through the coordinate conversion module, the position and attitude information of the fixed-wing aircraft is converted into the first camera posture information. Then, a camera relocalization network is constructed, and the image and the first camera posture information are input into the network for feature extraction, and a fused feature map is output. The camera is relocated according to the fused feature map, and finally the second camera posture information, that is, the position and attitude of the relocated camera, is output.

[0062] The technical effect of providing accurate aircraft positioning when signals are limited is achieved through the onboard forward-looking camera and relocation network.

[0063] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0064] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0065] This specification and the drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. A camera repositioning method for fixed-wing aircraft approach and landing, characterized in that: The method comprises: Use the onboard forward-looking camera to collect images of fixed-wing aircraft approaching and landing; Acquire the aircraft posture information of the fixed-wing aircraft, wherein the aircraft posture information includes the position information and attitude information of the aircraft; The aircraft posture information is converted by a coordinate conversion module to obtain first camera posture information, wherein the first camera posture information includes position information and posture information of the forward-looking camera in a world coordinate system; Constructing a camera relocalization network, wherein the camera relocalization network includes an initial feature extraction module, a branch feature extraction module, a feature interaction module and a pose regressor; The image and the first camera pose information are input into the camera relocalization network for feature extraction, a fused feature map is output, the camera is relocalized according to the fused feature map, and second camera pose information is output, wherein the second camera pose information is the relocated camera pose information.

2. The method according to claim 1, characterized in that The method of outputting the second camera pose information includes: According to the initial feature extraction module, obtaining an initial feature map; Inputting the initial feature map into the branch feature extraction module to perform enhanced feature extraction, and obtaining a local feature map and a global feature map; Inputting the local feature map and the global feature map into the feature interaction module for feature interaction, and outputting the fused feature map; The pose regressor performs regression analysis on the fused feature map and outputs camera pose information, that is, position information and pose information after relocation.

3. The method according to claim 2, characterized in that Inputting the initial feature map into the branch feature extraction module for enhanced feature extraction to obtain a local feature map and a global feature map, wherein the branch feature extraction module includes a CNN branch and a Transformer branch; The CNN branch includes four CNN convolutional structural blocks, and the convolution size of each convolutional structural block is different; the Transformer branch includes four Transformer structural blocks, and the segmentation granularity of each Transformer structural block is different; Perform enhanced feature extraction on the initial feature map according to the CNN branch, and output a local feature map; Enhanced feature extraction is performed on the initial feature map according to the Transformer branch, and a global feature map is output.

4. The method according to claim 3, characterized in that Inputting the local feature map and the global feature map into the feature interaction module for feature interaction, wherein the feature interaction module includes a global interaction module and a local interaction module; The global interaction module is used to upsample the local feature map output by the CNN branch, perform feature concatenation, dimensionality reduction and activation on the global feature map output by the Transformer branch, and output an optimized global feature map; The local interaction module is used to downsample the global feature map output by the Transformer branch, perform feature concatenation, dimension reduction and activation on the local feature map output by the CNN branch, and output an optimized local feature map; According to the optimized global feature map and the optimized local feature map, a fused feature map is output.

5. The method according to claim 2, characterized in that The pose regressor performs regression analysis on the fused feature map and outputs the pose information of the second camera. include: Wherein, the pose regressor is a multi-layer perceptron, including two fully connected layers and an activation function; Inputting the fused feature map into the pose regressor, concatenating the obtained three-dimensional position feature vector and three-dimensional pose feature vector, and outputting a six-dimensional pose feature vector; Output second camera pose information according to the six-dimensional pose feature vector.

6. The method according to claim 1, characterized in that The camera relocalization network is obtained by introducing a loss function for supervised learning training, and the loss function includes: ； in, Used to measure the distance error between the true position and the predicted position. is the dynamic weight factor of the position error, To dynamically adjust the weight of the position error, Used to measure the normalized difference between the true pose and the predicted pose, is the dynamic weight factor of the attitude error, To dynamically adjust the weight of the posture error; , is the actual camera position information, is the predicted camera position information, is the Euclidean norm used to calculate the Euclidean distance between two position vectors; , is the actual camera posture information, is the predicted camera pose information, Normalization of the predicted camera pose information.

7. The method according to claim 1, characterized in that The method of the coordinate conversion module includes: Defining a plurality of coordinate systems, the plurality of coordinate systems comprising an earth-centered earth-fixed coordinate system, a navigation coordinate system, a carrier coordinate system, a camera coordinate system, and a world coordinate system; The coordinates of the multiple coordinate systems are uniformly transformed, a coordinate transformation matrix is ​​output, and the position and posture information of the aircraft is analyzed according to the coordinate transformation matrix to obtain the position and posture information of the camera in the world coordinate system.

8. The method according to claim 7, characterized in that Expression for uniform coordinate transformation of the multiple coordinate systems: ； in, To convert the geographic location into three-dimensional coordinates in the camera coordinate system, Including converting the longitude, latitude and altitude in the WGS84 coordinate system into the three-dimensional coordinates of the Earth-centered Earth-fixed coordinate system ECEF, and converting the coordinates of the Earth-centered Earth-fixed coordinate system ECEF into the world coordinate system ; Including the navigation coordinate system To the carrier coordinate system The conversion and definition of the carrier coordinate system based on the camera installation relationship To the camera coordinate system conversion.

Images